Application Programming Interfaces (APIs) have become an essential part of modern web development. They allow different software systems to communicate with each other and exchange data. APIs provide a standardized way for systems to expose their functionality and data to other applications.
One common challenge when working with APIs is handling pagination. Many APIs will limit the number of records returned in a single API request to improve performance. This means the API consumer needs to make multiple requests to retrieve all the available data. Pagination allows retrieving a subset of the data at a time.
In this article, we will explore what pagination is, why it’s used, and how to implement pagination when consuming an API in your application. We will look at techniques like offset-based pagination, cursor-based pagination, and keyset pagination. We will also examine libraries and frameworks that can simplify working with paginated APIs.
Understanding pagination is crucial for building robust applications that interact with APIs efficiently. Let’s get started!
What is Pagination?
Pagination is a mechanism used to divide data into discrete pages when there are too many records to display at once. APIs commonly employ pagination to limit the number of objects returned in a single API call.
Here are some key facts about pagination:
– Pagination breaks up large data sets into smaller chunks called pages.
– Only a subset of records is returned per API request. For example, 50 records per page.
– Metadata is included in the API response to enable fetching the next/previous page of records.
– The consumer needs to make multiple API requests to get all records by iterating through the pages.
– Helps improve API performance and scalability by limiting data volumes per request.
– Reduces latency since less data is transmitted per API call.
The page size, which is the number of records per page, is configurable on the server-side. Some APIs may support client-specified page sizes as well. There are also options like getting the first N records or the last N records if you don’t need full pagination.
Why is Pagination Used?
There are several benefits of using pagination in APIs:
– **Scalability:** Paginating data enables the API to handle more requests since each call returns fewer records. This allows the API server to scale better.
– **Performance:** Transferring large responses over the network degrades performance. Pagination reduces payload sizes for better speed.
– **Caching:** Cached page data expires faster since each page is smaller in size compared to one huge response. This improves cache hits.
– **Usability:** Displaying all data on one page is suboptimal for the API consumer. Pagination makes data more manageable.
– **Efficiency:** The client can stop fetching when sufficient records are retrieved rather than getting everything in one giant response.
Overall, pagination lightens the load on the API server and provides a better experience for API consumers. The tradeoff is that it requires multiple requests to get the full data set.
Types of Pagination
There are three primary ways APIs can implement pagination:
Offset-based Pagination
This is the most common approach. The API accepts offset and limit parameters.
Offset indicates how many records to skip from the start. Limit specifies the page size or number of items to return.
For example, to get records 11 to 20, you would request:
“`
/api/v1/users?offset=10&limit=10
“`
The first call skips the first 10 records and returns the next 10. You increment the offset to navigate to subsequent pages.
Offset-based pagination is easy to understand but has some drawbacks:
– Performance degrades at higher offsets if the data is not indexed properly.
– Hard to modify data without impacting the offset values.
– Concurrent modifications can affect the consistency of pages.
Cursor-based Pagination
Here pagination is based on cursors which are pointers to specific records in the dataset.
The API accepts a page size and a cursor parameter which marks the start point for the page. The response includes a cursor to the next page.
For example:
“`
GET /api/v1/users?limit=15&cursor=MTAx
“`
This returns 15 records starting from the object with id 101. The response will contain a nextCursor to get the next page.
Benefits of cursor-based pagination:
– Handles gaps in the dataset better, like deleted records.
– Stable even if data changes, since cursors point to objects.
– Works well for fetching newer records by passing the last retrieved cursor.
Cursors are usually implemented using object identifiers or timestamps.
Keyset Pagination
This pagination method relies on a unique key or set of keys to paginate data.
The consumer provides the last seen keys with the request to get the next page. This returns records with keys after the specified ones.
For example, if the last returned record had id 555, the next request would be:
“`
GET /api/v1/users?limit=50&last_id=555
“`
This fetches the next 50 records with ids higher than 555.
Keyset pagination has some advantages:
– Provides stable pagination even with changing data.
– Good performance since records are queried based on keys.
– Can paginate on multiple keys like updated times.
The main challenges are picking the right indexes as keys and handling edge cases like duplicate keys.
Implementing Pagination
Let’s go through a simple example of paginating an API response in a React application using offset-based pagination.
We will use the JSON Placeholder API that provides fake data for testing. The endpoint we will work with is:
“`
https://jsonplaceholder.typicode.com/posts
“`
It returns 100 posts or records. We will add pagination to fetch these posts in pages of 10 records.
Paginated API Responses
First, let’s see how a paginated API normally structures its response.
A paginated response contains:
– The page data as an array of records.
– Metadata like total records, current page, next page details etc.
For example:
“`json
{
“data”: [
{
“post_id”: 1,
“title”: “Post 1”
},
…
],
“total”: 100,
“current_page”: 1,
“next_page”: 2
}
“`
The data field has the returned records for the page. Total records allow calculating how many pages there are. Current page tells us which page we are on. Next page provides the details to fetch the next set of data.
Some APIs may provide previous page, page size, or other info as well in the metadata.
Fetching Paginated Data
Now let’s look at how we can consume such a paginated API from a React app:
1. **Add state -** We need to track the current page number in component state:
“`jsx
const [page, setPage] = useState(1)
“`
2. **Define page size -** Define how many records needed per page:
“`js
const pageSize = 10
“`
3. **Fetch page data -** Hit the API on component mount:
“`js
useEffect(() => {
fetchPaginatedData()
}, [])
const fetchPaginatedData = () => {
fetch(`https://jsonplaceholder.typicode.com/posts?_page=${page}&_limit=${pageSize}`)
.then(res => res.json())
.then(data => {
// update state
})
}
“`
We use the `_page` and `_limit` query parameters supported by this API to implement pagination.
4. **Update state** – Store the fetched page data and metadata in state:
“`js
const { data, total, currentPage, nextPage } = apiResponse
setPosts(data)
setTotalRecords(total)
setCurrentPage(currentPage)
“`
5. **Fetch next page** – Call the API again with updated page number when the user clicks Next Page:
“`js
const handleNextPage = () => {
setPage(page + 1)
}
// Re-fetches data with new page
“`
That’s it! The posts will be fetched from the API paginated 10 records at a time. We can build on this logic to handle previous page, first/last page, pagination UI, and more.
Cursor-based Pagination Example
Let’s briefly see an example of cursor-based pagination as well. We will use the Github API to fetch a user’s repositories paginated with cursors.
The endpoint is:
“`
https://api.github.com/users/
“`
It supports `per_page` and `page` params similar to offset pagination. But we will leverage the `Link` header it provides to enable cursoring.
The `Link` header contains information to get the next and previous pages:
“`
Link:
“`
Here page 2 URL is provided in the `next` link.
To fetch with cursors:
1. Get first page – Fetch records without any cursor initially.
2. Extract next link – Parse the `Link` header for the next page URL.
3. Fetch next page – Use next link as cursor to hit the API again.
4. Repeat – Get subsequent pages by extracting and following next links.
So in essence, we use the next page details as the cursor for stable pagination.
Handling Pagination on Frontend
When consuming a paginated API in your frontend application, here are some tips:
– Initialize page state variable to track pagination.
– Extract required metadata like total records from response.
– Use previous page link/cursor to fetch earlier pages.
– Save last seen cursor to easily fetch newer pages next visit.
– Implement infinite scrolling by fetching next page on scroll reach.
– Display page numbers and navigation buttons for pagination UIs.
– Indicate loading states when fetching pages.
– Gracefully handle errors for missing pages or records.
– Cache page data to avoid redundant requests.
– Prefetch next page on current page render to improve latency.
– Make sure design adapts well to variable page lengths.
Pagination UI Examples
There are several options when it comes to displaying paginated data:
– **Page numbers** – Show clickable page numbers to let user navigate. Highlight current page.
– **Previous/next buttons** – Arrow buttons to go to prev/next page. Disable on first/last.
– **Infinite scrolling** – Fetch next page when user scrolls to bottom. Indicator for loading.
– **Load more button** – Button to trigger fetching the next page.
Combine approaches for most flexibility – e.g. infinite scroll + page numbers.
Here are some examples of different pagination UIs:
Numbered Pagination
![Numbered Pagination UI](pagination1.png)
Prev/Next Pagination
![Prev/Next Pagination](pagination2.png)
Infinite Scroll Pagination
![Infinite Scroll Pagination](pagination3.png)
Pagination Client Libraries
Implementing pagination logic on your own can be complex. Several client libraries and frameworks provide utilities to handle pagination for you:
**React Query** – React hook for fetching and caching remote data that supports paginated queries.
**Apollo Client** – GraphQL client that fetches connections with auto-pagination. Has React hooks.
**Relay** – Framework for building GraphQL apps with configurable pagination containers.
**SWR** – React hooks library with streaming data and pagination helpers.
These libraries take care of cursor or offset tracking, fetching pages automatically, combining page data, and updating UI.
For example, React Query allows declaring a paginated query by passing a keepPreviousData option:
“`js
const posts = useQuery(‘posts’, fetchPosts, {
keepPreviousData: true
})
“`
It handles caching and refetching pages in the background when enabled.
Leveraging a pagination library can help boost productivity and reduce boilerplate code when working with paginated APIs.
Pagination on Server-side
Let’s explore how pagination can be implemented in server APIs:
Offset-based Pagination
Steps to enable offset-based pagination in the API:
1. Accept `offset` and `limit` parameters in request
2. Validate input for positive integers
3. Fetch records using `OFFSET` and `LIMIT` clauses in SQL query
4. Return total count for number of available records
5. Include next page `offset` in response
Example MySQL query:
“`sql
SELECT * FROM posts
ORDER BY id
LIMIT 10 OFFSET 20;
“`
This returns records 21 to 30 from the table to implement paging.
Cursor-based Pagination
To build cursor-based pagination:
1. Accept a `cursor` parameter in API request
2. Map cursor to unique row identifier like primary key id
3. Fetch rows after the row identified by cursor
4. Return cursor for next page in response
For example:
“`sql
SELECT * FROM posts
WHERE id > :cursor_id
ORDER BY id ASC LIMIT 10;
“`
This fetches next page after the row with cursor id.
Keyset Pagination
For keyset pagination:
1. Define pagination key – unique indexed field like id or timestamp
2. Accept `last_key_value` parameter in API request
3. Fetch rows where pagination key is greater than `last_key_value`
4. Return first key value for next page
Example with id as pagination key:
“`sql
SELECT * FROM posts
WHERE id > :last_id
ORDER BY id ASC LIMIT 10;
“`
This returns the next page of rows based on the last seen id.
The backend needs to handle scenarios like duplicate keys and new inserts gracefully. Overall, the database should leverage indexes on pagination keys for optimal performance.
Conclusion
Paginating responses is essential for performance and usability when building robust APIs.
Offset-based pagination is the simplest approach to get started. Cursor and keyset pagination provide more robust alternatives.
On the client, leverage pagination libraries for React, GraphQL, and other frameworks whenever possible. They greatly simplify handling paginated data.
For smooth UX, combine approaches like infinite scroll and page number navigation. Indicate loading and empty states properly.
On the server, utilize database query optimizations and indexes to efficiently service paginated requests.
With the right techniques and libraries, consuming paginated APIs in client apps becomes easy. The end result is fast, scalable, and pleasant APIs both client and server side.