Are you sending all your records to the frontend, slowing down your app performance?
Worried about the performance downgrade that you will suffer as soon as more userbase starts to grow?
Look no further! Your problems are gone.
No, I'm not gonna sell you some random npm package or anything, instead, I'm gonna teach you how pagination works, and how you can implement for your backend.
Table of content
- But what is pagination anyways?
- Implementing pagination on MongoDB / Mongoose with node.js
- Implementing pagination on client
- Tips and Tricks
- Example repository
- Conclusion
But what is pagination anyways?
Let's imagine your database table (or collections) as an excel spreadsheet.
Every registry is on a row that has a particularly number and a particularly order
So, pagination is when you want just a chunk (or a page) of rows at the same time, let's say 20 per page.
On the first page, we want to select from the starting position 0 to the 20th element.
Then, on the second page, we want to select the elements from position 20 to position 40th.
And so on and so forth.
Implementation - Server Side
As far as I know, there are two ways to implement pagination, by letting the client decide what is the page size, or by not allowing it.
Option 1 - The client decides page size
Use this method when you have an API that can be consumed by different clients, such as a web application and a mobile application, therefore each dev team can decide what suits best for their needs
For this approach, it is useful to picture the pagination concept as if we were in an Excel spreadsheet, that way, the variables that we will send make a little bit more sense.
The first one is the limit that stands for the desired page size.
Then the second would be the skip, that means how many rows the database should "jump over".
// An example of a controller function for Express.js
async getAllUser (req, res) {
try {
const limit = parseInt(req.query.limit); // Make sure to parse the limit to number
const skip = parseInt(req.query.skip);// Make sure to parse the skip to number
// We are using the '3 layer' architecture explored on the 'bulletproof node.js architecture'
// Basically, it's just a class where we have our business logic
const userService = new userService();
const users = await userService.getAll(limit, skip);
return res.status(200).json(users);
} catch(e){
return res.status(500).json(e)
}
},Then when calling the database with Mongoose (or can be done with native MongoDB driver too)
class UserPaginationExample {
getAll(limit = 0, skip = 0) {
return UsersModel.find({}) // You may want to add a query
.skip(skip) // Always apply 'skip' before 'limit'
.limit(limit) // This is your 'page size'
}
}Another example using MongoDB's aggregation framework
class UserPaginationExample {
getAll(limit = 0, skip = 0) {
return UsersModel.aggregate([
{ $match: {} }, // This is your query
{ $skip: skip }, // Always apply 'skip' before 'limit'
{ $limit: limit }, // This is your 'page size'
])
}
}Option 2 - The server decides page size
For this method, the client just sent what page number they want, and trust the server of delivering the correct page size.
Inside the server, you still have limit and skip for internal usage, and the process is pretty much the same as before.
// An example of a controller function for Express.js
async getAllUser (req, res) {
try {
const page = parseInt(req.query.page); // Make sure to parse the page to number
// We are using the '3 layer' architecture explored on the 'bulletproof node.js architecture'
// Basically, it's just a class where we have our business logic
const userService = new userService();
const users = await userService.getAll(page);
return res.status(200).json(users);
} catch(e){
return res.status(500).json(e)
}
},Then when calling the database with Mongoose (or can be done with native MongoDB driver too)
class UserPaginationExample {
getAll(page = 1) {
const PAGE_SIZE = 20; // Similar to 'limit'
const skip = (page - 1) * PAGE_SIZE; // For page 1, the skip is: (1 - 1) * 20 => 0 * 20 = 0
return UsersModel.find({})
.skip(skip) // Same as before, always use 'skip' first
.limit(PAGE_SIZE)
}
}Another example using MongoDB's aggregation framework
class UserPaginationExample {
getAll(page = 1) {
const PAGE_SIZE = 20; // Similar to 'limit'
const skip = (page - 1) * PAGE_SIZE; // For page 1, the skip is: (1 - 1) * 20 => 0 * 20 = 0
return UsersModel.aggregate([
{ $match: {} },
{ $skip: (page - 1) * PAGE_SIZE },
{ $limit: PAGE_SIZE },
])
}
}I use this method when I don't want to bother my Frontend team with complex API definitions.
Implementation - Client Side
The client code for pagination is pretty easy to do, but I included anyways so you can copy and start using it right away!
React
Using Hooks
import React, { useState, useEffect } from 'react'
const fetchUsers = (limit, skip) => {
// Make sure you send 'limit' and 'skip' as query parameters to your node.js server
fetch(`/api/users?limit=${limit}&skip=${skip}`)
.then((res) => {
this.setState({
users: res.data;
})
})
}
const userList = () => {
const [users, setUsers] = useState([]);
const [limit, setLimit] = useState(20);
const [skip, setSkip] = useState(0);
const nextPage = () => {
setSkip(skip + limit)
}
const previousPage = () => {
setSkip(skip - limit)
}
useEffect(() => {
fetchUsers(limit, skip)
}, [skip, limit])
return (<div>
<div>
{
users.map(user =>
<div>
<span> { user.name } </span>
<span> { user.email } </span>
<span> { user.lastLogin } </span>
</div>
)
}
</div>
<div>
<div onClick={nextPage}> Previous Page </div>
<div onClick={previousPage}> Next Page </div>
</div>
</div>)
}
Using class components
import React from 'react';
class UsersList extends React.component {
constructor(super){
super();
this.state = {
users: [],
// Intial values to get first page
limit: 20,
skip: 0,
}
}
componendDidMount() {
this.fetchUsers();
}
fetchUsers() {
// Make sure you send 'limit' and 'skip' as query parameters to your node.js server
fetch(`/api/users?limit=${this.state.limit}&skip=${this.state.skip}`)
.then((res) => {
this.setState({
users: res.data;
})
})
}
nextPage() {
this.setState({
skip: this.state.skip + this.state.limit,
})
}
previousPage() {
if(this.state.skip > 0) {
this.setState({
skip: this.state.skip - this.state.limit,
})
}
}
componentDidUpdate(prevProps, prevState) {
// Try to avoid doing this, is pretty easy to mess up things with the lifecycle
// So instead, learn to use react hooks, you can read my guide on hooks here: https://softwareontheroad.com/react-hooks/
//
}
render() {
return (<div>
<div>
{
this.state.users.map(user =>
<div>
<span> { user.name } </span>
<span> { user.email } </span>
<span> { user.lastLogin } </span>
</div>
)
}
</div>
<div>
<div onClick={this.nextPage}> Previous Page </div>
<div onClick={this.previousPage}> Next Page </div>
</div>
</div>)
}
}Vue
(Side note: This is my favorite framework/library, it is so easy to use, and so easy to add to any project, I love it.)
Approach 1 - Page size is controlled by Client
<script>
const usersList = new Vue({
el: '#user-list'
data: {
users: [],
limit: 20,
skip: 0,
},
methods: {
nextPage() {
this.skip += this.limit; // For the next page you just increment 'skip' for the page size 'limit'
this.fetchPage();
},
previousPage() {
if(skip > 0) {
this.skip -= this.limit; // For the previous page, you just increment 'skip' for the page size 'limit'
this.fetchPage();
}
},
fetchPage() {
return fetch(`/api/users?limit=${this.limit}&skip=${this.skip}`) // Send 'limit' and 'skip' as query parameters to your node.js server
.then((res) => {
this.users = res.data;
})
},
},
mounted() {
this.fetchPage();
},
})
</script>Approach 2 - Page size is controlled by Server
<script>
const usersList = new Vue({
el: '#user-list'
data: {
users: [],
page: 1,
},
methods: {
nextPage() {
this.page += 1;
this.fetchPage();
},
previousPage() {
if(page > 1) {
this.page -= 1;
this.fetchPage();
}
},
fetchPage() {
return fetch(`/api/users?page=${this.page}`) // Send page number as query parameter to your node.js server
.then((res) => {
this.users = res.data;
})
}
},
mounted() {
this.fetchPage();
},
})
</script>Angular
I wanted to include a few snippets for every major frontend framework, but I don't use Angular pretty often, I'm sorry.
Tips & Tricks - Pagination with node.js and Mongoose
Send count of total documents
Whenever possible, try to add the total number of pages or the total number of documents.
That way, the front-end team can build some awesome pagination buttons.
Source: What's your fav pagination? by Dawson Whitfield
Always apply 'skip' first, 'limit' later.
A common error is to use apply the limit before the skip
class UserPaginationExample {
getAll(limit = 0, skip = 0) {
return UsersModel.aggregate([
{ $match: {} }, // This is your query
{ $limit: limit }, // This is your 'page size'
{ $skip: skip }, // Always apply 'skip' before 'limit'
])
}
}You will spot this problem when you see that your pagination is applying a limit of, for example, 30 documents and skip 10, therefore you only get 20 results.
Use a sorting filter for better pagination results
By default, Mongo sorts the documents from oldest to newest.
So, your first page will have the oldest records first.
Change this behavior by passing a sort parameter at the Mongo collection
class UserPaginationExample {
getAll(limit = 0, skip = 0) {
return UsersModel.find({}) // You may want to add a query
.sort({ _id: -1 }) // Use this to sort documents by newest first
.skip(skip) // Always apply 'skip' before 'limit'
.limit(limit) // This is your 'page size'
}
}
Here is how you do it with MongoDB's aggregation framework
class UserPaginationExample {
getAll(limit = 0, skip = 0) {
return UsersModel.aggregate([
{ $match: {} }, // This is your query
{ $sort: { _id: -1 } } // Use this to sort documents by newest first
{ $skip: skip }, // Always apply 'skip' before 'limit'
{ $limit: limit }, // This is your 'page size'
])
}
}
Conclusion
It doesn't really matter if you use MongoDB with native driver or Mongoose ODM, in fact, it doesn't mather the database you use, the concept is the same for all.
Pagination is a hell of powerful and easy to implement a solution when you have a lot of data that could be sent over to the client.
Use it to improve performance by only loading and displaying what is necessary for your users.
Sending more than 50 objects is never a good idea, in terms of user experience and data usage.
Implement it today to save bandwidth (and money!), we reduced 35% of bandwidth usage on our last project, it saved a couple of hundred dollars off the Netlify bill, our client was very happy.
