Paginators in the AWS SDK for Javascript V3

How many times have you found yourself implementing pagination manually while retrieving data from the AWS SDK for Javascript? Although the version 3 of the SDK has introduced numerous built-in pagination utilities for many of its clients, many users are still not aware of these features. As a result, most still implement pagination manually, which is more time-consuming and prone to errors.

In this post, we’ll discuss how to use the paginate* utilities found throughout the SDK by looking at a specific example about paginating DynamoDB queries with paginateQuery.

tl;dr - it will be this easy:

const paginator = paginateQuery(paginatorConfig, params);

const items: any[] = [];
for await (const page of paginator) {
  items.push(...page.Items);
}

Manual pagination

Let’s take the example of a DynamoDB query. When doing things manually, you would need to make a first query, check if there is a LastEvaluatedKey in the response, and if so, keep making queries with the ExclusiveStartKey set to the LastEvaluatedKey of the previous response until it is undefined.

By using the paginateQuery method instead, you can get rid of most of this boilerplate code. This same pattern can be applied to any other clients in the SDK that support paginators.

Using Paginators

Many AWS operations return paginated results when the response object is too big to be returned in a single response. For these operations, the AWS SDK for JavaScript usually exposes utility methods all named with the paginate<OperationName> pattern.

These methods are written using Async Generators and return Async Iterators that can be used with a for await...of loop.

A DynamoDB example using the paginateQuery method

Let’s see an example using paginateQuery to paginate over a query operation with the DynamoDB Document client.

Here is the signature of the relevant paginator from the documentation:

export declare function paginateQuery(
  config: DynamoDBPaginationConfiguration,
  input: QueryCommandInput,
  ...additionalArguments: any
): Paginator<QueryCommandOutput>;

As you can see, it takes as inputs:

The return type is a Paginator object which can be iterated over, where each iteration returns a QueryCommandOutput object containing the results of the query.

The following is a fully typed example showing its usage with TypeScript:

import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import {
  DynamoDBDocument,
  QueryCommandInput,
  paginateQuery,
  DynamoDBDocumentPaginationConfiguration
} from '@aws-sdk/lib-dynamodb';

const REGION = 'eu-west-1';
const TABLE_NAME = 'my-table';
const PK_QUERY_VALUE = 'my-pk';

// Create a DynamoDB Document client
const docClient = DynamoDBDocument.from(
  new DynamoDBClient({
    region: REGION
  })
);

// Create a paginator configuration
const paginatorConfig: DynamoDBDocumentPaginationConfiguration = {
  client: docClient,
  pageSize: 25
};

// Query parameters
const params: QueryCommandInput = {
  TableName: TABLE_NAME,
  KeyConditionExpression: 'pk = :pk',
  ExpressionAttributeValues: {
    ':pk': PK_QUERY_VALUE
  }
};

// Create a paginator
const paginator = paginateQuery(paginatorConfig, params);

// Paginate until there are no more results
const items: any[] = [];
for await (const page of paginator) {
  items.push(...page.Items);
}

As you can see, by using paginator methods, you don’t need to worry anymore about the underlying pagination logic which is translated into a simple for (await) loop. The same example could be used for the paginateScan method or any other SDK paginated operation with minimal changes.

In conclusion, if you are using the AWS SDK for Javascript, it’s worth taking the time to learn how to use paginators effectively to make your code easier to read and maintain, as they are available for many clients throughout the SDK.

References