How to read more than 1000 items from S3 in Java

AWS SDK for S3 provides API to manage S3 buckets and objects. It supports higher level abstractions for simplified development. In this article, you can learn how to integrate S3 with a Spring Boot application and make API requests to S3.

AWS S3 API can return a maximum of 1000 items per request. We need to work through several pages of API responses to fully list all items within the S3 bucket.

If we want to list items in a S3 bucket, we can easily list items using the following code snippet.

ListObjectsV2Request listObjectsV2Request = new ListObjectsV2Request();
List<S3ObjectSummary> objectSummaries =
        amazonS3Client.listObjectsV2(listObjectsV2Request).getObjectSummaries();

This code snippet works when you have less than 1000 items in a bucket. If there are more than 1000 items in a bucket, we need to use getNextContinuationToken which returns token to get next results.

According to Java doc, NextContinuationToken is sent when isTruncated is true meaning there are more keys in the bucket that can be listed. The next list requests to Amazon S3 can be continued by providing this NextContinuationToken.

final ListObjectsV2Request listObjectsV2Request = new ListObjectsV2Request();
listObjectsV2Request.setBucketName(myBucketName)…