Home Python Asynchronous (Parallel) REST API Requests with Pagination
Reply: 0

Python Asynchronous (Parallel) REST API Requests with Pagination

user5731 Published in September 21, 2018, 8:04 am

I am having trouble wrapping my head around asynchronous/parallel requests using Python 3.5+ asyncio (I believe more specifically aiohttp).

I have read multiple examples that define how to do asynchronous requests using aiohttp but they all seem to use a static predefined URL to hit multiple times or a static predefined list of URLs.

What I'm trying to accomplish is send multiple (say two) parallel requests at a time to a singular REST API endpoint that implements an offset counter to paginate records and continue to increment with each iteration until all records returned via the API are exhausted.

The REST API returns JSON data that looks like this:

    "user_id": "user1",
    "first_name": "User",
    "last_name": "One"
}, {
    "user_id": "user2",
    "first_name": "User",
    "last_name": "Two"
}, {
    "user_id": "user3",
    "first_name": "User",
    "last_name": "Three"

These repeated GET requests would return 45 records total, ten items at a time:

GET /api/users/?limit=10&offset=0
GET /api/users/?limit=10&offset=10
GET /api/users/?limit=10&offset=20
GET /api/users/?limit=10&offset=30
GET /api/users/?limit=10&offset=40
GET /api/users/?limit=10&offset=50

Given there are only 45 records, the last query would return an empty number of records and the second to last query would return only five records (out of a max of ten).

The goal is to stop incrementing the offset and stop generating new requests as the number of records returned would be less than the limit in the request (record exhaustion). It would also allow the existing threads to finish their full request and combine the results into one list that could be processed later in the code, unrelated to the API request itself.

My internet searches have returned very little, specifically around paginating requests, so I hope someone on Stack Overflow can help me grasp the basics of what I am missing so I can learn and build upon it.

share|improve this question

1 Answer 1

active oldest votes
up vote 0 down vote
You need to login account before you can post.

About| Privacy statement| Terms of Service| Advertising| Contact us| Help| Sitemap|
Processed in 0.323375 second(s) , Gzip On .

© 2016 Powered by mzan.com design MATCHINFO