Home Python Asynchronous (Parallel) REST API Requests with Pagination
Reply: 1

Python Asynchronous (Parallel) REST API Requests with Pagination

user2966902 Published in 2018-02-12 23:35:19Z

I am having trouble wrapping my head around asynchronous/parallel requests using Python 3.5+ asyncio (I believe more specifically aiohttp).

I have read multiple examples that define how to do asynchronous requests using aiohttp but they all seem to use a static predefined URL to hit multiple times or a static predefined list of URLs.

What I'm trying to accomplish is send multiple (say two) parallel requests at a time to a singular REST API endpoint that implements an offset counter to paginate records and continue to increment with each iteration until all records returned via the API are exhausted.

The REST API returns JSON data that looks like this:

    "user_id": "user1",
    "first_name": "User",
    "last_name": "One"
}, {
    "user_id": "user2",
    "first_name": "User",
    "last_name": "Two"
}, {
    "user_id": "user3",
    "first_name": "User",
    "last_name": "Three"

These repeated GET requests would return 45 records total, ten items at a time:

GET /api/users/?limit=10&offset=0
GET /api/users/?limit=10&offset=10
GET /api/users/?limit=10&offset=20
GET /api/users/?limit=10&offset=30
GET /api/users/?limit=10&offset=40
GET /api/users/?limit=10&offset=50

Given there are only 45 records, the last query would return an empty number of records and the second to last query would return only five records (out of a max of ten).

The goal is to stop incrementing the offset and stop generating new requests as the number of records returned would be less than the limit in the request (record exhaustion). It would also allow the existing threads to finish their full request and combine the results into one list that could be processed later in the code, unrelated to the API request itself.

My internet searches have returned very little, specifically around paginating requests, so I hope someone on Stack Overflow can help me grasp the basics of what I am missing so I can learn and build upon it.

Andrew Svetlov
Andrew Svetlov Reply to 2018-02-13 11:14:10Z

Your question is about forecast magic, not computer science.

If for next request you need a result from previous one -- you cannot execute two requests in parallel.

You need to login account before you can post.

About| Privacy statement| Terms of Service| Advertising| Contact us| Help| Sitemap|
Processed in 0.332348 second(s) , Gzip On .

© 2016 Powered by mzan.com design MATCHINFO