Task #2951
Updated by bmbouter over 7 years ago
The way we offer concurrency in the plugin API is with the Batch() object which is built on code that we have to carry and maintain. Concurrent I/O (such as concurrent downloads) is a great fit for "asyncio":https://docs.python.org/3/library/asyncio.html and we should: * Create a ConcurrentHTTPDownloader which is designed to be managed by an asyncio event loop created by the plugin writer * Create a ConcurrentFTPDownloader which is designed to be managed by an asyncio event loop created by the plugin writer * Delete the Batch() object and it's supporting machinery. I think that is effectively everything in "batch.py":https://github.com/pulp/pulp/blob/3.0-dev/platform/pulpcore/download/batch.py One side-effect of this switch is that we may need to switch from the requests library to the "aiohttp":http://aiohttp.readthedocs.io/en/stable/ http library. Whatever we use needs This is required because the fetching of remote content has to be "asyncio" aware. In other words it needs to support to co-routine pattern required by asyncio. This "SO post":https://stackoverflow.com/questions/22190403/how-could-i-use-requests-in-asyncio#22414756 claims that you *could* use requests if you wanted to.
.