Task #2951
Updated by bmbouter over 7 years ago
The way we offer concurrency in the plugin API is with the Batch() object which is built on code that we have to carry and maintain. Concurrent I/O (such as concurrent downloads) is a great fit for "asyncio":https://docs.python.org/3/library/asyncio.html and we should: * Create a ConcurrentHTTPDownloader which is designed to be managed by an asyncio event loop created by the plugin writer. This writer * Create a ConcurrentFTPDownloader which is described some on designed to be managed by an asyncio event loop created by the wiki as the "ConcurrentDownloader":https://pulp.plan.io/projects/pulp/wiki/Pulp3_Plugin_Brainstorming. plugin writer * Delete the Batch() object and it's supporting machinery. I think that is effectively everything in "batch.py":https://github.com/pulp/pulp/blob/3.0-dev/platform/pulpcore/download/batch.py * Add a ContentUnitDownloader One side-effect of this switch is that we may need to switch from the download API, which is described "here":https://pulp.plan.io/projects/pulp/wiki/Pulp3_Plugin_Brainstorming as requests library to the Content Unit Downloader "aiohttp":http://aiohttp.readthedocs.io/en/stable/ library. Whatever we use needs to be "asyncio" aware. In other words it needs to support to co-routine pattern required by asyncio. This "SO post":https://stackoverflow.com/questions/22190403/how-could-i-use-requests-in-asyncio#22414756 claims that you *could* use requests if you wanted to.
.