Story #1150: As a user, I can lazily fetch repositories
Refactor the pulp_streamer so that it can pool connections
Currently the streamer makes use of Nectar's ``download_one`` method, so we are not utilizing urllib3's connection pool. Under load (even from five concurrent nectar threads during the lazy download task) HTTP 503 Service Unavailable are occasionally received. The current work-around is to just dispatch another task to try again later, but it would be better if we pooled the connections.
Updated by bmbouter over 6 years ago
We were forced to use download_one because it was the only single threaded option that Nectar provided. We cannot use a thread pool with twisted unless the twisted reactor is in control of the thread pool. Given that, is there still an improvement we can make here?