Story #1150: As a user, I can lazily fetch repositories
Refactor the pulp_streamer so that it can pool connections
Currently the streamer makes use of Nectar's ``download_one`` method, so we are not utilizing urllib3's connection pool. Under load (even from five concurrent nectar threads during the lazy download task) HTTP 503 Service Unavailable are occasionally received. The current work-around is to just dispatch another task to try again later, but it would be better if we pooled the connections.
#1 Updated by bmbouter almost 6 years ago
We were forced to use download_one because it was the only single threaded option that Nectar provided. We cannot use a thread pool with twisted unless the twisted reactor is in control of the thread pool. Given that, is there still an improvement we can make here?
Please register to edit this issue