Issue #1524: Refactor the pulp_streamer so that it can pool connections - Pulp

Actions

Send by e-mail Copy link

Issue #1524

closed

Story #1150: As a user, I can lazily fetch repositories

Refactor the pulp_streamer so that it can pool connections

Added by jcline@redhat.com almost 9 years ago. Updated over 5 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

High

Assignee:

jcline@redhat.com

Category:

Sprint/Milestone:

Start date:

Due date:

Estimated time:

Severity:

3. High

Version:

Platform Release:

2.8.0

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Pulp 2

Sprint:

Quarter:

Description

Currently the streamer makes use of Nectar's ``download_one`` method, so we are not utilizing urllib3's connection pool. Under load (even from five concurrent nectar threads during the lazy download task) HTTP 503 Service Unavailable are occasionally received. The current work-around is to just dispatch another task to try again later, but it would be better if we pooled the connections.

Actions

Copy link

Updated by bmbouter almost 9 years ago

We were forced to use download_one because it was the only single threaded option that Nectar provided. We cannot use a thread pool with twisted unless the twisted reactor is in control of the thread pool. Given that, is there still an improvement we can make here?

Actions

Copy link

Updated by jcline@redhat.com almost 9 years ago

I think it's definitely worth evaluating carefully. Without it, we will not perform well under moderate to heavy load, as I believe many clients will receive a healthy number of HTTP 503s.

Actions

Copy link