Project

Profile

Help

Issue #1524

Story #1150: As a user, I can lazily fetch repositories

Refactor the pulp_streamer so that it can pool connections

Added by jcline@redhat.com over 5 years ago. Updated about 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
2.8.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Currently the streamer makes use of Nectar's ``download_one`` method, so we are not utilizing urllib3's connection pool. Under load (even from five concurrent nectar threads during the lazy download task) HTTP 503 Service Unavailable are occasionally received. The current work-around is to just dispatch another task to try again later, but it would be better if we pooled the connections.

Associated revisions

Revision 6f917920 View on GitHub
Added by Jeremy Cline over 5 years ago

The streamer now pools connections.

closes #1524

Revision 6f917920 View on GitHub
Added by Jeremy Cline over 5 years ago

The streamer now pools connections.

closes #1524

Revision ae45a691 View on GitHub
Added by Jeremy Cline over 5 years ago

Create a proxy worker that pools TCP connections.

History

#1 Updated by bmbouter over 5 years ago

We were forced to use download_one because it was the only single threaded option that Nectar provided. We cannot use a thread pool with twisted unless the twisted reactor is in control of the thread pool. Given that, is there still an improvement we can make here?

#2 Updated by jcline@redhat.com over 5 years ago

I think it's definitely worth evaluating carefully. Without it, we will not perform well under moderate to heavy load, as I believe many clients will receive a healthy number of HTTP 503s.

#3 Updated by mhrivnak over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jcline@redhat.com
  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High
  • Platform Release set to 2.8.0
  • Triaged changed from No to Yes

#4 Updated by jcline@redhat.com over 5 years ago

  • Status changed from ASSIGNED to POST

#7 Updated by jcline@redhat.com over 5 years ago

  • Status changed from POST to MODIFIED

#8 Updated by dkliban@redhat.com about 5 years ago

  • Status changed from MODIFIED to 5

#9 Updated by dkliban@redhat.com about 5 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#10 Updated by bmbouter about 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF