Project

Profile

Help

Issue #1524

closed

Story #1150: As a user, I can lazily fetch repositories

Refactor the pulp_streamer so that it can pool connections

Added by jcline@redhat.com about 8 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
2.8.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Currently the streamer makes use of Nectar's ``download_one`` method, so we are not utilizing urllib3's connection pool. Under load (even from five concurrent nectar threads during the lazy download task) HTTP 503 Service Unavailable are occasionally received. The current work-around is to just dispatch another task to try again later, but it would be better if we pooled the connections.

Actions #1

Updated by bmbouter about 8 years ago

We were forced to use download_one because it was the only single threaded option that Nectar provided. We cannot use a thread pool with twisted unless the twisted reactor is in control of the thread pool. Given that, is there still an improvement we can make here?

Actions #2

Updated by jcline@redhat.com about 8 years ago

I think it's definitely worth evaluating carefully. Without it, we will not perform well under moderate to heavy load, as I believe many clients will receive a healthy number of HTTP 503s.

Actions #3

Updated by mhrivnak about 8 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jcline@redhat.com
  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High
  • Platform Release set to 2.8.0
  • Triaged changed from No to Yes
Actions #4

Updated by jcline@redhat.com about 8 years ago

  • Status changed from ASSIGNED to POST

Added by Jeremy Cline about 8 years ago

Revision 6f917920 | View on GitHub

The streamer now pools connections.

closes #1524

Added by Jeremy Cline about 8 years ago

Revision 6f917920 | View on GitHub

The streamer now pools connections.

closes #1524

Added by Jeremy Cline about 8 years ago

Revision ae45a691 | View on GitHub

Create a proxy worker that pools TCP connections.

Actions #7

Updated by jcline@redhat.com about 8 years ago

  • Status changed from POST to MODIFIED
Actions #8

Updated by dkliban@redhat.com about 8 years ago

  • Status changed from MODIFIED to 5
Actions #9

Updated by dkliban@redhat.com almost 8 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #10

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF