Project

Profile

Help

Issue #1682

Puppet sync starts a new TCP connection for each package rather than pooling

Added by rbarlow almost 5 years ago. Updated over 1 year ago.

Status:
CLOSED - WONTFIX
Priority:
High
Assignee:
-
Start date:
Due date:
Estimated time:
Severity:
1. Low
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

If you start to sync Puppet Forge, you will notice that the log shows that it is creating a connection for each package that it downloads.

$ pulp-admin puppet repo create --repo-id=forge --feed=http://forge.puppetlabs.com
$ time pulp-admin puppet repo sync run --repo-id=forge

You will see many messages like this in the log:

Feb 16 18:04:54 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
Feb 16 18:04:55 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
Feb 16 18:04:56 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
Feb 16 18:04:57 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
Feb 16 18:04:58 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
Feb 16 18:04:59 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
…

I would expect to see it making new connections and then pooling them. 5 connections would look like this in the log:

Feb 16 18:04:54 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
Feb 16 18:04:55 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (2): forge.puppetlabs.com
Feb 16 18:04:56 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (3): forge.puppetlabs.com
Feb 16 18:04:57 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (4): forge.puppetlabs.com
Feb 16 18:04:58 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (5): forge.puppetlabs.com

If you watch the log during an RPM sync, you will see something similar to the above.

The user-facing issue here is that the sync takes a very long time. It is also not particularly friendly to Puppet forge.


Related issues

Has duplicate Puppet Support - Story #230: [RFE] The Puppet Forge downloader should download in parallelCLOSED - DUPLICATE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

History

#1 Updated by rbarlow almost 5 years ago

  • Has duplicate Story #230: [RFE] The Puppet Forge downloader should download in parallel added

#2 Updated by mhrivnak almost 5 years ago

  • Subject changed from Puppet sync starts a new HTTP connection for each package rather than pooling to Puppet sync starts a new TCP connection for each package rather than pooling
  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 1. Low
  • Triaged changed from No to Yes

#3 Updated by rbarlow almost 5 years ago

  • Description updated (diff)

#4 Updated by bmbouter over 1 year ago

  • Status changed from NEW to CLOSED - WONTFIX

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

#5 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF