Project

Profile

Help

Issue #1682

closed

Puppet sync starts a new TCP connection for each package rather than pooling

Added by rbarlow about 8 years ago. Updated almost 5 years ago.

Status:
CLOSED - WONTFIX
Priority:
High
Assignee:
-
Start date:
Due date:
Estimated time:
Severity:
1. Low
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

If you start to sync Puppet Forge, you will notice that the log shows that it is creating a connection for each package that it downloads.

$ pulp-admin puppet repo create --repo-id=forge --feed=http://forge.puppetlabs.com
$ time pulp-admin puppet repo sync run --repo-id=forge

You will see many messages like this in the log:

Feb 16 18:04:54 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
Feb 16 18:04:55 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
Feb 16 18:04:56 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
Feb 16 18:04:57 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
Feb 16 18:04:58 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
Feb 16 18:04:59 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
…

I would expect to see it making new connections and then pooling them. 5 connections would look like this in the log:

Feb 16 18:04:54 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
Feb 16 18:04:55 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (2): forge.puppetlabs.com
Feb 16 18:04:56 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (3): forge.puppetlabs.com
Feb 16 18:04:57 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (4): forge.puppetlabs.com
Feb 16 18:04:58 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (5): forge.puppetlabs.com

If you watch the log during an RPM sync, you will see something similar to the above.

The user-facing issue here is that the sync takes a very long time. It is also not particularly friendly to Puppet forge.


Related issues

Has duplicate Puppet Support - Story #230: [RFE] The Puppet Forge downloader should download in parallelCLOSED - DUPLICATE

Actions

Also available in: Atom PDF