Issue #1682
closed
Puppet sync starts a new TCP connection for each package rather than pooling
Description
If you start to sync Puppet Forge, you will notice that the log shows that it is creating a connection for each package that it downloads.
$ pulp-admin puppet repo create --repo-id=forge --feed=http://forge.puppetlabs.com
$ time pulp-admin puppet repo sync run --repo-id=forge
You will see many messages like this in the log:
Feb 16 18:04:54 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
Feb 16 18:04:55 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
Feb 16 18:04:56 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
Feb 16 18:04:57 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
Feb 16 18:04:58 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
Feb 16 18:04:59 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
…
I would expect to see it making new connections and then pooling them. 5 connections would look like this in the log:
Feb 16 18:04:54 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): forge.puppetlabs.com
Feb 16 18:04:55 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (2): forge.puppetlabs.com
Feb 16 18:04:56 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (3): forge.puppetlabs.com
Feb 16 18:04:57 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (4): forge.puppetlabs.com
Feb 16 18:04:58 dev pulp[4674]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (5): forge.puppetlabs.com
If you watch the log during an RPM sync, you will see something similar to the above.
The user-facing issue here is that the sync takes a very long time. It is also not particularly friendly to Puppet forge.
Related issues
Updated by rbarlow over 7 years ago
- Has duplicate Story #230: [RFE] The Puppet Forge downloader should download in parallel added
Updated by mhrivnak over 7 years ago
- Subject changed from Puppet sync starts a new HTTP connection for each package rather than pooling to Puppet sync starts a new TCP connection for each package rather than pooling
- Priority changed from Normal to High
- Severity changed from 2. Medium to 1. Low
- Triaged changed from No to Yes
Updated by bmbouter over 4 years ago
- Status changed from NEW to CLOSED - WONTFIX
Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.
.