Issue #1762
closedUnable to sync repositories while using authenticated HTTP proxy - RPM1004: Error retrieving metadata: Not found
Description
Description of problem:
Getting "RPM1004: Error retrieving metadata: Not found" while trying to sync a repository on a satellite configured to use squid HTTP proxy.
the message seen from satellite is
"RPM1004: Error retrieving metadata: Not found"
while squid/access.log shows:
1457543658.360 0 <ip> TCP_DENIED/407 3605 CONNECT cdn.redhat.com:443 - HIER_NONE/- text/html
the /etc/pulp/server/plugins.conf.d/yum_importer.json:
{
"proxy_host": "http://my.squid_proxy.server",
"proxy_port": 3128,
"proxy_username": "login",
"proxy_password": "pass"
}
- after removing the proxy configuration from yum_importer.json, the sync works fine.
Version-Release number of selected component (if applicable):
- rpm -qa pulp
pulp-rpm-plugins-2.8.0.1-1.beta.el6sat.noarch
python-pulp-puppet-common-2.8.0.1-2.beta.el6sat.noarch
python-pulp-oid_validation-2.8.0.1-0.7.beta.el6sat.noarch
pulp-nodes-parent-2.8.0.1-0.7.beta.el6sat.noarch
python-pulp-docker-common-2.0.0.1-1.beta.el6sat.noarch
pulp-selinux-2.8.0.1-0.7.beta.el6sat.noarch
python-pulp-bindings-2.8.0.1-0.7.beta.el6sat.noarch
pulp-nodes-common-2.8.0.1-0.7.beta.el6sat.noarch
python-pulp-streamer-2.8.0.1-0.7.beta.el6sat.noarch
python-pulp-common-2.8.0.1-0.7.beta.el6sat.noarch
python-pulp-rpm-common-2.8.0.1-1.beta.el6sat.noarch
pulp-client-1.0-1.noarch
rubygem-smart_proxy_pulp-1.2.0-1.el6sat.noarch
python-pulp-repoauth-2.8.0.1-0.7.beta.el6sat.noarch
pulp-katello-1.0-3.el6sat.noarch
pulp-puppet-plugins-2.8.0.1-2.beta.el6sat.noarch
pulp-puppet-tools-2.8.0.1-2.beta.el6sat.noarch
pulp-docker-plugins-2.0.0.1-1.beta.el6sat.noarch
pulp-server-2.8.0.1-0.7.beta.el6sat.noarch
How reproducible:
- always with authed http proxy
Steps to Reproduce:
1. setup a squid http proxy server with http authentication
2. configure satellite to use the proxy server
3. try to sync some RH or custom repo
Actual results:
RPM1004: Error retrieving metadata: Not found
Expected results:
sync works
Additional info:
I was able to reproduce the err 407 using a simple netcat session:
$ nc my_proxy 3128
CONNECT cdn.redhat.com:443
(complete output: http://pastebin.test.redhat.com/355348)
When i specify the HTTP/1.1 at the end of the first request line, squid proxy waits for me to send the headers (including the auth header):
$ nc my_proxy 3128
CONNECT cdn.redhat.com:443 HTTP/1.1
Proxy-Authorization: Basic HLbtd5HnrVdfHWA7
HTTP/1.1 200 Connection established
Updated by pthomas@redhat.com over 8 years ago
I used the authenticated proxy that corey gave me
pulp-admin rpm repo create --repo-id rhel6 --feed https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.7/x86_64/kickstart/ --feed-ca-cert cdn.crt --feed-cert feed.crt --feed-key cdn/feed.key
[root@celeno ~]# pulp-admin rpm repo update --repo-id rhel6 --proxy-host http://my.squid.proxy.com --proxy-port 8889 --proxy-user "user" --proxy-pass "pass"
Repository [rhel6] successfully updated
[root@celeno ~]#
[root@celeno ~]#
[root@celeno ~]# pulp-admin rpm repo sync run --repo-id rhel6
--------------------------------------------------------------------
Synchronizing Repository [rhel6]
--------------------------------------------------------------------
This command may be exited via ctrl+c without affecting the request.
Task Failed
Error retrieving metadata: Not found
Mar 9 16:02:38 celeno pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[79e2f9d9-9e7f-4fe0-a57d-b7c0a9dc8c1c]
Mar 9 16:02:38 celeno pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[abf6a010-0f7d-49b0-9866-5c2379f576d2]
Mar 9 16:02:38 celeno pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[ceb72371-b5bf-4559-b90e-7d4780f0dba4]
Mar 9 16:02:38 celeno pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[79e2f9d9-9e7f-4fe0-a57d-b7c0a9dc8c1c] succeeded in 0.064851136005s: None
Mar 9 16:02:38 celeno pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.7/x86_64/kickstart/.
Mar 9 16:02:38 celeno pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
Mar 9 16:02:38 celeno pulp: requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=4, connect=5, read=5, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required',))': /content/dist/rhel/rhui/server/6/6.7/x86_64/kickstart/repodata/repomd.xml
Mar 9 16:02:38 celeno pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): cdn.redhat.com
Mar 9 16:02:40 celeno pulp: requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=3, connect=5, read=5, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required',))': /content/dist/rhel/rhui/server/6/6.7/x86_64/kickstart/repodata/repomd.xml
Mar 9 16:02:40 celeno pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): cdn.redhat.com
Mar 9 16:02:44 celeno pulp: requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=2, connect=5, read=5, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required',))': /content/dist/rhel/rhui/server/6/6.7/x86_64/kickstart/repodata/repomd.xml
Mar 9 16:02:44 celeno pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (4): cdn.redhat.com
Mar 9 16:02:53 celeno pulp: requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=1, connect=5, read=5, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required',))': /content/dist/rhel/rhui/server/6/6.7/x86_64/kickstart/repodata/repomd.xml
Mar 9 16:02:53 celeno pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (5): cdn.redhat.co
Updated by pthomas@redhat.com over 8 years ago
Proxy without authentication works fine.
[root@celeno ~]# pulp-admin rpm repo update --repo-id zoo --proxy-host http://myproxy.com --proxy-port 3128 --proxy-user "" --proxy-pass ""
Repository [zoo] successfully updated
[root@celeno ~]# pulp-admin rpm repo sync run --repo-id zoo
--------------------------------------------------------------------
Synchronizing Repository [zoo]
--------------------------------------------------------------------
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[|]
... completed
Downloading repository content...
[-]
[==================================================] 100%
RPMs: 0/0 items
Delta RPMs: 0/0 items
... completed
Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed
Importing errata...
[-]
... completed
Importing package groups/categories...
[-]
... completed
Cleaning duplicate packages...
[-]
... completed
Task Succeeded
Copying files
[-]
... completed
Initializing repo metadata
[-]
... completed
Publishing Distribution files
[-]
... completed
Publishing RPMs
[-]
... completed
Publishing Delta RPMs
... skipped
Publishing Errata
[==================================================] 100%
4 of 4 items
... completed
Publishing Comps file
[==================================================] 100%
3 of 3 items
... completed
Publishing Metadata.
[-]
... completed
Closing repo metadata
[-]
... completed
Generating sqlite files
... skipped
Publishing files to web
[-]
... completed
Writing Listings File
[-]
... completed
Task Succeeded
Updated by mhrivnak over 8 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to pcreech
- Priority changed from Normal to High
We started using a fancy new feature of our download library that auto-detects what kind of proxy auth to use. That seems to be failing.
Patrick is the expert in this area and will continue to dig.
I reproduced locally, and I can verify that auth credentials are getting passed into the GuessProxyAuth class.
Using wireshark, it appears that the request is being made that has no credentials, and a 407 response happens with this important header:
Proxy-Authenticate: Basic realm="Squid proxy-caching web server"
I think some part of the requests library should then make a new request with basic auth added, but that isn't happening.
Added by pcreech over 8 years ago
Updated by pcreech over 8 years ago
- Status changed from ASSIGNED to MODIFIED
- % Done changed from 0 to 100
Applied in changeset nectar:4658d63dd3cc992e152e1de7c7b6abe5040a62a7.
Added by ipanova@redhat.com over 8 years ago
Revision e7c4abef | View on GitHub
Removing python-requests-toolbelt from deps.
Due to the reverted changes in digest proxy in python-nectar, we don't need to carry python-requests-toolbelt as a dependency.
Updated by semyers over 8 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
- Platform Release set to 2.8.0
This was included in the nectar 1.5.1-1 release, which was in turn included with pulp 2.8.0.
Revert changes for digest proxy
Due to interesting circumstances surrounding the way httplib opens proxy connections, our method for guessing proxy authentication appears to not work for certain instances. This patch reverts those changes till we can find a more robust way to handle the situation.
refs: #469 https://pulp.plan.io/issues/469
closes: #1762 https://pulp.plan.io/issues/1762