Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #1762

closed

Unable to sync repositories while using authenticated HTTP proxy - RPM1004: Error retrieving metadata: Not found

Added by mmccune@redhat.com over 8 years ago. Updated over 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
pcreech
Start date:
Due date:
Estimated time:
Severity:
4. Urgent
Version - Nectar:
Platform Release:
2.8.0
Target Release - Nectar:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Description of problem:
Getting "RPM1004: Error retrieving metadata: Not found" while trying to sync a repository on a satellite configured to use squid HTTP proxy.

the message seen from satellite is
"RPM1004: Error retrieving metadata: Not found"

while squid/access.log shows:
1457543658.360 0 <ip> TCP_DENIED/407 3605 CONNECT cdn.redhat.com:443 - HIER_NONE/- text/html

the /etc/pulp/server/plugins.conf.d/yum_importer.json:
{
"proxy_host": "http://my.squid_proxy.server",
"proxy_port": 3128,
"proxy_username": "login",
"proxy_password": "pass"
}

- after removing the proxy configuration from yum_importer.json, the sync works fine.

Version-Release number of selected component (if applicable):

  1. rpm -qa pulp
    pulp-rpm-plugins-2.8.0.1-1.beta.el6sat.noarch
    python-pulp-puppet-common-2.8.0.1-2.beta.el6sat.noarch
    python-pulp-oid_validation-2.8.0.1-0.7.beta.el6sat.noarch
    pulp-nodes-parent-2.8.0.1-0.7.beta.el6sat.noarch
    python-pulp-docker-common-2.0.0.1-1.beta.el6sat.noarch
    pulp-selinux-2.8.0.1-0.7.beta.el6sat.noarch
    python-pulp-bindings-2.8.0.1-0.7.beta.el6sat.noarch
    pulp-nodes-common-2.8.0.1-0.7.beta.el6sat.noarch
    python-pulp-streamer-2.8.0.1-0.7.beta.el6sat.noarch
    python-pulp-common-2.8.0.1-0.7.beta.el6sat.noarch
    python-pulp-rpm-common-2.8.0.1-1.beta.el6sat.noarch
    pulp-client-1.0-1.noarch
    rubygem-smart_proxy_pulp-1.2.0-1.el6sat.noarch
    python-pulp-repoauth-2.8.0.1-0.7.beta.el6sat.noarch
    pulp-katello-1.0-3.el6sat.noarch
    pulp-puppet-plugins-2.8.0.1-2.beta.el6sat.noarch
    pulp-puppet-tools-2.8.0.1-2.beta.el6sat.noarch
    pulp-docker-plugins-2.0.0.1-1.beta.el6sat.noarch
    pulp-server-2.8.0.1-0.7.beta.el6sat.noarch

How reproducible:
- always with authed http proxy

Steps to Reproduce:
1. setup a squid http proxy server with http authentication
2. configure satellite to use the proxy server
3. try to sync some RH or custom repo

Actual results:
RPM1004: Error retrieving metadata: Not found

Expected results:
sync works

Additional info:

I was able to reproduce the err 407 using a simple netcat session:
$ nc my_proxy 3128
CONNECT cdn.redhat.com:443

(complete output: http://pastebin.test.redhat.com/355348)

When i specify the HTTP/1.1 at the end of the first request line, squid proxy waits for me to send the headers (including the auth header):

$ nc my_proxy 3128
CONNECT cdn.redhat.com:443 HTTP/1.1
Proxy-Authorization: Basic HLbtd5HnrVdfHWA7

HTTP/1.1 200 Connection established

Actions
Copy link
 #1

Updated by pthomas@redhat.com over 8 years ago

I used the authenticated proxy that corey gave me

pulp-admin rpm repo create --repo-id rhel6 --feed https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.7/x86_64/kickstart/ --feed-ca-cert cdn.crt --feed-cert feed.crt --feed-key cdn/feed.key

[root@celeno ~]# pulp-admin rpm repo update --repo-id rhel6 --proxy-host http://my.squid.proxy.com --proxy-port 8889 --proxy-user "user" --proxy-pass "pass"
Repository [rhel6] successfully updated

[root@celeno ~]#
[root@celeno ~]#
[root@celeno ~]# pulp-admin rpm repo sync run --repo-id rhel6
--------------------------------------------------------------------
Synchronizing Repository [rhel6]
--------------------------------------------------------------------

This command may be exited via ctrl+c without affecting the request.

Task Failed

Error retrieving metadata: Not found

Mar  9 16:02:38 celeno pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[79e2f9d9-9e7f-4fe0-a57d-b7c0a9dc8c1c]
Mar  9 16:02:38 celeno pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[abf6a010-0f7d-49b0-9866-5c2379f576d2]
Mar  9 16:02:38 celeno pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[ceb72371-b5bf-4559-b90e-7d4780f0dba4]
Mar  9 16:02:38 celeno pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[79e2f9d9-9e7f-4fe0-a57d-b7c0a9dc8c1c] succeeded in 0.064851136005s: None
Mar  9 16:02:38 celeno pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.7/x86_64/kickstart/.
Mar  9 16:02:38 celeno pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
Mar  9 16:02:38 celeno pulp: requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=4, connect=5, read=5, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required',))': /content/dist/rhel/rhui/server/6/6.7/x86_64/kickstart/repodata/repomd.xml
Mar  9 16:02:38 celeno pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): cdn.redhat.com
Mar  9 16:02:40 celeno pulp: requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=3, connect=5, read=5, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required',))': /content/dist/rhel/rhui/server/6/6.7/x86_64/kickstart/repodata/repomd.xml
Mar  9 16:02:40 celeno pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): cdn.redhat.com
Mar  9 16:02:44 celeno pulp: requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=2, connect=5, read=5, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required',))': /content/dist/rhel/rhui/server/6/6.7/x86_64/kickstart/repodata/repomd.xml
Mar  9 16:02:44 celeno pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (4): cdn.redhat.com
Mar  9 16:02:53 celeno pulp: requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=1, connect=5, read=5, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required',))': /content/dist/rhel/rhui/server/6/6.7/x86_64/kickstart/repodata/repomd.xml
Mar  9 16:02:53 celeno pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (5): cdn.redhat.co
Actions
Copy link
 #2

Updated by pthomas@redhat.com over 8 years ago

Proxy without authentication works fine.

[root@celeno ~]# pulp-admin rpm repo update --repo-id zoo --proxy-host http://myproxy.com --proxy-port 3128 --proxy-user "" --proxy-pass ""
Repository [zoo] successfully updated

[root@celeno ~]# pulp-admin rpm repo sync run --repo-id zoo
--------------------------------------------------------------------
Synchronizing Repository [zoo]
--------------------------------------------------------------------

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[|]
... completed

Downloading repository content...
[-]
[==================================================] 100%
RPMs: 0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Cleaning duplicate packages...
[-]
... completed

Task Succeeded

Copying files
[-]
... completed

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[-]
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
4 of 4 items
... completed

Publishing Comps file
[==================================================] 100%
3 of 3 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

Actions
Copy link
 #3

Updated by mhrivnak over 8 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to pcreech
  • Priority changed from Normal to High

We started using a fancy new feature of our download library that auto-detects what kind of proxy auth to use. That seems to be failing.

Patrick is the expert in this area and will continue to dig.

I reproduced locally, and I can verify that auth credentials are getting passed into the GuessProxyAuth class.

Using wireshark, it appears that the request is being made that has no credentials, and a 407 response happens with this important header:

Proxy-Authenticate: Basic realm="Squid proxy-caching web server"

I think some part of the requests library should then make a new request with basic auth added, but that isn't happening.

Added by pcreech over 8 years ago

Revision 4658d63d | View on GitHub

Revert changes for digest proxy

Due to interesting circumstances surrounding the way httplib opens proxy connections, our method for guessing proxy authentication appears to not work for certain instances. This patch reverts those changes till we can find a more robust way to handle the situation.

refs: #469 https://pulp.plan.io/issues/469

closes: #1762 https://pulp.plan.io/issues/1762

Actions
Copy link
 #4

Updated by pcreech over 8 years ago

  • Status changed from ASSIGNED to MODIFIED
  • % Done changed from 0 to 100
Actions
Copy link
 #5

Updated by pcreech over 8 years ago

  • Triaged changed from No to Yes
Actions
Copy link
 #6

Updated by mhrivnak over 8 years ago

  • Project changed from RPM Support to Nectar

Added by ipanova@redhat.com over 8 years ago

Revision e7c4abef | View on GitHub

Removing python-requests-toolbelt from deps.

Due to the reverted changes in digest proxy in python-nectar, we don't need to carry python-requests-toolbelt as a dependency.

refs: #1762 https://pulp.plan.io/issues/1762

Actions
Copy link
 #8

Updated by semyers over 8 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
  • Platform Release set to 2.8.0

This was included in the nectar 1.5.1-1 release, which was in turn included with pulp 2.8.0.

Actions
Copy link
 #9

Updated by bmbouter over 5 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF