Project

Profile

Help

Issue #1977

closed

basic auth in URL fails when using authenticated proxy

Added by mhrivnak almost 8 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
2.11.0
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 6
Quarter:

Description

If you perform a sync where all of these are true:

- the proxy requires auth
- the repo requires auth
- repo auth credentials are specified in the URL

You will see this error message:

RPM1004: Error retrieving metadata: Not found

If any of those three conditions are not true, the error will not occur.

Assuming you have an authenticated proxy configured in /etc/pulp/server/plugins.conf.d/yum_importer.json, this will work because credentials are not in the URL:

pulp-admin rpm repo create --repo-id=rplevka --feed=https://rplevka.fedorapeople.org/fakerepo01/ --basicauth-user=admin --basicauth-pass=changeme
pulp-admin rpm repo sync run --repo-id=rplevka

And this will fail:

pulp-admin rpm repo create --repo-id=rplevka --feed=https://admin:changeme@rplevka.fedorapeople.org/fakerepo01/
pulp-admin rpm repo sync run --repo-id=rplevka

Pulp previously did not have explicit options for setting the basic auth credentials, so users' only option was to put them in the URL. Even though that's deprecated per RFC and is generally not advisable, we should continue supporting those users.

This bug likely affects other plugins as well. I haven't tried reproducing, but at least one user reported the same problem doing a puppet sync.


Related issues

Related to Pulp - Issue #2520: credentials in feed URL are not url-unquotedCLOSED - CURRENTRELEASEdaviddavisActions
Actions #1

Updated by mhrivnak almost 8 years ago

  • Description updated (diff)
Actions #2

Updated by amacdona@redhat.com almost 8 years ago

  • Triaged changed from No to Yes
Actions #3

Updated by pcreech almost 8 years ago

  • Project changed from RPM Support to Nectar
  • Version - Nectar set to 1.5.1

After investigation, I discovered that this particular combination has not been a viable pathway for quite some time (Doesn't work in pulp 2.6 or 2.7, introduced here [0] with python-nectar-1.1.5-1)

When you specify a proxy_password and proxy_username in the importer config, it invokes this code pathway here:

https://github.com/pulp/nectar/blob/master/nectar/downloaders/threaded.py#L412

Which replaces the authentication provider with the HTTPProxyAuth or HTTPBasicWithProxyAuth provider (for authenticated proxies). When this is done with requests, it breaks requests ability to authenticate the requested URL with HTTPBasicAuth when credentials are provided in the url (HTTPBasicAuth also intercepts the user:pass@host information)

With provided basic_auth_username and basic_auth_password credentials, it utilizes HTTPBasicWithProxyAuth instead, which is why the first example works.

There are two functioning workarounds, whch are:

  1. Use the first form provided in this issue
  2. Instead of providing proxy_username and proxy_password in the yum_importer.json file, provide the credentials in the proxy host i.e.
    "proxy_host": "http://user:pass@host"

To fix the issue in the codebase, we should find a way to intercept credentials before

https://github.com/pulp/nectar/blob/python-nectar-1.5.1-1/nectar/downloaders/threaded.py#L369

[0] https://github.com/pulp/nectar/commit/0063b1b6d3494fbcf400fdad653c34f73fa5c879#diff-609b5250f930300d62fd3da027ef9608R312

Actions #4

Updated by amacdona@redhat.com over 7 years ago

  • Sprint Candidate changed from No to Yes
Actions #5

Updated by mhrivnak over 7 years ago

  • Sprint/Milestone set to 24
Actions #6

Updated by amacdona@redhat.com over 7 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to amacdona@redhat.com
Actions #7

Updated by amacdona@redhat.com over 7 years ago

  • Project changed from Nectar to Pulp
  • Status changed from ASSIGNED to POST

I determined that this is not possible to fix in nectar. The reason is that at the time that a threaded downloader object is created, nectar is not aware of the urls, which are only passed in later as request objects. Instead, the proposed change moves the basic auth information to the importer config before creating the downloader.

https://github.com/pulp/pulp/pull/2697

Added by Austin Macdonald over 7 years ago

Revision 206cf5fd | View on GitHub

Pull basic auth info out of feed url

Previous versions of Pulp allowed users to include their basic auth information in the url. This is obviously less ideal than passing this in the config, but in order to remain backwards compatibile, this removes the auth info from the url and adds it to the importer config. For importer configurations that were created before this change, we pass the configuration through the same cleaner function as we do during importer creation, so a temporary importer configuration includes the basic auth information.

closes #1977

Added by Austin Macdonald over 7 years ago

Revision 206cf5fd | View on GitHub

Pull basic auth info out of feed url

Previous versions of Pulp allowed users to include their basic auth information in the url. This is obviously less ideal than passing this in the config, but in order to remain backwards compatibile, this removes the auth info from the url and adds it to the importer config. For importer configurations that were created before this change, we pass the configuration through the same cleaner function as we do during importer creation, so a temporary importer configuration includes the basic auth information.

closes #1977

Actions #8

Updated by Anonymous over 7 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #9

Updated by semyers over 7 years ago

  • Platform Release set to 2.11.0
Actions #10

Updated by semyers over 7 years ago

  • Status changed from MODIFIED to 5
Actions #11

Updated by pthomas@redhat.com over 7 years ago

  • Status changed from 5 to 6

verified

pulp-admin rpm repo create  --feed http://file.rdu.redhat.com/~cduryee/protected/ --basicauth-user testuser --basicauth-pass testpass    --repo-id proxy1 --proxy-host="http://ginger.lab.eng.rdu2.redhat.com" --proxy-user="admin" --proxy-port 3128 --proxy-pass redhat
Successfully created repository [proxy1]

[root@cloud-qe-22 ~]# pulp-admin rpm repo sync run --repo-id proxy1
+----------------------------------------------------------------------+
                   Synchronizing Repository [proxy1]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[|]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       2/2 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Cleaning duplicate packages...
[-]
... completed

Task Succeeded

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
2 of 2 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[-]
... completed

Publishing Comps file
[-]
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Generating HTML files
... skipped

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed

Task Succeeded
Actions #13

Updated by pcreech over 7 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions #14

Updated by mhrivnak about 7 years ago

  • Related to Issue #2520: credentials in feed URL are not url-unquoted added
Actions #16

Updated by bmbouter about 6 years ago

  • Sprint set to Sprint 6
Actions #17

Updated by bmbouter about 6 years ago

  • Sprint/Milestone deleted (24)
Actions #18

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF