Story #2772
Updated by bizhang over 7 years ago
When syncing with an upstream RedHat repo, the sync task complete successfully but no repository contents are downloaded. Pulp log show "failed with code 401: Authorization Required".
Upstream repo: <code>https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x84_64/os</code>
To accomplish this, we need to add the ability to yum_importer to specify headers with each request for content similar to the implementation of Issue #1282.
For example you should be able to do the following. The URLs need to be changed, but this is the idea:
<pre>
pulp-admin rpm repo update --repo-id=redhatLocal --feed=https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/os --verify-feed-ssl false
pulp-admin rpm repo sync run --repo-id=redhatLocal
</pre>
Without this feature you'll get errors on sync like:
Pulp Log:
<pre>
pulp: requests.packages.urllib3.connectionpool:INFO: [7307f362] Starting new HTTPS connection (1): rhui2-cds01.us-west-2.aws.ce.redhat.com
pulp: nectar.downloaders.threaded:INFO: [7307f362] Download succeeded: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x84_64/os.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7307f362] Downloading metadata from https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os.
pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): rhui2-cds02.us-west-2.aws.ce.redhat.com
pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os/repodata/repomd.xml failed with code 401: Authorization Required
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7307f362] Downloading additional units.
pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): rhui2-cds02.us-west-2.aws.ce.redhat.com
pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os/.treeinfo failed with code 401: Authorization Required
pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): rhui2-cds02.us-west-2.aws.ce.redhat.com
pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os/treeinfo failed with code 401: Authorization Required
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7307f362] Sync complete.
</pre>
h2. Implementation:
Nectar has two ways of taking a headers options: through a NectarConfig object or through a headers attribute on the request [0].
If the former, a change needs to be made in the nectar_config [1] to add the headers option. This option should take a header of the form "header-name=header-value" and can be parsed into a dict with the parse_header function [2] before being passed to the nectarconfig object
If the latter there's a couple of requests that needs the options added: for downloading the rpm, srpm, and repomd [3][4][5]
[0] https://github.com/pulp/nectar/blob/master/nectar/downloaders/threaded.py#L253
[1] https://github.com/pulp/pulp/blob/master/server/pulp/plugins/util/nectar_config.py#L14
[2] https://github.com/pulp/pulp/blob/master/common/pulp/common/config.py#L146
[3] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/repomd/alternate.py#L70
[4] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/repomd/packages.py#L99
[5] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/repomd/metadata.py#L143
h2. Testing:
1. Create a repo with the headers config set
<code>pulp-admin rpm repo create --repo-id=zoo --relative-url=zoo </code>
<code>curl --user admin:admin --data '{"importer_type_id":"yum_importer", "importer_id":"yum_importer", "importer_config": {"feed":"http://bizhang.fedorapeople.org/zoo/", "headers":"X-RHUI-ID=blah"}}' https://dev.example.com/pulp/api/v2/repositories/zoo/importers/</code>
2. start some network monitoring service
<code>sudo tcpflow -p -c -i eth0 port 80</code>
3. Sync the repo and validate that the headers are in the network calls
<code>pulp-admin rpm repo sync run --repo-id=zoo</code>
.