Pulp

When syncing with an upstream RedHat repo, the sync task complete successfully but no repository contents are downloaded. Pulp log show "failed with code 401: Authorization Required".

Upstream repo: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x84_64/os

To accomplish this, we need to add the ability to yum_importer to specify headers with each request for content similar to the implementation of Issue #1282.

For example you should be able to do the following. The URLs need to be changed, but this is the idea:

pulp-admin rpm repo update --repo-id=redhatLocal --feed=https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/os --verify-feed-ssl false
pulp-admin rpm repo sync run --repo-id=redhatLocal

Without this feature you'll get errors on sync like:

Pulp Log:

pulp: requests.packages.urllib3.connectionpool:INFO: [7307f362] Starting new HTTPS connection (1): rhui2-cds01.us-west-2.aws.ce.redhat.com
pulp: nectar.downloaders.threaded:INFO: [7307f362] Download succeeded: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x84_64/os.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7307f362] Downloading metadata from https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os.
pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): rhui2-cds02.us-west-2.aws.ce.redhat.com
pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os/repodata/repomd.xml failed with code 401: Authorization Required
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7307f362] Downloading additional units.
pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): rhui2-cds02.us-west-2.aws.ce.redhat.com
pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os/.treeinfo failed with code 401: Authorization Required
pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): rhui2-cds02.us-west-2.aws.ce.redhat.com
pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os/treeinfo failed with code 401: Authorization Required
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7307f362] Sync complete.

Implementation:¶

Nectar has two ways of taking a headers options: through a NectarConfig object or through a headers attribute on the request [0].

If the former, a change needs to be made in the nectar_config [1] to add the headers option. This option should take a header of the form "header-name=header-value" and can be parsed into a dict with the parse_header function [2] before being passed to the nectarconfig object. This would add a headers option for all importers.

If the latter there's a couple of requests that needs the options added: for downloading the rpm, srpm, and repomd [3][4][5]. This would only add the header option for the yum importer. The headers option being passed in can be obtained here [6]

[0] https://github.com/pulp/nectar/blob/master/nectar/downloaders/threaded.py#L253
[1] https://github.com/pulp/pulp/blob/master/server/pulp/plugins/util/nectar_config.py#L14
[2] https://github.com/pulp/pulp/blob/master/common/pulp/common/config.py#L146

[3] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/repomd/alternate.py#L70
[4] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/repomd/packages.py#L99
[5] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/repomd/metadata.py#L143
[6] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/sync.py#L86

Testing:¶

1. Create a repo with the headers config set
pulp-admin rpm repo create --repo-id=zoo --relative-url=zoo
curl --user admin:admin --data '{"importer_type_id":"yum_importer", "importer_id":"yum_importer", "importer_config": {"feed":"http://bizhang.fedorapeople.org/zoo/", "headers":"X-RHUI-ID=blah"}}' https://dev.example.com/pulp/api/v2/repositories/zoo/importers/

2. start some network monitoring service
sudo tcpflow -p -c -i eth0 port 80

3. Sync the repo and validate that the headers are in the network calls
pulp-admin rpm repo sync run --repo-id=zoo