Story #2772
Updated by bizhang over 7 years ago
When syncing with an upstream RedHat repo, the sync task complete successfully but no repository contents are downloaded. Pulp log show "failed with code 401: Authorization Required". Upstream repo: <code>https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x84_64/os</code> To accomplish this, we need to add the ability to yum_importer to specify headers with each request for content similar to the implementation of Issue #1282. For example you should be able to do the following. The URLs need to be changed, but this is the idea: <pre> pulp-admin rpm repo update --repo-id=redhatLocal --feed=https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/os --verify-feed-ssl false pulp-admin rpm repo sync run --repo-id=redhatLocal </pre> Without this feature you'll get errors on sync like: Pulp Log: <pre> pulp: requests.packages.urllib3.connectionpool:INFO: [7307f362] Starting new HTTPS connection (1): rhui2-cds01.us-west-2.aws.ce.redhat.com pulp: nectar.downloaders.threaded:INFO: [7307f362] Download succeeded: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x84_64/os. pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7307f362] Downloading metadata from https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os. pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): rhui2-cds02.us-west-2.aws.ce.redhat.com pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os/repodata/repomd.xml failed with code 401: Authorization Required pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7307f362] Downloading additional units. pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): rhui2-cds02.us-west-2.aws.ce.redhat.com pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os/.treeinfo failed with code 401: Authorization Required pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): rhui2-cds02.us-west-2.aws.ce.redhat.com pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os/treeinfo failed with code 401: Authorization Required pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7307f362] Sync complete. </pre> h2. Implementation: Nectar has two ways of taking a headers options: through a NectarConfig object or through a headers attribute on the request [0]. If the former, a change needs to be made in the nectar_config [1] to add the headers option. This option should take a header of the form "header-name=header-value" and can be parsed into a dict with the parse_header function [2] before being passed to the nectarconfig object. This would add a headers option for all importers. object If the latter there's a couple of requests that needs the options added: for downloading the rpm, srpm, and repomd [3][4][5]. This would only add the header option for the yum importer. The headers option being passed in can be obtained here [6] [3][4][5] [0] https://github.com/pulp/nectar/blob/master/nectar/downloaders/threaded.py#L253 [1] https://github.com/pulp/pulp/blob/master/server/pulp/plugins/util/nectar_config.py#L14 [2] https://github.com/pulp/pulp/blob/master/common/pulp/common/config.py#L146 [3] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/repomd/alternate.py#L70 [4] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/repomd/packages.py#L99 [5] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/repomd/metadata.py#L143 [6] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/sync.py#L86 h2. Testing: 1. Create a repo with the headers config set <code>pulp-admin rpm repo create --repo-id=zoo --relative-url=zoo </code> <code>curl --user admin:admin --data '{"importer_type_id":"yum_importer", "importer_id":"yum_importer", "importer_config": {"feed":"http://bizhang.fedorapeople.org/zoo/", "headers":"X-RHUI-ID=blah"}}' https://dev.example.com/pulp/api/v2/repositories/zoo/importers/</code> 2. start some network monitoring service <code>sudo tcpflow -p -c -i eth0 port 80</code> 3. Sync the repo and validate that the headers are in the network calls <code>pulp-admin rpm repo sync run --repo-id=zoo</code>
.