Project

Profile

Help

Story #2772

Updated by bizhang almost 7 years ago

When syncing with an upstream RedHat repo, the sync task complete successfully but no repository contents are downloaded. Pulp log show "failed with code 401: Authorization Required". 

 Upstream repo: <code>https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x84_64/os</code> 

 To accomplish this, we need to add the ability to yum_importer to specify headers with each request for content similar to the implementation of Issue #1282. 

 For example you should be able to do the following. The URLs need to be changed, but this is the idea: 

 <pre> 
 pulp-admin rpm repo update --repo-id=redhatLocal --feed=https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/os --verify-feed-ssl false 
 pulp-admin rpm repo sync run --repo-id=redhatLocal 
 </pre> 

 Without this feature you'll get errors on sync like: 

 Pulp Log:  
 <pre> 
 pulp: requests.packages.urllib3.connectionpool:INFO: [7307f362] Starting new HTTPS connection (1): rhui2-cds01.us-west-2.aws.ce.redhat.com 
 pulp: nectar.downloaders.threaded:INFO: [7307f362] Download succeeded: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x84_64/os. 
 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7307f362] Downloading metadata from https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os. 
 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): rhui2-cds02.us-west-2.aws.ce.redhat.com 
 pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os/repodata/repomd.xml failed with code 401: Authorization Required 
 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7307f362] Downloading additional units. 
 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): rhui2-cds02.us-west-2.aws.ce.redhat.com 
 pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os/.treeinfo failed with code 401: Authorization Required 
 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): rhui2-cds02.us-west-2.aws.ce.redhat.com 
 pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os/treeinfo failed with code 401: Authorization Required 
 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7307f362] Sync complete. 
 </pre> 


 h2. Implementation: 

 Nectar has two ways of taking a headers options: through a NectarConfig object or through a headers attribute on the request [0].  

 If the former, a change needs to be made in the nectar_config [1] to add the headers option. This option should take a header of the form "header-name=header-value" and can be parsed into a dict with the parse_header function [2] before being passed to the nectarconfig object. This would add a headers option for all importers. object 

 If the latter there's a couple of requests that needs the options added: for downloading the rpm, srpm, and repomd [3][4][5]. This would only add the header option for the yum importer. The headers option being passed in can be obtained here [6] [3][4][5] 

 [0] https://github.com/pulp/nectar/blob/master/nectar/downloaders/threaded.py#L253 
 [1] https://github.com/pulp/pulp/blob/master/server/pulp/plugins/util/nectar_config.py#L14 
 [2] https://github.com/pulp/pulp/blob/master/common/pulp/common/config.py#L146 

 [3] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/repomd/alternate.py#L70 
 [4] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/repomd/packages.py#L99 
 [5] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/repomd/metadata.py#L143 
 [6] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/sync.py#L86 

 h2. Testing: 

 1. Create a repo with the headers config set 
 <code>pulp-admin rpm repo create --repo-id=zoo --relative-url=zoo </code> 
 <code>curl --user admin:admin --data '{"importer_type_id":"yum_importer", "importer_id":"yum_importer", "importer_config": {"feed":"http://bizhang.fedorapeople.org/zoo/", "headers":"X-RHUI-ID=blah"}}' https://dev.example.com/pulp/api/v2/repositories/zoo/importers/</code> 

 2. start some network monitoring service 
 <code>sudo tcpflow -p -c -i eth0 port 80</code> 

 3. Sync the repo and validate that the headers are in the network calls 
 <code>pulp-admin rpm repo sync run --repo-id=zoo</code> 

Back