Project

Profile

Help

Story #2772

closed

As a user, I can sync from a RedHat mirror provided by and hosted in AWS

Added by shaimon almost 7 years ago. Updated almost 5 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

When syncing with an upstream RedHat repo, the sync task complete successfully but no repository contents are downloaded. Pulp log show "failed with code 401: Authorization Required".

Upstream repo: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x84_64/os

To accomplish this, we need to add the ability to yum_importer to specify headers with each request for content similar to the implementation of Issue #1282.

For example you should be able to do the following. The URLs need to be changed, but this is the idea:

pulp-admin rpm repo update --repo-id=redhatLocal --feed=https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/os --verify-feed-ssl false
pulp-admin rpm repo sync run --repo-id=redhatLocal

Without this feature you'll get errors on sync like:

Pulp Log:

pulp: requests.packages.urllib3.connectionpool:INFO: [7307f362] Starting new HTTPS connection (1): rhui2-cds01.us-west-2.aws.ce.redhat.com
pulp: nectar.downloaders.threaded:INFO: [7307f362] Download succeeded: https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x84_64/os.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7307f362] Downloading metadata from https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os.
pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): rhui2-cds02.us-west-2.aws.ce.redhat.com
pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os/repodata/repomd.xml failed with code 401: Authorization Required
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7307f362] Downloading additional units.
pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): rhui2-cds02.us-west-2.aws.ce.redhat.com
pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os/.treeinfo failed with code 401: Authorization Required
pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): rhui2-cds02.us-west-2.aws.ce.redhat.com
pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os/treeinfo failed with code 401: Authorization Required
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7307f362] Sync complete.

Implementation:

Nectar has two ways of taking a headers options: through a NectarConfig object or through a headers attribute on the request [0].

If the former, a change needs to be made in the nectar_config [1] to add the headers option. This option should take a header of the form "header-name=header-value" and can be parsed into a dict with the parse_header function [2] before being passed to the nectarconfig object. This would add a headers option for all importers.

If the latter there's a couple of requests that needs the options added: for downloading the rpm, srpm, and repomd [3][4][5]. This would only add the header option for the yum importer. The headers option being passed in can be obtained here [6]

[0] https://github.com/pulp/nectar/blob/master/nectar/downloaders/threaded.py#L253
[1] https://github.com/pulp/pulp/blob/master/server/pulp/plugins/util/nectar_config.py#L14
[2] https://github.com/pulp/pulp/blob/master/common/pulp/common/config.py#L146

[3] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/repomd/alternate.py#L70
[4] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/repomd/packages.py#L99
[5] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/repomd/metadata.py#L143
[6] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/sync.py#L86

Testing:

1. Create a repo with the headers config set
pulp-admin rpm repo create --repo-id=zoo --relative-url=zoo
curl --user admin:admin --data '{"importer_type_id":"yum_importer", "importer_id":"yum_importer", "importer_config": {"feed":"http://bizhang.fedorapeople.org/zoo/", "headers":"X-RHUI-ID=blah"}}' https://dev.example.com/pulp/api/v2/repositories/zoo/importers/

2. start some network monitoring service
sudo tcpflow -p -c -i eth0 port 80

3. Sync the repo and validate that the headers are in the network calls
pulp-admin rpm repo sync run --repo-id=zoo


Related issues

Related to Pulp - Story #1282: As an EC2 user, I would like to set up a RHUI as an alternate content sourceCLOSED - CURRENTRELEASE

Actions
Actions #1

Updated by ipanova@redhat.com almost 7 years ago

Do you provide any kind of cert in order to sync this repo?

$ wget https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os/repodata/repomd.xml --no-check-certificate
--2017-05-22 18:54:28--  https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist/rhel/rhui/server/7/7Server/x84_64/os/repodata/repomd.xml
Resolving rhui2-cds02.us-west-2.aws.ce.redhat.com (rhui2-cds02.us-west-2.aws.ce.redhat.com)... 50.112.120.25
Connecting to rhui2-cds02.us-west-2.aws.ce.redhat.com (rhui2-cds02.us-west-2.aws.ce.redhat.com)|50.112.120.25|:443... connected.
WARNING: cannot verify rhui2-cds02.us-west-2.aws.ce.redhat.com's certificate, issued by ‘emailAddress=ca-support@redhat.com,CN=Red Hat Entitlement Operations Authority,OU=Red Hat Network,O=Red Hat\\, Inc.,ST=North Carolina,C=US’:
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 401 Authorization Required

Username/Password Authentication Failed.
Actions #2

Updated by shaimon almost 7 years ago

I don't provide any cert when using yum commands. I am also able to download all the packages from reposync command.

Actions #3

Updated by ipanova@redhat.com almost 7 years ago

Since it is a RH repo is should be protected, i am pretty sure you have some certs for it. Please check your yum config.

Actions #4

Updated by mhrivnak almost 7 years ago

  • Related to Story #1282: As an EC2 user, I would like to set up a RHUI as an alternate content source added
Actions #5

Updated by bmbouter almost 7 years ago

I believe the CDN mirror that AWS operates uses special headers for their content protection and not certificates. For example alternate content sources recently had the ability to add headers which allows an ACS to present the necessary headers to receive content. I think this story is the same as Issue 1282 except that this is on sync while 1282 is about using an alternate content source.

Actions #6

Updated by bmbouter almost 7 years ago

  • Description updated (diff)

@shaimon, can you add the pulp-admin command you used to create the redhatLocal repo?

Actions #7

Updated by ttereshc almost 7 years ago

  • Tracker changed from Issue to Story
  • % Done set to 0

bmbouter will rewrite it.

Actions #8

Updated by shaimon almost 7 years ago

Commands used to create Pulp repo:

pulp-admin rpm repo update --repo-id=redhatLocal --feed=https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/os --verify-feed-ssl false

Sync repo with upstream:

pulp-admin rpm repo sync run --repo-id=redhatLocal

This sync command is not downloading packages

When accessing https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/mirror/content/dist/rhel/rhui/server/7/7Server/x86_64/os it is responding with two URLs. I updated the pulp repo with below command

pulp-admin rpm repo update --repo-id=redhatCDN --feed=https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/repos//content/dist
/rhel/rhui/server/7/7Server/x86_64/os --verify-feed-ssl false

Sync repo with upstream after updating feed URL:

pulp-admin rpm repo sync run --repo-id=redhatLocal

This sync command is not downloading packages

Actions #9

Updated by bmbouter almost 7 years ago

  • Subject changed from Unable to Sync with upstream Redhat repo in AWS to As a user, I can sync from a RedHat mirror provided by and hosted in AWS
  • Description updated (diff)

Rewriting based on my understanding of the feature request.

Actions #10

Updated by ipanova@redhat.com almost 7 years ago

I did some more investigation on this. To make this work 2 things should be provided.

1) Certs for the repo
2) Headers. Apparently these headers are set even in the yum plugin. rh-amazon-rhui-client rpm package, which is automatically installed, provides the patch.
Set Headers are present in /usr/lib/yum-plugins/amazon-id.py

Actions #12

Updated by bizhang over 6 years ago

  • Description updated (diff)
Actions #13

Updated by bizhang over 6 years ago

  • Description updated (diff)
Actions #14

Updated by ipanova@redhat.com over 6 years ago

@shaimon the story is ready to be worked on, feel free to ping us on the channel whenever you are in the need.

Actions #15

Updated by hartmel over 5 years ago

Excuse my intrusion as a new user (through Foreman/Katello), I would like to comment about another kind (at least I think it s different, I may be wrong) of repository hosted on Amazon. The example https://packages.graylog2.org/el/stable/2.4/x86_64 is hosted on Amazon but use arguments generated at request on repomd.xml and on the bz2 files referenced in repomd.xml. A 302 redirect is answered with the URL with some access keys and token in the query string. It cannot be synchronized by Pulp as far as I tested. Can/should I open another issue for this kind of repository ?

(And another behaviour is seen on http://repos.bigtop.apache.org/releases/1.2.1/centos/7/x86_64/ hosted on Amazon too, but Pulp can sync it)

Actions #16

Updated by bmbouter almost 5 years ago

  • Status changed from NEW to CLOSED - WONTFIX
Actions #17

Updated by bmbouter almost 5 years ago

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

Actions #18

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF