Project

Profile

Help

Issue #1608

closed

sync from cdn is broken

Added by amacdona@redhat.com about 8 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Master
Platform Release:
2.8.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Syncing from the feed below fails but reports that the sync was successful. Syncs from this feed works on 2.7-dev branch.

pulp-admin rpm repo  create --repo-id test --feed-ca-cert ./redhat-uep.pem --feed-cert ./cert.pem --feed-key ./key.pem  --feed https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/sat-tools/6.1/os/
Successfully created repository [test]
(pulp)[vagrant@dev pulp]$ pulp-admin rpm repo sync run --repo-id=test
+----------------------------------------------------------------------+
                    Synchronizing Repository [test]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[|]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       79/79 items
Delta RPMs: 0/0 items

... completed

Individual package errors encountered during sync:

Package:
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/sat-tools/6.1/o
s/python-qpid-proton-0.9-4.el7.x86_64.rpm
Error:   

Package:
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/sat-tools/6.1/o
s/rubygem-hammer_cli_foreman-0.1.4.14-1.el7sat.noarch.rpm
Error:   

Package:
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/sat-tools/6.1/o
s/rubygem-hammer_cli_foreman_docker-0.0.3.9-1.el7sat.noarch.rpm
Error:   

Package:
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/sat-tools/6.1/o
s/rubygem-powerbar-1.0.11-8.el7sat.noarch.rpm
Error:   

Package:
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/sat-tools/6.1/o
s/rubygem-hammer_cli_gutterball-0.0.1.3-1.el7sat.noarch.rpm
Error:   

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Task Succeeded

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[-]
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
6 of 6 items
... completed

Publishing Comps file
[-]
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed

Task Succeeded
Actions #1

Updated by amacdona@redhat.com about 8 years ago

  • Platform Release set to 2.8.0
Actions #2

Updated by bmbouter about 8 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bmbouter

The symptom is that all 79 packages are erroring so 0 are actually downloaded.

Actions #3

Updated by bmbouter about 8 years ago

I reproduced this in my Vagrant environment w/ a 2.8.0 checkout install. I confirmed that nectar is receiving a 404 for each rpm requested. For example:

https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/sat-tools/6.1/os/rubygem-hammer_cli_foreman-0.1.4.14-1.el7sat.noarch.rpm

I confirmed that curl reports the same thing:

curl --cacert ./redhat-uep.pem --cert ./cert.pem --key ./key.pem https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/sat-tools/6.1/os/rubygem-hammer_cli_foreman-0.1.4.14-1.el7sat.noarch.rpm
File not found."

@beav suggested that that URL was incorrect and that is the root cause of the 404. He suggested it should be the URL below, which he got from the CDN primary.xml

https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/sat-tools/6.1/os/Packages/rubygem-hammer_cli_foreman-0.1.4.14-1.el7sat.noarch.rpm

Which sounds right because it returns a 200 with curl as follows:

curl --cacert ./redhat-uep.pem --cert ./cert.pem --key ./key.pem https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/sat-tools/6.1/os/Packages/rubygem-hammer_cli_foreman-0.1.4.14-1.el7sat.noarch.rpm
Actions #4

Updated by bmbouter about 8 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (bmbouter)

I need to put this back to NEW since I can't continue at the moment, but there is something going wrong with the parsing of the metadata in primary.xml which is causing the urls to be incorrect and 404.

Actions #5

Updated by amacdona@redhat.com about 8 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to amacdona@redhat.com

Added by Austin Macdonald about 8 years ago

Revision b01406f6 | View on GitHub

use download path rather than file name

closes #1608

Actions #6

Updated by Anonymous about 8 years ago

  • Status changed from ASSIGNED to MODIFIED
  • % Done changed from 0 to 100
Actions #7

Updated by amacdona@redhat.com about 8 years ago

  • Status changed from MODIFIED to POST
Actions #8

Updated by bmbouter about 8 years ago

Lots of comments are posted on that PR with more information about why some repos are affected and others aren't.

Actions #9

Updated by amacdona@redhat.com about 8 years ago

  • Status changed from POST to MODIFIED
Actions #10

Updated by jortel@redhat.com about 8 years ago

  • Triaged changed from No to Yes
Actions #11

Updated by dkliban@redhat.com about 8 years ago

  • Status changed from MODIFIED to 5
Actions #12

Updated by pthomas@redhat.com about 8 years ago

  • Status changed from 5 to 6

verified

[root@mgmt5 ~]# pulp-admin rpm repo list --repo-id rhel6 --details
--------------------------------------------------------------------
RPM Repositories
--------------------------------------------------------------------

Id: rhel6
Display Name: None
Description: None
Content Unit Counts:
Distribution: 1
Package Category: 10
Package Group: 202
Rpm: 3819
Yum Repo Metadata File: 1
Notes:
Scratchpad:
Checksum Type: sha256
Importers:
Config:
Download Policy: on_demand
Feed: https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.7/
x86_64/kickstart/
Version: 3 (0x2)ificate:
91:8a:cf:ca:85:00:00:00
Issuer: Signature Algorithm: sha1WithRSAEncryption
C=US, ST=North Carolina, L=Raleigh, O=Red Hat, Inc., OU=Red
Hat Network, CN=Entitlement Master
Validity CA/emailAddress=
Not Afterot Before: Mar 18 11:24:54 2010 GMT
Subject: C=US, ST=North:24:54 2030 GMT
Carolina, O=Red Hat, Inc., OU=Red Hat Network, CN=Red Hat
Entitlement Operations
Authority/emailAddress=
Public Key Algorithm:c Key Info:
RSA Public Key: (4096 bit)
Modulus (4096 bit):
00:bb:26:8a:88:43:9c:db:c8:a4:13:19:54:94:79:
a5:b3:fa:1a:b1:39:07:87:68:52:6b:d3:79:1e:17:
27:90:ba:ea:eb:c5:4f:0b:86:4d:58:a0:35:70:0b:
fb:45:16:7f:da:f6:03:0c:4a:10:f0:b2:eb:98:7b:

Exponent: 65537 (0x10001)
             X509v3 Subject Keyensions:
    Identifier: 
    C4:49:78:56:45:9D:1E:2C:61:60:C3:66:4B:58:05:37:2F:03:22:7B
    X509v3 Authority Key Identifier: 
    keyid:88:4B:A6:45:C4:46:EC:8F:D6:CF:A6:F6:1A:CF:26:3C:90:CC
DirName:/C=US/ST=North
    Carolina/L=Raleigh/O=Red Hat, Inc./OU=Red Hat
    Network/CN=Entitlement Master
    CA/emailAddress=ca-support@redhat.com
             X509v3 Basical:E6:FE:42:28:25:C9:E6:5E
             X509v3:TRUEstraints: 
Certificate Sign, CRL Sign
SSL CA, S/MIME CA Type: 
    X509v3 Subject Alternative Name: 
             X509v3 Issuer:ca-support@redhat.com
    Alternative Name: 
     Signature Algorithm:l:ca-support@redhat.com
    sha1WithRSAEncryption
    5b:4d:2c:fe:50:85:cf:01:52:13:d1:95:4b:d5:f9:42:1d:1d:
    75:dd:a8:f6:8f:75:01:66:46:a8:0e:44:b1:b4:47:da:d6:7a:
    b2:b2:63:74:96:5a:6a:87:83:55:04:bd:da:9c:41:58:c6:89:
    0b:85:0d:8f:07:c9:a6:b8:3e:44:b9:66:8d:c6:75:d3:73:84:
    ab:e5:db:0e:72:39:05:89:4d:7d:ee:5c:9b:68:92:bb:c3:83:
    b3:43:64:20:fd:7f:ed:e3:e4:7b:f1:c7:cc:64:af:ea:1e:98:
    ee:c9:3d:cd:c8:71:ef:0a:e8:3f:5b:39:2f:53:4f:69:46:bd:
    9a:54:72:3e:7e:7e:bc:bb:5f:13:47:33:c9:34:ab:de:81:1e:
    18:78:0d:ef:b3:24:16:06:01:1c:f2:c5:3b:2a:b0:2f:8a:41:
a6:38:7d:45:fa:f7:ee:99:92:b3:5b:bf:0b:95:50:f6:41:8d:
79:ae:60:a1:d2:2f:7e:01:40:a2:52:c3:72:61:d2:5b:49:90:
 -----BEGIN CERTIFICATE-----3:c1:c2:5c:14:a1
MIIHZTCCBU2gAwIBAgIJAJGKz8qFAAAAMA0GCSqGSIb3DQEBBQUAMIGwMQs
wCQYD
-----ENDAUCiUsNyYdJbSZDPiFPBwlwUoQ==
-----BEGIN CERTIFICATE-----CATE-----
                    MIIHZDCCBUygAwIBAgIJAOb+QiglyeZeMA0GCSqGSIb3DQEBBQUAMIGwMQs
13PQU jboSsRM+ -----END CERTIFICATE----- 
    SSL Client Key:  -----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAz17SsKSWsn/O6zofxfB6qgrcOTaGM55e3KN6g+cBMIC
                   m
CkURC
eRS776hNmYNWreUbFoXfcW9WpPWAxF0FcSPtUQAXrnOz0pL5EEc38A==
-----END RSA PRIVATE KEY----- 
  Id:               yum_importer
  Importer Type Id: yum_importer
  Last Sync:        2016-03-14T14:50:41Z
  Repo Id:          rhel6
  Scratchpad:       
    Previous Skip List: 
    Repomd Revision:    1435822714
Distributors:         
  Auto Publish:        True
  Config:              
    Checksum Type: sha256
    Http:          False
    Https:         True
    Relative URL:  content/dist/rhel/rhui/server/6/6.7/x86_64/kickstart/
  Distributor Type Id: yum_distributor
  Id:                  yum_distributor
  Last Publish:        2016-03-14T14:52:03Z
  Repo Id:             rhel6
  Scratchpad:          
  Auto Publish:        False
  Config:              
    Http:         False
    Https:        True
    Relative URL: content/dist/rhel/rhui/server/6/6.7/x86_64/kickstart/
  Distributor Type Id: export_distributor
  Id:                  export_distributor
  Last Publish:        None
  Repo Id:             rhel6
  Scratchpad:          

[root@mgmt5 ~]# pulp-admin rpm repo sync run --repo-id rhel6
--------------------------------------------------------------------
Synchronizing Repository [rhel6]
--------------------------------------------------------------------

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... completed

Downloading repository content...
[-]
[==================================================] 100%
RPMs: 0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Cleaning duplicate packages...
[-]
... completed

Task Succeeded

Copying files
[-]
... completed

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[\]
... completed

Publishing RPMs
[\]
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[-]
... completed

Publishing Comps file
[==================================================] 100%
212 of 212 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[/]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

Actions #13

Updated by dkliban@redhat.com about 8 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions #15

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF