RPM Support

@midhuhk - this is a Pulp2 error-msg. Can you tell us what version of Pulp2 you're using? Also, can you tell us the URL you're using to sync rhel8-appstream? Thanks

Running on pulp-2.21.5, the following script sync'd rhel8-base and rhel8-appstream from cdn.redhat.com successfully:

#!/bin/bash -v
BASE='rhel8-baseos'
STREAM='rhel8-appstream'
REMOTE1='http://cdn.redhat.com/content/dist/rhel8/8/x86_64/baseos/os/'
REMOTE2='http://cdn.redhat.com/content/dist/rhel8/8/x86_64/appstream/os/'
DEST='destination'
pulp-admin rpm repo create --repo-id=$BASE --relative-url=$BASE --feed=$REMOTE1 \
        --download-policy on_demand \
        --generate-sqlite false --repoview false \
        --feed-ca-cert /home/vagrant/devel/pulp_startup/CDN_cert/redhat-uep.pem \
        --feed-cert /home/vagrant/devel/pulp_startup/CDN_cert/cdn.crt \
        --feed-key /home/vagrant/devel/pulp_startup/CDN_cert/cdn.key
pulp-admin rpm repo sync run --repo-id=$BASE
pulp-admin rpm repo create --repo-id=$STREAM --relative-url=$STREAM --feed=$REMOTE2 \
        --download-policy on_demand \
        --generate-sqlite false --repoview false \
        --feed-ca-cert /home/vagrant/devel/pulp_startup/CDN_cert/redhat-uep.pem \
        --feed-cert /home/vagrant/devel/pulp_startup/CDN_cert/cdn.crt \
        --feed-key /home/vagrant/devel/pulp_startup/CDN_cert/cdn.key
pulp-admin rpm repo sync run --repo-id=$STREAM
pulp-admin rpm repo create --repo-id=$DEST
pulp-admin repo list

If the problem persists, you might want to open a support ticket with Red Hat - it's more likely a problem with bad data on CDN, and there isn't anything Pulp can do about that.

Pulp version used : 2.21.3-1.el7. I see issue only with appstream repo . Base repo works fine. Case has already been opened with redhat . Just want to make sure on pulp side as well

link : https://cdn.redhat.com/content/dist/rhel8/8/x86_64/appstream/os/

Redhat confirmed CDN is fine and validated by syncing with satellite in lab setup and it was fine . But with pulp sync , its still showing same error for me . Any suggestions ??

midhuhk wrote:

Redhat confirmed CDN is fine and validated by syncing with satellite in lab setup and it was fine . But with pulp sync , its still showing same error for me . Any suggestions ??

On CentOS7 on 2.21.3, I can't recreate - whatever the problem is, it's not Pulp.

I'm leaning towards "bad repo on a specific akamai node". Syncing "from a lab machine" won't help if that's the case, because you don't know what CDN node you're getting the content from.

If you ping cdn.redhat.com from the pulp2 box, what Ip/machine does it return? I'll see if I can coerce my reproducer into using that specific node for its feed and see what happens.

cdn.redhat.com is connected via proxy like all other repo . For failed repo sync ,destination IP is pointing to 95.101.92.251

midhuhk wrote:

cdn.redhat.com is connected via proxy like all other repo . For failed repo sync ,destination IP is pointing to 95.101.92.251

Well, there;s another idea down the tubes. I linked cdn.redhat.com to that specific IP/akamai-node and sync'd rhel8-appstream with no problem (other than it being slow due to coming from Far Away).

Somewhere there is a cache of filelists.xml and other.xml, that is broken, that your sync is finding. Is your proxy perchance a caching proxy?

The repo-metadata appears to be ok, we've had no other reports of pulp2 having problems with rhel8-appstream. I'm out of ideas, I'm afraid. :(

Deleted the repo and created again , but still issue perisist . Looks like corrupted data still exist some where even after deleting and recreating the repo.

I'm also seeing the same thing with pulp version 2.21.5. I also opened a case with Redhat and they confirmed that their repo data is correct. We too use a proxy but we disable caching.

it looks like issue is with pulp . We tried to set retain-old-count to a high value (100) and sync works fine with this setting .Before the value set was retain-old-count =2 .

ggainey , Can you set retain-old-count to 2 in your setup and see if its working or not ?

midhuhk wrote:

it looks like issue is with pulp . We tried to set retain-old-count to a high value (100) and sync works fine with this setting .Before the value set was retain-old-count =2 .

ggainey , Can you set retain-old-count to 2 in your setup and see if its working or not ?

Ah! That was the missing piece, I can now reproduce the problem with this script:

#!/bin/bash -v
STREAM='rhel8-appstream'
REMOTE2='http://cdn.redhat.com/content/dist/rhel8/8/x86_64/appstream/os/'
pulp-admin rpm repo create --repo-id=$STREAM --relative-url=$STREAM --feed=$REMOTE2 \
        --retain-old-count 2 \
        --download-policy on_demand \
        --generate-sqlite false --repoview false \
        --feed-ca-cert /home/vagrant/devel/pulp_startup/CDN_cert/redhat-uep.pem \
        --feed-cert /home/vagrant/devel/pulp_startup/CDN_cert/cdn.crt \
        --feed-key /home/vagrant/devel/pulp_startup/CDN_cert/cdn.key
pulp-admin rpm repo sync run --repo-id=$STREAM
pulp-admin repo list

The code involved in retain-old-counts hasn't changed in two years, so I'm not sure what's going on here. Maybe something about the shape of rhel8-appstream collides with the "only keep so many versions" logic?

For the moment, the workaround is "don't use retain-old-versions on rhel8-appstream repo". Given where Pulp2 is in its lifecycle, I can't guarantee we'll get any further than that, I'm afraid.

Came back to this issue and started trying to figure out what is/was going on. However, my reproducer from #14 now runs successfully. My pulp2 hasn't changed (it's 2.21.5). @midhuhk , @aarvelobt , do you still see this happening?

Have you seen this continue happening?

Thank you for checking . Yes it works fine now with retain-old-count =2 . Pulp version is same here and not sure what caused it .