Issue #3020
closedDistribution files mismatch
Description
We mirror CentOS repositories in our Pulp and we've hit the issue described in [1] with releases 7.2 and 7.4.
It affects all files in the LiveOS
and images
directories as follows (note the 7.2 and 7.4 symlink pointing to the same file):
:/var/lib/pulp] find ./published/yum/master/yum_distributor/centos-7* -iname vmlinuz -ls
11163912 4 lrwxrwxrwx 1 apache apache 129 Sep 19 12:17 ./published/yum/master/yum_distributor/centos-72/1505816224.61/images/pxeboot/vmlinuz -> /var/lib/pulp/content/units/distribution/de/d64602f1a2165af46f6cd4c8c2645a0422607173fb34617458c9f419e2c7bf/images/pxeboot/vmlinuz
34740815 4 lrwxrwxrwx 1 apache apache 129 Sep 19 13:16 ./published/yum/master/yum_distributor/centos-73/1505819770.72/images/pxeboot/vmlinuz -> /var/lib/pulp/content/units/distribution/8d/d927134b78ab15e3c803b686a6baf564c5ccb97154da48f9ead7041eb53188/images/pxeboot/vmlinuz
11021555 4 lrwxrwxrwx 1 apache apache 129 Sep 19 08:39 ./published/yum/master/yum_distributor/centos-74/1505803143.84/images/pxeboot/vmlinuz -> /var/lib/pulp/content/units/distribution/de/d64602f1a2165af46f6cd4c8c2645a0422607173fb34617458c9f419e2c7bf/images/pxeboot/vmlinuz
Even when the repos are deleted and created again, pulp still thinks the 7.2 and 7.4 distribution files are the same. When downloaded manually form upstream we feed our pulp from [2, 3], the files indeed differ.
md5sum 72/vmlinuz 74/vmlinuz
418a1a0074554d7d7d63b514990c2d1f 72/vmlinuz
43a395b89485ec908e6c0622273299c4 74/vmlinuz
I use a workaround of removng the feed
and sync schedule
of 7.4 to have up-to-date 7.2 distribution files. This allows me to serve both 7.2 and 7.4 rpms, but I'm unable to mirror both 7.2 and 7.4 distribution files simultaneously
[1] https://www.redhat.com/archives/pulp-list/2016-January/msg00003.html
[2] http://vault.centos.org/centos/7.2.1511/os/x86_64/
[3] http://mirror.centos.org/centos/7.4.1708/os/x86_64/
Updated by mhrivnak over 6 years ago
This comes down to how Pulp defines uniqueness among different "distribution" units. It does so based on these fields taken from the treeinfo file:
- family
- variant
- version
- arch
With RHEL we expect to see versions such as "7.2" or "7.4", but with the CentOS repos you referenced, I see the version as just "7" for all of them.
It also appears that for CentOS 7.3, the "family" is arbitrarily different from 7.2 and 7.4, which would have allowed it to be seen as a unique distribution unit.
Here are the three treeinfo files I'm looking at:
http://vault.centos.org/centos/7.2.1511/os/x86_64/.treeinfo
http://mirror.centos.org/centos/7.3.1611/os/x86_64/.treeinfo
http://mirror.centos.org/centos/7.4.1708/os/x86_64/.treeinfo
This is a general problem that Pulp should find a better way to establish uniqueness for these units, but it could be difficult to make that big of a data model change with Pulp 2.
Updated by ttereshc over 6 years ago
- Project changed from Pulp to RPM Support
- Priority changed from Normal to High
- Triaged changed from No to Yes
- Tags Pulp 3 added
Candidate to be fixed in Pulp3
Updated by beny over 6 years ago
Thanks for the root cause analysis. Reported the treeinfo value mismatch to CentOS. https://bugs.centos.org/view.php?id=13904
Updated by ttereshc over 3 years ago
- Status changed from NEW to CLOSED - WORKSFORME
Worked in Pulp 3. Feel free to open a bug if you encounter it again.