Story #213
closed
Add upload option to have an rpm/srpm/drpm/etc overwrite an existing one with the same unit_key fields (omitting checksums)
0%
Description
++ This bug was initially created as a clone of Bugzilla Bug #1041317 ++
Description of problem:
Created attachment 835839
Metadata file with duplicate packages that have the same name/version/release number but have different checksums
Description of problem:
Uploading to pulp an rpm with the same name/version/release but with a different checksum doesn't overwrite the existing one, it creates 2 versions of the same rpm package.
Version-Release number of selected component (if applicable):
pulp-server-2.2.0-1.el6.noarch running on CentOS 6.4
How reproducible:
Upload to a pulp repo the package <name>-0.0.1-70 , publish the pulp repo.
Upload the same <name>-0.0.1-70 package but with a different checksum.
On querying pulp will return 2 versions of the same package name and yum install, on a server using the pulp server as yum repo, will fail
[stefcoco@pulp-001 ~]$ pulp-admin rpm repo content rpm --repo-id cps-stable --match 'name=goggins' --match 'version=0.0.1' --match 'r
elease=70'
Arch: noarch
Buildhost: inf-jks-002.cpp.o.itv.net.uk
Checksum: 5f29f29217bf710260f04fa080bb199e49610851eb5806de2b95aed02416b464
Checksumtype: sha256
Description: Goggins java app
Epoch: 0
Filename: goggins-0.0.1-70.noarch.rpm
License: Proprietary
Name: goggins
Provides: config(goggins) = 0.0.1-70-0, goggins = 0.0.1-70-0
Release: 70
Requires: /bin/sh, /bin/sh, java-1.6.0-openjdk
Vendor: None
Version: 0.0.1
Arch: noarch
Buildhost: inf-jks-002.cpp.o.itv.net.uk
Checksum: d2dec47020faa72d01bb93117181659231bb3fd12334808afbb372198c91b284
Checksumtype: sha256
Description: Goggins java app
Epoch: 0
Filename: goggins-0.0.1-70.noarch.rpm
License: Proprietary
Name: goggins
Provides: config(goggins) = 0.0.1-70-0, goggins = 0.0.1-70-0
Release: 70
Requires: /bin/sh, /bin/sh, java-1.6.0-openjdk
Vendor: None
Version: 0.0.1
Yum install fails with:
[Errno -1] Package does not match intended download. Suggestion: run yum --enablerepo=itv-pulp-cps-stable clean metadata
Trying other mirror.
Error Downloading Packages:
goggins-0.0.1-70.noarch: failure: goggins-0.0.1-70.noarch.rpm from itv-pulp-cps-stable: [Errno 256] No more mirrors to try.
because yum metadata contains the same package twice with different checksums.
Have a look at the attached metadata files to see what I mean ( search for goggins rel=70 )
Steps to Reproduce:
1. Create a pulp repo
2. Upload an rpm to it and publish it
3. Upload a second rpm with the same name/version/release but with a different checksum and publish it
4. Try installing, on another server, from this repository via regular yum repos.
Actual results:
Pulp treats these packages as 2 different packages but yum install fails with
[Errno -1] Package does not match intended download. Suggestion: run yum --enablerepo=itv-pulp-cps-stable clean metadata
Trying other mirror.
Error Downloading Packages:
goggins-0.0.1-70.noarch: failure: goggins-0.0.1-70.noarch.rpm from itv-pulp-cps-stable: [Errno 256] No more mirrors to try.
Expected results:
Pulp should re-place the current package with the one that is being uploaded. That will create just one checksum inside the metadata files and yum would be able to install that package.
Additional info:
--- Additional comment from stefan.cocora@gmail.com at 12/12/2013 14:53:47 ---
The published repo, on disk, only contains one package version of goggins rel=70 but the metadata file contains 2 checksums which breaks yum.
[stefcoco@pulp-001 ~]$ ll /var/lib/pulp/published/http/repos/cps-stable/
total 60
lrwxrwxrwx 1 apache apache 144 Nov 27 16:58 dawkins-0.0.1-118.noarch.rpm -> /var/lib/pulp/content/rpm/dawkins/0.0.1/118/noarch/c5b2b5
b48ab700ef8f2c15fb46dff6623052b8e328b95a8fdf6728520c109bfd/dawkins-0.0.1-118.noarch.rpm
lrwxrwxrwx 1 apache apache 140 Nov 27 16:58 fnord-0.0.1-215.noarch.rpm -> /var/lib/pulp/content/rpm/fnord/0.0.1/215/noarch/a9a196e85d
95de9d9fdc1a6f675fdcc55f248b62f99514515b11a9546b9bee46/fnord-0.0.1-215.noarch.rpm
lrwxrwxrwx 1 apache apache 142 Nov 27 16:58 goggins-0.0.1-61.noarch.rpm -> /var/lib/pulp/content/rpm/goggins/0.0.1/61/noarch/6b1f0459
16b9aa6b6425f9249af15c3475325193d5b379cc5d270c1404c5fbef/goggins-0.0.1-61.noarch.rpm
lrwxrwxrwx 1 apache apache 142 Nov 29 16:01 goggins-0.0.1-65.noarch.rpm -> /var/lib/pulp/content/rpm/goggins/0.0.1/65/noarch/152fa050
6d259da3652ebb849f9832e65ed3db5804496c0ee4974891ff85eca1/goggins-0.0.1-65.noarch.rpm
lrwxrwxrwx 1 apache apache 142 Dec 12 12:15 goggins-0.0.1-67.noarch.rpm -> /var/lib/pulp/content/rpm/goggins/0.0.1/67/noarch/97df5a25
e4ee78c0cdcc2ae1b71a7b43e0acca8c87eabcca93133fdc1fec2caa/goggins-0.0.1-67.noarch.rpm
lrwxrwxrwx 1 apache apache 142 Dec 12 11:29 goggins-0.0.1-70.noarch.rpm -> /var/lib/pulp/content/rpm/goggins/0.0.1/70/noarch/d2dec470
20faa72d01bb93117181659231bb3fd12334808afbb372198c91b284/goggins-0.0.1-70.noarch.rpm
lrwxrwxrwx 1 apache apache 146 Nov 27 16:58 greendale-0.0.1-87.noarch.rpm -> /var/lib/pulp/content/rpm/greendale/0.0.1/87/noarch/f2b9
e97e9bd6214e6d04f9dda523d752bac9f6d2b13e98ff94ef17a2cdb1ce82/greendale-0.0.1-87.noarch.rpm
lrwxrwxrwx 1 apache apache 136 Nov 27 16:58 jess-0.0.1-58.noarch.rpm -> /var/lib/pulp/content/rpm/jess/0.0.1/58/noarch/1c7be39c2d4284
ece87f3eb0b1d98ad8102d102a9535559ed4c1cc26a60b4461/jess-0.0.1-58.noarch.rpm
lrwxrwxrwx 1 apache apache 142 Nov 27 16:58 minerva-0.0.1-40.noarch.rpm -> /var/lib/pulp/content/rpm/minerva/0.0.1/40/noarch/97d40821
de5be66b2fc6f04538b3185645ae21a1b90bdd9c71b5338ab25c5e28/minerva-0.0.1-40.noarch.rpm
lrwxrwxrwx 1 apache apache 67 Nov 27 14:22 Packages -> /var/lib/pulp/working/repos/cps-stable/distributors/yum_distributor
lrwxrwxrwx 1 apache apache 134 Nov 27 16:58 pat-0.0.1-75.noarch.rpm -> /var/lib/pulp/content/rpm/pat/0.0.1/75/noarch/c15cd481d2349fab
28af275f3a3374969f28829ac759f173981bed3fea2acf47/pat-0.0.1-75.noarch.rpm
drwxr-xr-x 2 apache apache 4096 Dec 12 12:17 repodata
drwxr-xr-x 2 apache apache 4096 Dec 12 12:15 repodata.old
lrwxrwxrwx 1 apache apache 148 Nov 27 16:58 swagger-ui-0.0.1-39.noarch.rpm -> /var/lib/pulp/content/rpm/swagger-ui/0.0.1/39/noarch/a4
5184d839b5459b1b9d4fc5f7d9089b1388be957be5d02560f0476308755c45/swagger-ui-0.0.1-39.noarch.rpm
lrwxrwxrwx 1 apache apache 140 Nov 27 16:58 venus-0.0.1-127.noarch.rpm -> /var/lib/pulp/content/rpm/venus/0.0.1/127/noarch/5878912453
6dca80510c3dbfe2a511644e6f0d82194e37a3c34f05a8981119c3/venus-0.0.1-127.noarch.rpm
--- Additional comment from bcourt@redhat.com at 12/12/2013 15:12:16 ---
To add clarification here. The problem is that pulp is using the checksum as part of the uniqueness key for an RPM. A shorter term fix could be to remove the checksum value from the uniqueness constraint and keep the checksum type as part of the uniqueness constraint.
In the long term the request requires decoupling the checksum type & value from the uniqueness key and having pulp generate multiple checksum types and the various checksum values for a particular rpm.
--- Additional comment from mhrivnak@redhat.com at 12/18/2013 17:09:47 ---
We will consider validation in the yum distributor at publish time as a future feature.
Related issues
Updated by cristi.falcas@gmail.com almost 9 years ago
Hello,
Any way to fix this manually? If I delete the rpms from disk would this be enough?
Thank you,
Cristian Falcas
Updated by cristi.falcas@gmail.com almost 9 years ago
Just to let others know what I did in my case:
repo_id=id_of_repo_with_multiple_identical_rpms;
pulp-admin rpm repo content rpm --repo-id $repo_id --fields name,checksum | sed 's/\(Checksum:\|Name:\)//g' | gawk -v repo_id=$repo_id -v RS="\n\n" '{ gsub(/ /, "", $0); if ($2 in a) print "pulp-admin rpm repo remove rpm --repo-id "repo_id" --match name="$2" --match checksum="$1; else a[$2]=1;}' | sh
pulp-admin rpm repo publish run --repo-id $repo_id
What it should do:
- list all rpm names and the corresponding checksum
- skip the first occurrence
- remove from repo the other rpms by matching the name and checksum
Updated by cristi.falcas@gmail.com almost 9 years ago
That went bad fast :)
It removed all but one versions from the same rpm.
Updated to make rpm unique by name, version and release:
repo_id=id_of_repo_with_multiple_identical_rpms;
pulp-admin rpm repo content rpm --repo-id $repo_id --fields name,checksum,version,release | sed 's/\(Checksum:\|Name:\|Version:\)//g' | gawk -v repo_id=$repo_id -v RS="\n\n" '{ gsub(/ /, "", $0); if ($2$3$4 in a) print "pulp-admin rpm repo remove rpm --repo-id "repo_id" --match name="$2" --match version="$4" --match checksum="$1; else a[$2$3$4]=1;}' | sh
pulp-admin rpm repo publish run --repo-id $repo_id
Updated by cristi.falcas@gmail.com almost 9 years ago
New update. Looks like --match was not the best flag to be used for versions like 1.0.0.beta3, it's better to use --str-eq
repo_id=id_of_repo_with_multiple_identical_rpms;
pulp-admin rpm repo content rpm --repo-id $repo_id --fields name,checksum,version,release | \
sed 's/\(Checksum:\|Name:\|Version:\|Release:\)//g' | \
gawk -v repo_id=$repo_id -v RS="\n\n" '{
gsub(/ /, "", $0);
if ($2$3$4 in a)
print "pulp-admin rpm repo remove rpm --repo-id "repo_id" --str-eq name="$2" --str-eq version="$4" --str-eq checksum="$1;
else a[$2$3$4]=1;
}' | sh
pulp-admin rpm repo publish run --repo-id $repo_id
Updated by bmbouter over 8 years ago
- Related to Issue #1406: Uploading the same Content Unit twice causes a 500 error added
Updated by bmbouter over 8 years ago
- Related to Issue #494: better handle repositories with duplicate NVREAs added
Updated by bmbouter over 8 years ago
- Subject changed from [RFE] Uploading an rpm to pulp with the same name/version/release should overwrite the existing one. to Uploading an rpm to pulp with the same name/version/release should overwrite the existing one
- Groomed set to No
- Sprint Candidate set to No
I believe this feature request is related to #494 which improves duplicate NEVRA for association purposes, but does not address having two nevras that are the same inside of pulp itself. I believe with the upcoming 2.8.0 release, the second upload will fail with a duplicate key error preventing the second one from being imported into pulp at all.
@Christi.falcas Given all that, is this feature still being requested?
Updated by cristi.falcas@gmail.com over 8 years ago
I'm not the original poster, but 494 seems to be the same issue.
Updated by bmbouter over 8 years ago
- Subject changed from Uploading an rpm to pulp with the same name/version/release should overwrite the existing one to Add upload option to have an rpm/srpm/drpm/etc overwrite an existing one with the same unit_key fields (omitting checksums)
bmbouter wrote:
I believe with the upcoming 2.8.0 release, the second upload will fail with a duplicate key error preventing the second one from being imported into pulp at all.
The above statement is not exactly the same situation as the feature request is requesting resolve. The second upload will fail starting with 2.8.0 if you upload the exact same RPM twice, but if you upload the a different rpm as this feature request describes you'll be able to upload it and you'll still get the duplicate.
If the overwrite feature is implemented, I'll suggest it be done with an option of some kind instead of a default to overwrite approach. I've retitled the feature request accordingly.
Updated by bmbouter about 5 years ago
- Status changed from NEW to CLOSED - WONTFIX
Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.
Updated by robert1833 over 4 years ago
- File clipboard-201908031719-jxf2b.png added
- File clipboard-201908031719-ejuw0.png added
Updated by SarkariResult. over 4 years ago
- File Forner-Christian_730x475_acf_cropped1.jpg added
Updated by ggainey about 4 years ago
- File deleted (
clipboard-201908031719-jxf2b.png)
Updated by ggainey about 4 years ago
- File deleted (
clipboard-201908031719-ejuw0.png)
Updated by ggainey about 4 years ago
- File deleted (
Forner-Christian_730x475_acf_cropped1.jpg)
.