Issue #2503
closed
remove_missing does not seem to remove on_demand catalog entries
Description
working with a katello user, it seems that if you've configured a yum repo to use on_demand and remove_missing, the catalog entry isn't getting removed and even though that rpm was removed from the upstream repo, pulp will still try to download it from that repo. In this case it was removed from one repo but added to 2nd repo, but pulp was trying to still pull it from the 1st one.
This is compounded by the fact that it seems that pulp uses the oldest cataloge entry rather than the newest (this might need to be another issue).
Updated by mhrivnak over 7 years ago
I reproduced this on master.
1. sync a repo with policy "on_demand"
2. change the feed to a repo with different packages (this simulates RPMs getting removed)
3. update the repo to use the "remove_missing" option
4. sync the repo
You can see that packages get removed from the repo, but no catalog entries were removed. This is not expected, because catalog entries for those RPMs that were removed no longer can be used to retrieve the content.
Updated by bizhang over 7 years ago
- Priority changed from Normal to High
- Sprint/Milestone set to 31
- Severity changed from 2. Medium to 3. High
- Triaged changed from No to Yes
Updated by jortel@redhat.com over 7 years ago
Few things need to be fixed:
1. The streamer should use the newest entry first. Fix here [1].
2. The RPM importer needs to remove catalog entries for "missing" units.
Note: The long term design for the streamer was to sort the entries newest to oldest. Then, try them in order and only fail when all entries have failed. I will write a pulp3 story to address this gap. As for the RPM importer, the ChangeSet will handle removal of the deleted content.
[1] https://github.com/pulp/pulp/blob/master/streamer/pulp/streamer/server.py#L184
Updated by jortel@redhat.com over 7 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to jortel@redhat.com
Added by jortel@redhat.com over 7 years ago
Updated by jortel@redhat.com over 7 years ago
- Status changed from ASSIGNED to POST
Updated by pthomas@redhat.com over 7 years ago
Updated by dkliban@redhat.com about 7 years ago
- Sprint/Milestone changed from 31 to 32
Updated by jortel@redhat.com about 7 years ago
- Status changed from POST to MODIFIED
Applied in changeset pulp_rpm:459464346d2c47261697cb8222fb80ba3b8ef110.
Updated by pthomas@redhat.com about 7 years ago
[root@mgmt2 ~]# pulp-admin rpm repo create --repo-id zoo --feed https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-signed/ --download-policy on_demand
Successfully created repository [zoo]
[root@mgmt2 ~]#
[root@mgmt2 ~]#
[root@mgmt2 ~]#
[root@mgmt2 ~]#
[root@mgmt2 ~]# pulp-admin rpm repo sync run --repo-id zoo
+----------------------------------------------------------------------+
Synchronizing Repository [zoo]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[|]
... completed
Downloading repository content...
[==================================================] 100%
RPMs: 32/32 items
Delta RPMs: 0/0 items
... completed
Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed
Importing errata...
[-]
... completed
Importing package groups/categories...
[\]
... completed
Cleaning duplicate packages...
[-]
... completed
Task Succeeded
Initializing repo metadata
[-]
... completed
Publishing Distribution files
[-]
... completed
Publishing RPMs
[==================================================] 100%
32 of 32 items
... completed
Publishing Delta RPMs
... skipped
Publishing Errata
[==================================================] 100%
4 of 4 items
... completed
Publishing Comps file
[==================================================] 100%
4 of 4 items
... completed
Publishing Metadata.
[-]
... completed
Closing repo metadata
[-]
... completed
Generating sqlite files
... skipped
Generating HTML files
... skipped
Publishing files to web
[-]
... completed
Writing Listings File
[-]
... completed
Task Succeeded
[root@mgmt2 ~]# pulp-admin rpm repo update --repo-id zoo --feed https://repos.fedorapeople.org/pulp/pulp/fixtures/drpm-signed/
Repository [zoo] successfully updated
[root@mgmt2 ~]#
[root@mgmt2 ~]#
[root@mgmt2 ~]# pulp-admin rpm repo update --repo-id zoo --remove-missing true
Repository [zoo] successfully updated
[root@mgmt2 ~]#
[root@mgmt2 ~]#
[root@mgmt2 ~]#
[root@mgmt2 ~]# pulp-admin rpm repo sync run --repo-id zoo
+----------------------------------------------------------------------+
Synchronizing Repository [zoo]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[-]
... completed
Downloading repository content...
[==================================================] 100%
RPMs: 5/5 items
Delta RPMs: 4/4 items
... completed
Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed
Importing errata...
[-]
... completed
Importing package groups/categories...
[-]
... completed
Cleaning duplicate packages...
[-]
... completed
Task Succeeded
Initializing repo metadata
[-]
... completed
Publishing Distribution files
[-]
... completed
Publishing RPMs
[==================================================] 100%
5 of 5 items
... completed
Publishing Delta RPMs
[==================================================] 100%
4 of 4 items
... completed
Publishing Errata
[-]
... completed
Publishing Comps file
[-]
... completed
Publishing Metadata.
[-]
... completed
Closing repo metadata
[-]
... completed
Generating sqlite files
... skipped
Generating HTML files
... skipped
Publishing files to web
[-]
... completed
Writing Listings File
[-]
... completed
Task Succeeded
> db.lazy_content_catalog.find()
Updated by bizhang about 7 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
Updated by bmbouter about 6 years ago
- Sprint changed from Sprint 16 to Sprint 14
Updated by bmbouter almost 4 years ago
- Category deleted (
14)
We are removing the 'API' category per open floor discussion June 16, 2020.
.
Delete catalog entries for units no longer in the remote repository. closes #2503