Project

Profile

Help

Issue #3551

RemoveOldRepodataStep for yum publisher not checking repomd.xml to remove old files

Added by mtahir about 2 years ago. Updated about 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Master
Platform Release:
2.17.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:

Description

Current code in yum distributor publish doesn't actually check the files within the repodata.xml to compare the files that should be kept and files that should be deleted. Instead it makes the assumption that the latest one of each file type (to be removed) is present and removes the files older then the threshold (default 14 days). Given that mtime could be changed by someone touching a file or even moving it on the file system, it may lead to deletion of incorrect files.

related code:

        for key, val in to_remove.iteritems():
            # preserve at least one file of each kind - pop out latest
            if not set(groupped[key]) - set(val):
                val.pop(0)
            for f in val:
                self.remove_repodata_file(f[0])

Related issues

Related to RPM Support - Story #2788: As a user i can configure removal of old published repodata CLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to Pulp - Issue #5573: Publish won't create multiple checkecksummed copies of primary.xml, fileliststs.xml etc even when in fast-forward modeCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 3774e387 View on GitHub
Added by mtahir about 2 years ago

Fix RemoveOldRepodataStep for yum publisher

Previously RemoveOldRepodataStep step didn't actually check the files within the repodata.xml to find the files that should be deleted. Instead it made the assumption that the latest one of each file type (to be removed) was present, and removed the files older than the threshold

fixes #3551 https://pulp.plan.io/issues/3551

Co-authored-by: Blake McIvor

History

#1 Updated by bmbouter about 2 years ago

  • Triaged changed from No to Yes

I think in order to do this more accurately we would need to store a record of the published data. Where would we store this type of data?

The rpm devs are really Pulp3 focused right now and Pulp2 is almost near it's maintenance phase. For Pulp2 we could work with you to help prepare a fix. Is there any possibility you would be able to contribute an improvement in this area?

#2 Updated by mtahir about 2 years ago

I have created a fix[1] and pushed it for review. With regards to storing the record of published data, I created the fix with the assumption that any file that is not in repomd.xml, and older than the threshold should be removed.

[1] https://github.com/mztahir/pulp_rpm/commit/0b1c9edaa853b28bdead8af6e5ed36686aa6caa6

#3 Updated by ipanova@redhat.com about 2 years ago

@mtahir, can you actually submit this as a PR to upstream pulp_rpm? thank you.

#4 Updated by bmcivor about 2 years ago

@ipanova, PR has been created on github for this issue. If there's any more information you need, feel free to ask.

#5 Updated by ipanova@redhat.com about 2 years ago

  • Status changed from NEW to POST

#6 Updated by ipanova@redhat.com about 2 years ago

  • Related to Story #2788: As a user i can configure removal of old published repodata added

#8 Updated by mtahir about 2 years ago

  • Status changed from POST to MODIFIED

#10 Updated by dkliban@redhat.com almost 2 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
  • Platform Release set to 2.17.0

#11 Updated by bmbouter about 1 year ago

  • Tags Pulp 2 added

#12 Updated by mihai.ibanescu@gmail.com 9 months ago

  • Related to Issue #5573: Publish won't create multiple checkecksummed copies of primary.xml, fileliststs.xml etc even when in fast-forward mode added

Please register to edit this issue

Also available in: Atom PDF