RPM Support

I was getting some more info on how yum works in this case and i found out that:
- the old repodata files are stored not for 14 days but for 6 hours only( source man yum.conf, option 'metadata_expire')
- I do understand that in case there was some change in metadata and re-publish occurred, repomd.xml will be different, but yum will still try to fetch the old files. In this case yum will complain, you can run 'yum clean expire-cache' and next time yum will fetch fresh metadata.

I don't understand why we cannot leave this kind of situations managed directly by yum? Because all our repos are regular yum repos.

More information on this will be appreciated, maybe i am missing some points here.

I think we need a clear reproducer before starting work. Would it look something like this?

• publish a yum repo with pulp
• yum update on a client aimed at that repo
• add a newer rpm to the repo in pulp and re-publish
• yum update on the client again, and yum breaks? how exactly?

Reproducing this and showing output from yum would be ideal.

FYI I noticed today that createrepo provides an option called --retain-old-md which retains the number of previous metadata outputs by count. This in a way would provide Pulp with a similar feature capability.

My createrepo_c also has --retain-old-md-by-age=AGE , (but createrepo doesn't.)

I would like a recap of Pulp's behavior regarding repodata retention when using fast-forward, regular, and force-full publishes. I don't currently know what the behaviors are in these areas. It seems that what is written in this ticket is the preferred behavior for all of those publish types so maybe what it does currently doesn't matter.

All publishes by upstream pulp_rpm discard old repo metadata files. There is currently no retention, regardless of whether a publish was "full" or "incremental".

rmcgover, since the behavior currently in place for you retains all old metadata files, I wonder if we could simply preserve that behavior when you upgrade to a recent upstream Pulp release.

I believe the goal is: make sure repo metadata files that might be requested by a client don't disappear.

When using rsync to transfer a newly published repo to a remote destination, could you simply have that rsync operation preserve existing metadata files on the other end? That is the default behavior of rsync. This would allow old metadata files to accumulate on the remote side only. Each Pulp publication would still only have the most recently-published metadata files. That should fit the use case, and it can be done with current upstream Pulp.

Thoughts on that?

In the typical case, the requirement is met by the behavior you describe, as our publishes by default won't be deleting old files from the remote. However, there are cases where old files will have to be deleted.

Here's a scenario:

1. User runs "yum update <package>", it downloads repomd.xml and other metadata but not filelists.
2. Release engineers remove an RPM from the repo. They need it to no longer be downloadable, so they trigger the yum distributor followed by rsync distributor with "delete" option. This deletes the repodata referenced by the repomd.xml fetched by the user at (1).
3. User runs "yum provides ..." which tries and fails to download filelists.

It seems like relying on the rsync layer will give something that works most of the time, by accident; and I think this issue is hoping for a more robust solution which works by design.

rmcgover, great, thanks for the feedback.

I'm taking from this discussion that you would like to prioritize this feature, but it is not a blocker for you. Is that correct?

As for the 3 steps you describe, can you explain how that should work in the ideal case, once we've implemented this RFE? It seems like if you want to normally keep (some) old metadata, but occasionally delete it, any ill effects of the deletion will happen regardless of whether this RFE is in place. So I'm not quite following what you have in mind.

Yes, I don't think we have to consider this a blocker.

If I run through the same scenario again, where pulp_rpm had implemented the behavior currently described in this issue: (behavior change in bold)

1. User runs "yum update <package>", it downloads repomd.xml and other metadata but not filelists.
2. Release engineers remove an RPM from the repo. They need it to no longer be downloadable, so they trigger the yum distributor followed by rsync distributor with "delete" option. Since the repodata downloaded by user at (1) isn't yet 6 hours old, it still exists in Pulp's published yum repo, so rsync doesn't delete it.
3. User runs "yum provides ..." which is able to successfully download filelists.

mhrivnak wrote:

All publishes by upstream pulp_rpm discard old repo metadata files. There is currently no retention, regardless of whether a publish was "full" or "incremental".

I just did some testing on 2.13 and that seems to be the case for XML files but not sqlite files. If I publish with generate_sqlite=true, and don't include force_full=true, then the destination path will retain all sqlite files.

Unfortunately the current behavior seems like the worst of both worlds as our consumers suffer from attempts to download XML files which were removed, while we suffer wasted disk space and publish time from the unlimited retention of sqlite files.

Tested on pulp 63aa4d9ce5de374d01d8bfdcda3f84594de4e5af, pulp_rpm d5bded6b2db0640ab5d262368335940c971ba3c7.

Comment https://pulp.plan.io/issues/1684#note-24 seems like a separate bug worth filing and fixing.

OK, I've filed separate issue https://pulp.plan.io/issues/2788 for the sqlite behavior. The expected behavior references back to this issue.