RPM Support

This is the PR and change that broke this: https://github.com/pulp/pulp_rpm/pull/1000/files#diff-68fa85b69e45aa1c5b5a032166c5ad0dL134

This needs to be done unconditionally:

catalog.add(unit, wanted[unit.unit_key_as_named_tuple].download_path)

back

The suggested fix will undo the performance gains provided by #2457 (which has 2 related bugzillas) by 50%.

Long term, we should have the importer(s) managing catalog entries regardless of download policy.

The advantages are:

• overhead is proportional to the units being added/removed.
• users can more freely change policies.

The disadvantages:

• Add small about of overhead to initial sync.
• Customer not using lazy is storing data they don't need.

I think to test the performance degradation, sync should be operational.
If it will be no-op, then it won't reach the modified code, the suggested change has no influence on no-op sync, only on the case when there are very few changes to the repo.

My suggestion for step 2 and step after the patch applied is to remove (aka unassiociate) any unit from the repo and only then sync.

ttereshc wrote:

I think to test the performance degradation, sync should be operational.
If it will be no-op, then it won't reach the modified code, the suggested change has no influence on no-op sync, only on the case when there are very few changes to the repo.

My suggestion for step 2 and step after the patch applied is to remove (aka unassiociate) any unit from the repo and only then sync.

Agreed. I left out (or implied) the step of doing:

mongo pulp_database --eval 'db.repo_importers.update({"scratchpad": {$ne: null}}, {$set: {"scratchpad.repomd_revision": null}}, {"multi":true})'

so the importer will continue through the affected code path.

f25¶

Created an f25 repository containing 50k RPMs.

Benchmarks:

7:45
7:55
7:41

With catalog management fixed[1] (one of the two optimizations undone):

9:25
9:27
9:23

EL7¶

Created an EL7 repository containing 5k RPMs.

Benchmarks:

1:44
1:43
2:08

With catalog management fixed[1] (one of the two optimizations undone):

2:10
1:57
1:55

*No real impact.

Notes:

Nothing had changed upstream but forced the importer to process the metadata anyway by setting the stored metadata version to NULL. In the real world, when nothing has changed upstream the metadata would be the same and the performance is 8 seconds. The difference introduced in this proposed change will be moot.

[1] https://github.com/pulp/pulp_rpm/pull/1038/commits/8058ec8ee34736e83243a8d5096b2caa7626dc61