Project

Profile

Help

Issue #3736

When the checksum type of the upstream repository changed, syncing a "on_demand" repository will fail

Added by hyu over 1 year ago. Updated 7 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Severity:
2. Medium
Version:
Platform Release:
2.16.4
Blocks Release:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 39

Description

When the checksum type of the upstream repository changed, syncing a "on_demand" repository will fail with the following error:
------------
Checksum type "sha1" is not available for all units in the repository. Make sure those units have been downloaded
------------

When syncing a repository, Pulp will update the checksum type in the repo scratchpad to match the checksum type of the upstream repository, but the checksum_type configuration in the distributor will remain no change. This will cause the above error due to data/configuration inconsistency.

The "get_repo_checksum_type" function in [1] will attempt to save the checksum type to the distributor configuration if it is not already set. In my opinion, It should not save the checksum type after falling back to the scratchpad. If the checksum type for the distributor is not explicitly set, it should always fallback to the scratchpad.

In other word, the distributor should follow the checksum type of the metadata when it is not explicitly set to use other checksum type.

[1] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/distributors/yum/configuration.py#L345

Associated revisions

Revision 9f230475 View on GitHub
Added by daviddavis over 1 year ago

No longer automatically update the checksum_type on distributors

We were exposing checksum_type via the distributor config which required
automatically setting it. However, we began exposing the repo scratchpad
(which contains the repo's checksum type) in #1172. Setting the
distributor checksum_type automatically causes problems for on-demand
repos when the checksum type changes upstream. Therefore, stop
automatically setting it.

fixes #3736
https://pulp.plan.io/issues/3736

Revision 36d08173 View on GitHub
Added by daviddavis over 1 year ago

No longer automatically update the checksum_type on distributors

We were exposing checksum_type via the distributor config which required
automatically setting it. However, we began exposing the repo scratchpad
(which contains the repo's checksum type) in #1172. Setting the
distributor checksum_type automatically causes problems for on-demand
repos when the checksum type changes upstream. Therefore, stop
automatically setting it.

fixes #3736
https://pulp.plan.io/issues/3736

(cherry picked from commit 9f23047578b658ad352def34007678a47d32eb29)

History

#2 Updated by CodeHeeler over 1 year ago

  • Triaged changed from No to Yes

#3 Updated by dkliban@redhat.com over 1 year ago

The first time Pulp uses a distributor to publish a repository, it checks if the distributor has a checksum type configured. This value determines which checksum type is published with repo metadata. If this value is not set, Pulp looks up the checksum type stored in repository 'scratchpad'. Pulp then configures the distributor with the value found in the scratchpad. This means that if the checksum type changes in the scratchpad because the upstream repository changed what it is publishing, Pulp will continue trying publish with the original checksum type. This is a problem for repositories that use the 'on_demand' download policy. An on_demand repository may be first synced and published using SHA1 checksums. When the upstream repository starts publishing SHA256 checksums, Pulp downloads the new metadata with the SHA256 checksums, but when it tries to publish, it attempts to publish the SHA1 checksums that it no longer has. Since it is an on_demand repository, it can't calculate the SHA1 checksum from files that haven't been downloaded. This is when Pulp emits RPM1008 exception.

#4 Updated by rchan over 1 year ago

  • Sprint set to Sprint 39

#5 Updated by daviddavis over 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to daviddavis

#6 Updated by daviddavis over 1 year ago

My initial thought was to not set the distributor checksum in get_repo_checksum_type(). However, doing some research, I found that this code was created to solve a use case whereby repos are being copied and the checksum type must be preserved. From https://bugzilla.redhat.com/show_bug.cgi?id=1029057#c16:

Currently, the yum importer notices the use of sha1 upstream and makes a note of it in the repo scratchpad. The distributor then sees the note and publishes with sha1. The problem is that when copying content, attributes of the repo (such as its scratchpad) are not copied. This is by design, but makes this particular use case difficult.
The solution is that when the distributor sees the note in the scratchpad, it will look at its own config, and if a checksum has not been set, it will save the checksum in its config. This allows someone who wants to create a copy of a repo to copy the distributor's config, and thus take advantage of this checksum detection.

I'm a bit unsure of how to solve this bug and not break the use case @mhrivnak describes. Thoughts?

#7 Updated by daviddavis over 1 year ago

Looks like we began exposing the repo scratchpad a while ago:

https://pulp.plan.io/issues/1172

Therefore, we should not automatically set the checksum_type on distributor. It looks like we were setting it to expose the value to users. However, it has unintended side effects and now there's another way to view the repository's checksum type (which is set in the scratchpad).

#8 Updated by daviddavis over 1 year ago

  • Status changed from ASSIGNED to POST

#9 Updated by daviddavis over 1 year ago

  • Status changed from POST to MODIFIED

#10 Updated by daviddavis over 1 year ago

  • Platform Release set to 2.16.4

#12 Updated by daviddavis over 1 year ago

  • Status changed from MODIFIED to ON_QA

#13 Updated by daviddavis over 1 year ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#15 Updated by bmbouter 7 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF