Project

Profile

Help

Issue #3736

closed

When the checksum type of the upstream repository changed, syncing a "on_demand" repository will fail

Added by hyu over 6 years ago. Updated over 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.16.4
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 39
Quarter:

Description

When the checksum type of the upstream repository changed, syncing a "on_demand" repository will fail with the following error:
------------
Checksum type "sha1" is not available for all units in the repository. Make sure those units have been downloaded
------------

When syncing a repository, Pulp will update the checksum type in the repo scratchpad to match the checksum type of the upstream repository, but the checksum_type configuration in the distributor will remain no change. This will cause the above error due to data/configuration inconsistency.

The "get_repo_checksum_type" function in [1] will attempt to save the checksum type to the distributor configuration if it is not already set. In my opinion, It should not save the checksum type after falling back to the scratchpad. If the checksum type for the distributor is not explicitly set, it should always fallback to the scratchpad.

In other word, the distributor should follow the checksum type of the metadata when it is not explicitly set to use other checksum type.

[1] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/distributors/yum/configuration.py#L345

Actions #2

Updated by CodeHeeler over 6 years ago

  • Triaged changed from No to Yes
Actions #3

Updated by dkliban@redhat.com over 6 years ago

The first time Pulp uses a distributor to publish a repository, it checks if the distributor has a checksum type configured. This value determines which checksum type is published with repo metadata. If this value is not set, Pulp looks up the checksum type stored in repository 'scratchpad'. Pulp then configures the distributor with the value found in the scratchpad. This means that if the checksum type changes in the scratchpad because the upstream repository changed what it is publishing, Pulp will continue trying publish with the original checksum type. This is a problem for repositories that use the 'on_demand' download policy. An on_demand repository may be first synced and published using SHA1 checksums. When the upstream repository starts publishing SHA256 checksums, Pulp downloads the new metadata with the SHA256 checksums, but when it tries to publish, it attempts to publish the SHA1 checksums that it no longer has. Since it is an on_demand repository, it can't calculate the SHA1 checksum from files that haven't been downloaded. This is when Pulp emits RPM1008 exception.

Actions #4

Updated by rchan over 6 years ago

  • Sprint set to Sprint 39
Actions #5

Updated by daviddavis over 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to daviddavis
Actions #6

Updated by daviddavis over 6 years ago

My initial thought was to not set the distributor checksum in get_repo_checksum_type(). However, doing some research, I found that this code was created to solve a use case whereby repos are being copied and the checksum type must be preserved. From https://bugzilla.redhat.com/show_bug.cgi?id=1029057#c16:

Currently, the yum importer notices the use of sha1 upstream and makes a note of it in the repo scratchpad. The distributor then sees the note and publishes with sha1. The problem is that when copying content, attributes of the repo (such as its scratchpad) are not copied. This is by design, but makes this particular use case difficult.
The solution is that when the distributor sees the note in the scratchpad, it will look at its own config, and if a checksum has not been set, it will save the checksum in its config. This allows someone who wants to create a copy of a repo to copy the distributor's config, and thus take advantage of this checksum detection.

I'm a bit unsure of how to solve this bug and not break the use case mhrivnak describes. Thoughts?

Actions #7

Updated by daviddavis over 6 years ago

Looks like we began exposing the repo scratchpad a while ago:

https://pulp.plan.io/issues/1172

Therefore, we should not automatically set the checksum_type on distributor. It looks like we were setting it to expose the value to users. However, it has unintended side effects and now there's another way to view the repository's checksum type (which is set in the scratchpad).

Actions #8

Updated by daviddavis over 6 years ago

  • Status changed from ASSIGNED to POST

Added by daviddavis over 6 years ago

Revision 9f230475 | View on GitHub

No longer automatically update the checksum_type on distributors

We were exposing checksum_type via the distributor config which required automatically setting it. However, we began exposing the repo scratchpad (which contains the repo's checksum type) in #1172. Setting the distributor checksum_type automatically causes problems for on-demand repos when the checksum type changes upstream. Therefore, stop automatically setting it.

fixes #3736 https://pulp.plan.io/issues/3736

Actions #9

Updated by daviddavis over 6 years ago

  • Status changed from POST to MODIFIED
Actions #10

Updated by daviddavis over 6 years ago

  • Platform Release set to 2.16.4

Added by daviddavis over 6 years ago

Revision 36d08173 | View on GitHub

No longer automatically update the checksum_type on distributors

We were exposing checksum_type via the distributor config which required automatically setting it. However, we began exposing the repo scratchpad (which contains the repo's checksum type) in #1172. Setting the distributor checksum_type automatically causes problems for on-demand repos when the checksum type changes upstream. Therefore, stop automatically setting it.

fixes #3736 https://pulp.plan.io/issues/3736

(cherry picked from commit 9f23047578b658ad352def34007678a47d32eb29)

Actions #11

Updated by daviddavis over 6 years ago

Actions #12

Updated by daviddavis over 6 years ago

  • Status changed from MODIFIED to 5
Actions #13

Updated by daviddavis over 6 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #15

Updated by bmbouter over 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF