Project

Profile

Help

Issue #4246

closed

Story #3693: Lazy for Pulp3

Lazy sync does not create additional RemoteArtifact objects

Added by bmbouter over 5 years ago. Updated over 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Sprint:
Sprint 48
Quarter:

Description

Say you sync a repo with Remote.policy=on_demand and you get unit foo but no Artifact.

Now you sync a different repo w/ a different remote with Remote.policy=on_demand and that repo also contains unit foo. During this second sync the stages API will recognize the unit is already in Pulp and will not create it. It will also not create a RemoteArtifact. It will associate the existing unit foo with the new repo version.

Now delete the first Remote, which will delete all RemoteArtifact objects.

Now a user goes to fetch foo from the second repo. The user will receive a 404 from the content app because when it find the ContentArtifact, and no Artifact to serve it will not find a RemoteArtifact like it should. Thus the content app can only 404.

I expect since there is still a Remote that can remotely fetch that Artifact that if should be fetched.

Overall I think every sync needs to ensure that in the case where a Content unit is already know to Pulp that the Stages pipeline will still create a RemoteArtifact object where today it does not.

Solution

1. have the RemoteArtifact codepath be created even if the ContentUnit exists. (currently it doesn't run at all if .pk != None)
2. Have RemoteArtifact switch to bulk_get_or_create() to handle the IntegrityErrors.


Related issues

Related to Pulp - Story #2507: The streamer needs to try all RemoteArtifact entires related to a ContentArtifactCLOSED - CURRENTRELEASEbmbouter

Actions
Copied to Pulp - Test #4465: Lazy sync does not create additional RemoteArtifact objectsCLOSED - WONTFIXActions
Actions #1

Updated by bmbouter over 5 years ago

  • Description updated (diff)
Actions #2

Updated by bmbouter over 5 years ago

I think to fix this, we need to have the ContentUnitSaver stage attempt to create a RemoteArtifact for each ContentArtifact for that remote and handle the IntegrityError. The performance of that is going to suck though, but for correctness I'm not sure what else we can do.

To accopmlish ^, maybe we:
1. have the RemoteArtifact codepath be created even if the ContentUnit exists. (currently it doesn't run at all if .pk != None)
2. Have RemoteArtifact switch to bulk_get_or_create() to handle the IntegrityErrors.

Actions #3

Updated by jortel@redhat.com over 5 years ago

Agreed. A RemoteArtifact should always be created for each DeclarativeArtifact.

Actions #4

Updated by CodeHeeler over 5 years ago

  • Triaged changed from No to Yes
  • Tags Pulp 3 RC Blocker added
Actions #5

Updated by bmbouter over 5 years ago

  • Description updated (diff)
  • Sprint Candidate changed from No to Yes

Adding solution to the main ticket to get it groomed.

Actions #6

Updated by dkliban@redhat.com over 5 years ago

  • Related to Story #2507: The streamer needs to try all RemoteArtifact entires related to a ContentArtifact added
Actions #7

Updated by dkliban@redhat.com over 5 years ago

  • Groomed changed from No to Yes
  • Sprint set to Sprint 46
Actions #8

Updated by rchan over 5 years ago

  • Sprint changed from Sprint 46 to Sprint 47
Actions #9

Updated by jortel@redhat.com over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com
Actions #10

Updated by jortel@redhat.com over 5 years ago

  • Status changed from ASSIGNED to POST
Actions #11

Updated by jortel@redhat.com over 5 years ago

  • Status changed from POST to ASSIGNED
Actions #12

Updated by jortel@redhat.com over 5 years ago

  • Status changed from ASSIGNED to POST
Actions #13

Updated by rchan about 5 years ago

  • Sprint changed from Sprint 47 to Sprint 48
Actions #14

Updated by jortel@redhat.com about 5 years ago

  • Status changed from POST to MODIFIED

Applied in changeset commit:pulpcore-plugin|cbf69d2dd0ffd139915b05f5f7a1f0a4972ad282.

Actions #15

Updated by bherring about 5 years ago

  • Copied to Test #4465: Lazy sync does not create additional RemoteArtifact objects added
Actions #16

Updated by daviddavis about 5 years ago

  • Sprint/Milestone set to 3.0.0
Actions #17

Updated by bmbouter almost 5 years ago

  • Tags deleted (Pulp 3, Pulp 3 RC Blocker)
Actions #18

Updated by bmbouter over 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF