Project

Profile

Help

Issue #4246

Story #3693: Lazy for Pulp3

Lazy sync does not create additional RemoteArtifact objects

Added by bmbouter about 2 years ago. Updated about 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Sprint:
Sprint 48
Quarter:

Description

Say you sync a repo with Remote.policy=on_demand and you get unit foo but no Artifact.

Now you sync a different repo w/ a different remote with Remote.policy=on_demand and that repo also contains unit foo. During this second sync the stages API will recognize the unit is already in Pulp and will not create it. It will also not create a RemoteArtifact. It will associate the existing unit foo with the new repo version.

Now delete the first Remote, which will delete all RemoteArtifact objects.

Now a user goes to fetch foo from the second repo. The user will receive a 404 from the content app because when it find the ContentArtifact, and no Artifact to serve it will not find a RemoteArtifact like it should. Thus the content app can only 404.

I expect since there is still a Remote that can remotely fetch that Artifact that if should be fetched.

Overall I think every sync needs to ensure that in the case where a Content unit is already know to Pulp that the Stages pipeline will still create a RemoteArtifact object where today it does not.

Solution

1. have the RemoteArtifact codepath be created even if the ContentUnit exists. (currently it doesn't run at all if .pk != None)
2. Have RemoteArtifact switch to bulk_get_or_create() to handle the IntegrityErrors.


Related issues

Related to Pulp - Story #2507: The streamer needs to try all RemoteArtifact entires related to a ContentArtifactCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Copied to Pulp - Test #4465: Lazy sync does not create additional RemoteArtifact objectsCLOSED - WONTFIX<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

History

#1 Updated by bmbouter about 2 years ago

  • Description updated (diff)

#2 Updated by bmbouter about 2 years ago

I think to fix this, we need to have the ContentUnitSaver stage attempt to create a RemoteArtifact for each ContentArtifact for that remote and handle the IntegrityError. The performance of that is going to suck though, but for correctness I'm not sure what else we can do.

To accopmlish ^, maybe we:
1. have the RemoteArtifact codepath be created even if the ContentUnit exists. (currently it doesn't run at all if .pk != None)
2. Have RemoteArtifact switch to bulk_get_or_create() to handle the IntegrityErrors.

#3 Updated by jortel@redhat.com about 2 years ago

Agreed. A RemoteArtifact should always be created for each DeclarativeArtifact.

#4 Updated by CodeHeeler about 2 years ago

  • Triaged changed from No to Yes
  • Tags Pulp 3 RC Blocker added

#5 Updated by bmbouter about 2 years ago

  • Description updated (diff)
  • Sprint Candidate changed from No to Yes

Adding solution to the main ticket to get it groomed.

#6 Updated by dkliban@redhat.com about 2 years ago

  • Related to Story #2507: The streamer needs to try all RemoteArtifact entires related to a ContentArtifact added

#7 Updated by dkliban@redhat.com about 2 years ago

  • Groomed changed from No to Yes
  • Sprint set to Sprint 46

#8 Updated by rchan about 2 years ago

  • Sprint changed from Sprint 46 to Sprint 47

#9 Updated by jortel@redhat.com about 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com

#10 Updated by jortel@redhat.com about 2 years ago

  • Status changed from ASSIGNED to POST

#11 Updated by jortel@redhat.com about 2 years ago

  • Status changed from POST to ASSIGNED

#12 Updated by jortel@redhat.com about 2 years ago

  • Status changed from ASSIGNED to POST

#13 Updated by rchan almost 2 years ago

  • Sprint changed from Sprint 47 to Sprint 48

#14 Updated by jortel@redhat.com almost 2 years ago

  • Status changed from POST to MODIFIED

Applied in changeset commit:pulpcore-plugin|cbf69d2dd0ffd139915b05f5f7a1f0a4972ad282.

#15 Updated by bherring almost 2 years ago

  • Copied to Test #4465: Lazy sync does not create additional RemoteArtifact objects added

#16 Updated by daviddavis over 1 year ago

  • Sprint/Milestone set to 3.0.0

#17 Updated by bmbouter over 1 year ago

  • Tags deleted (Pulp 3, Pulp 3 RC Blocker)

#18 Updated by bmbouter about 1 year ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF