Project

Profile

Help

Issue #4246

Story #3693: Lazy for Pulp3

Lazy sync does not create additional RemoteArtifact objects

Added by bmbouter 11 months ago. Updated 6 months ago.

Status:
MODIFIED
Priority:
Normal
Category:
-
Sprint/Milestone:
Start date:
Due date:
Severity:
2. Medium
Version:
Platform Release:
Blocks Release:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 48

Description

Say you sync a repo with Remote.policy=on_demand and you get unit foo but no Artifact.

Now you sync a different repo w/ a different remote with Remote.policy=on_demand and that repo also contains unit foo. During this second sync the stages API will recognize the unit is already in Pulp and will not create it. It will also not create a RemoteArtifact. It will associate the existing unit foo with the new repo version.

Now delete the first Remote, which will delete all RemoteArtifact objects.

Now a user goes to fetch foo from the second repo. The user will receive a 404 from the content app because when it find the ContentArtifact, and no Artifact to serve it will not find a RemoteArtifact like it should. Thus the content app can only 404.

I expect since there is still a Remote that can remotely fetch that Artifact that if should be fetched.

Overall I think every sync needs to ensure that in the case where a Content unit is already know to Pulp that the Stages pipeline will still create a RemoteArtifact object where today it does not.

Solution

1. have the RemoteArtifact codepath be created even if the ContentUnit exists. (currently it doesn't run at all if .pk != None)
2. Have RemoteArtifact switch to bulk_get_or_create() to handle the IntegrityErrors.


Related issues

Related to Pulp - Story #2507: The streamer needs to try all RemoteArtifact entires related to a ContentArtifact MODIFIED Actions
Copied to Pulp - Test #4465: Lazy sync does not create additional RemoteArtifact objects NEW Actions

Associated revisions

Revision cbf69d2d View on GitHub
Added by jortel@redhat.com 8 months ago

Add separate remote artifact stage.
closes #4246

History

#1 Updated by bmbouter 11 months ago

  • Description updated (diff)

#2 Updated by bmbouter 11 months ago

I think to fix this, we need to have the ContentUnitSaver stage attempt to create a RemoteArtifact for each ContentArtifact for that remote and handle the IntegrityError. The performance of that is going to suck though, but for correctness I'm not sure what else we can do.

To accopmlish ^, maybe we:
1. have the RemoteArtifact codepath be created even if the ContentUnit exists. (currently it doesn't run at all if .pk != None)
2. Have RemoteArtifact switch to bulk_get_or_create() to handle the IntegrityErrors.

#3 Updated by jortel@redhat.com 10 months ago

Agreed. A RemoteArtifact should always be created for each DeclarativeArtifact.

#4 Updated by CodeHeeler 10 months ago

  • Triaged changed from No to Yes
  • Tags Pulp 3 RC Blocker added

#5 Updated by bmbouter 10 months ago

  • Description updated (diff)
  • Sprint Candidate changed from No to Yes

Adding solution to the main ticket to get it groomed.

#6 Updated by dkliban@redhat.com 10 months ago

  • Related to Story #2507: The streamer needs to try all RemoteArtifact entires related to a ContentArtifact added

#7 Updated by dkliban@redhat.com 10 months ago

  • Groomed changed from No to Yes
  • Sprint set to Sprint 46

#8 Updated by rchan 10 months ago

  • Sprint changed from Sprint 46 to Sprint 47

#9 Updated by jortel@redhat.com 10 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com

#10 Updated by jortel@redhat.com 10 months ago

  • Status changed from ASSIGNED to POST

#11 Updated by jortel@redhat.com 9 months ago

  • Status changed from POST to ASSIGNED

#12 Updated by jortel@redhat.com 9 months ago

  • Status changed from ASSIGNED to POST

#13 Updated by rchan 9 months ago

  • Sprint changed from Sprint 47 to Sprint 48

#14 Updated by jortel@redhat.com 8 months ago

  • Status changed from POST to MODIFIED

#15 Updated by bherring 8 months ago

  • Copied to Test #4465: Lazy sync does not create additional RemoteArtifact objects added

#16 Updated by daviddavis 6 months ago

  • Sprint/Milestone set to 3.0

#17 Updated by bmbouter 6 months ago

  • Tags deleted (Pulp 3, Pulp 3 RC Blocker)

Please register to edit this issue

Also available in: Atom PDF