Project

Profile

Help

Story #3693

Lazy for Pulp3

Added by bmbouter over 1 year ago. Updated 6 months ago.

Status:
MODIFIED
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

84%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
No
Sprint Candidate:
No
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:

Description

Motivation

Pulp3 users would greatly benefit from the "lazy" feature capabilities that Pulp2 RPM users enjoy. Unfortunately that support was specific to Pulp2 RPM. In Pulp2 we are motivated to introduce this feature and do it in a way that places minimal requirements on plugin writers.

Usage

The user can specify to use "lazy" when syncing any given Remote by setting the policy attribute on the Remote. The default policy is "immediate", so to use lazy this option must be specified explicitly. Lazy behavior is not the default because not all plugins can support it, but all plugins can support "immediate".

Here are the modes of lazy operation:

policy=immediate  -> downloads now while the task runs (no lazy). Also the default if unspecified.
policy=on_demand   -> All the steps in the diagram. Content that is downloaded is saved so that it's only ever downloaded once.
policy=streamed     -> All the steps in the diagram except step the saving of the Artifact. If squid pushes the bits out of the cache, it will be re-downloaded again to serve to other clients requesting the same bits.

A plugin must specifically add "lazy support", so verify that a plugin supports the lazy feature set.

Plugin Involvement

To add "lazy support" to any given Pulp3 plugin, there are a few things a plugin writer must do.

1. Add/Remove RemoteArtifact as needed to reflect the artifacts in the remote repository.
2. Check the Remote.policy attribute. If 'immediate' do the normal thing. If 'streamed' or 'on_demand' continue to step 3.
3. For each Artifact object, instead of downloading and saving it locally, save the equivalent RemoteArtifact instead and associate that with the ContentArtifact object.

Once supported, plugin writer's will be able to use both streamed and on_demand options.

Design

See the call sequence diagram below:

lazy diagram revised after streamer + content app merged

^ diagram built by this image builder page using this gist data.

lazy_diagram_v4.png (86.3 KB) lazy diagram revised after streamer + content app merged bmbouter, 12/13/2018 10:57 PM lazy_diagram_v4.png
250

Subtasks

Python Support - Story #1884: As a user, I can lazily sync python packagesMODIFIEDCodeHeeler

Actions
Story #2507: The streamer needs to try all RemoteArtifact entires related to a ContentArtifactMODIFIEDbmbouter

Actions
Story #3320: Support On_demand and Background downloadersCLOSED - DUPLICATE

Actions
Task #3694: Document how to install and configure squidCLOSED - WONTFIX

Actions
Task #3695: Document how to add lazy to a pluginMODIFIEDbmbouter

Actions
Task #3696: Add the streamer's settings to settings.yaml in the content app areaMODIFIEDjortel@redhat.com

Actions
Task #3697: Have Pulp do a simple redirect (no signing) to squid when the content app needs to serve a RemoteArtifactMODIFIEDjortel@redhat.com

Actions
Task #3698: Make the streamer download using a configured downloader, configured by the correct remote.MODIFIEDbmbouter

Actions
Task #3699: Make the streamer create an Artifact and update ContentArtifact with the foreign keyMODIFIEDbmbouter

Actions
Task #3763: Add "policy" attribute to RemoteMODIFIEDbmbouter

Actions
Task #3776: Bootstrap streamerCLOSED - WONTFIXbmbouter

Actions
Story #3894: As a user, I can use Pulp as a pass-through cacheMODIFIEDdkliban@redhat.com

Actions
File Support - Task #4160: Add lazy sync to pulp_fileMODIFIEDbmbouter

Actions
Story #4181: As a streamer user, I have content protection like in Pulp itselfMODIFIEDbmbouter

Actions
Story #4209: As a plugin writer, I can implement lazy sync for plugins with multiple metadata filesMODIFIEDmdellweg

Actions
RPM Support - Story #4212: As a user, I have lazy syncMODIFIEDdalley

Actions
Story #4218: As a user, I have documentation on the lazy-sync workflowMODIFIEDbmbouter

Actions
Story #4239: merge the content-app & streamerMODIFIEDbmbouter

Actions
Task #4243: Rename 'cache_only' policyMODIFIEDbmbouter

Actions
Issue #4246: Lazy sync does not create additional RemoteArtifact objectsMODIFIEDjortel@redhat.comActions
File Support - Issue #4249: README has example of policy='cache_only' but no info about the streamerCLOSED - WORKSFORMEActions
Story #4273: As a plugin writer, I can import the Content app, subclass it, and register that with the aiohttp app instance which is a singletonMODIFIEDbmbouter

Actions
Issue #4289: Content delivery breaks if remote is deletedMODIFIEDbmbouterActions
Task #4302: Ansible installer for Pulp3 needs to add a systemd unit file for the pulp_content_app serviceMODIFIEDamacdona@redhat.com

Actions
Test #4362: Ansible installer for Pulp3 needs to add a systemd unit file for the pulp_content_app serviceCLOSED - COMPLETEActions

Related issues

Related to Pulp - Test #4126: Test sync with different download policies CLOSED - COMPLETE Actions
Related to Pulp - Test #4496: Test that one can consume content from Pulp using lazy sync CLOSED - COMPLETE Actions

History

#1 Updated by bmbouter over 1 year ago

  • File lazy_diagram_v1.png added
  • Description updated (diff)

Adding sequence diagram.

#2 Updated by bmbouter over 1 year ago

  • File lazy_diagram_v2.png added
  • Description updated (diff)

#3 Updated by bmbouter over 1 year ago

  • File deleted (lazy_diagram_v1.png)

#4 Updated by bmbouter over 1 year ago

  • File lazy_diagram_v3.png added
  • File deleted (lazy_diagram_v2.png)
  • Description updated (diff)

#5 Updated by amacdona@redhat.com over 1 year ago

Based on the principle of least surprise, I'd like to request that we explictly state this goal:

As a user, on-demand and immediate syncs are eventually consistent. (Assuming all content is requested from Pulp)

Specifically, I am hoping that when the upstream content is missing, it is handled the same way. With Pulp 2 IIRC, if there was a 404, lazy kept the content but immediate removed the unit. It would be very cool if we could do this in way that makes this guarantee for all plugins.

#6 Updated by jortel@redhat.com over 1 year ago

  • Description updated (diff)

#7 Updated by jortel@redhat.com over 1 year ago

  • Description updated (diff)

#8 Updated by jortel@redhat.com over 1 year ago

  • Description updated (diff)

#9 Updated by bmbouter over 1 year ago

  • Description updated (diff)

Updates to reflect the new names of these options and their changes in feature set.

#10 Updated by bmbouter over 1 year ago

  • Description updated (diff)

#11 Updated by bmbouter over 1 year ago

  • Description updated (diff)

#12 Updated by bmbouter over 1 year ago

  • Description updated (diff)

#13 Updated by CodeHeeler about 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to CodeHeeler

#14 Updated by CodeHeeler 12 months ago

  • Assignee changed from CodeHeeler to bmbouter

#15 Updated by kersom 12 months ago

  • Related to Test #4126: Test sync with different download policies added

#16 Updated by dalley 11 months ago

Without having created an issue for it, this PR will add lazy support to the plugin template (but no documentation changes).

https://github.com/pulp/plugin_template/pull/17

#17 Updated by bmbouter 10 months ago

  • File deleted (lazy_diagram_v3.png)

#19 Updated by bmbouter 10 months ago

  • Description updated (diff)

#20 Updated by bmbouter 10 months ago

  • Description updated (diff)

#21 Updated by bmbouter 9 months ago

  • Status changed from ASSIGNED to MODIFIED

I'm closing this epic. It only has 1 story which is likely to finish today. The remaining open items are exclusively test issues or "enable for plugin X" issues. Those can be handled outside of this feature epic.

#22 Updated by kersom 8 months ago

  • Related to Test #4496: Test that one can consume content from Pulp using lazy sync added

#23 Updated by daviddavis 6 months ago

  • Sprint/Milestone set to 3.0

#24 Updated by bmbouter 6 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF