Lazy for Pulp3
Pulp3 users would greatly benefit from the "lazy" feature capabilities that Pulp2 RPM users enjoy. Unfortunately that support was specific to Pulp2 RPM. In Pulp2 we are motivated to introduce this feature and do it in a way that places minimal requirements on plugin writers.
The user can specify to use "lazy" when syncing any given Remote by setting the
policy attribute on the Remote. The default
policy is "immediate", so to use lazy this option must be specified explicitly. Lazy behavior is not the default because not all plugins can support it, but all plugins can support "immediate".
Here are the modes of lazy operation:
policy=immediate -> downloads now while the task runs (no lazy). Also the default if unspecified.
policy=on_demand -> All the steps in the diagram. Content that is downloaded is saved so that it's only ever downloaded once.
policy=streamed -> All the steps in the diagram except step the saving of the Artifact. If squid pushes the bits out of the cache, it will be re-downloaded again to serve to other clients requesting the same bits.
A plugin must specifically add "lazy support", so verify that a plugin supports the lazy feature set.
To add "lazy support" to any given Pulp3 plugin, there are a few things a plugin writer must do.
RemoteArtifact as needed to reflect the artifacts in the remote repository.
2. Check the Remote.policy attribute. If 'immediate' do the normal thing. If 'streamed' or 'on_demand' continue to step 3.
3. For each
Artifact object, instead of downloading and saving it locally, save the equivalent
RemoteArtifact instead and associate that with the
Once supported, plugin writer's will be able to use both
See the call sequence diagram below:
Updated by firstname.lastname@example.org over 5 years ago
Based on the principle of least surprise, I'd like to request that we explictly state this goal:
As a user, on-demand and immediate syncs are eventually consistent. (Assuming all content is requested from Pulp)
Specifically, I am hoping that when the upstream content is missing, it is handled the same way. With Pulp 2 IIRC, if there was a 404, lazy kept the content but immediate removed the unit. It would be very cool if we could do this in way that makes this guarantee for all plugins.
Updated by bmbouter about 5 years ago
- Status changed from ASSIGNED to MODIFIED
I'm closing this epic. It only has 1 story which is likely to finish today. The remaining open items are exclusively test issues or "enable for plugin X" issues. Those can be handled outside of this feature epic.