Project

Profile

Help

Task #1200

Story #1150: As a user, I can lazily fetch repositories

Make all plugin sync importers and plugin distributors lazy aware

Added by bmbouter about 5 years ago. Updated over 1 year ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Start date:
Due date:
% Done:

10%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Each repo has 'lazy passive' or 'lazy active' enabled by setting a key named 'lazy_sync' whose value takes on either 'active' or 'passive'. These configs are read by each plugin's implementation of the 'sync' importer.

Plugin updates

The following two things will need to be done in each plugins. <sad face>

  • Build "the unit catalog"
  • If lazy_sync is set at to any value, sync the metadata and do not download non-metadata units. This will need to be done in each plugin's implementation of the sync handler. FYI, the plugin specific code is called by the platform sync task here.

Platform updates

The handling of the lazy_sync = active case can be 100% in platform code. The task it is calling (#1199) is plugin agnostic, and also all it needs to do is dispatch the task and save the spawned task on the TaskResult for the sync. I recommend doing this here. One important point is that the publish spawned task needs to be listed before the generic #1199 download task. If it is not done in this order then the CLI won't show sync, then publish, then the status of the generic #1199. Instead it would should sync, a long delay while #1199 completes, and then the publish which on a large repo sync would not be a great CLI experience.


Checklist


Related issues

Blocked by Pulp - Task #1199: Add a generic task for a lazy-sync "active" that the plugins can dispatchCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 039329ce View on GitHub
Added by jortel@redhat.com about 5 years ago

ref #1200 - add lazy properties to import configuration.

Revision 039329ce View on GitHub
Added by jortel@redhat.com about 5 years ago

ref #1200 - add lazy properties to import configuration.

History

#1 Updated by bmbouter about 5 years ago

Should the 'lazy active' case publish the same as 'lazy passive' and have the asynchronous download task provided by #1199 kicked off during the publish operation?

If so can the publish distributor even access this given that these config values are stored in the sync importer's config?

#2 Updated by bmbouter about 5 years ago

  • Parent task set to #1150

#3 Updated by bmbouter about 5 years ago

  • Description updated (diff)

#4 Updated by bmbouter about 5 years ago

  • Description updated (diff)

#5 Updated by bmbouter about 5 years ago

  • Description updated (diff)

#6 Updated by bmbouter about 5 years ago

  • Blocked by Task #1199: Add a generic task for a lazy-sync "active" that the plugins can dispatch added

#7 Updated by bmbouter about 5 years ago

bmbouter wrote:

Should the 'lazy active' case publish the same as 'lazy passive' and have the asynchronous download task provided by #1199 kicked off during the publish operation?

If so can the publish distributor even access this given that these config values are stored in the sync importer's config?

After talking with bcourt we've determined this can be kicked off as a spawned task after a sync. It's not related to a publish at all and can occur concurrently. The publish will publish its links the same regardless of the amount of units the asynchronous task has done. I've updated the story to reflect this understanding.

#8 Updated by jortel@redhat.com about 5 years ago

Added importer configuration properties for lazy. https://github.com/pulp/pulp/pull/2085

#9 Updated by jortel@redhat.com about 5 years ago

  • % Done changed from 0 to 10

#10 Updated by jortel@redhat.com about 5 years ago

  • Checklist item changed from to [x] Importer configuration constants, [x] Importer configuration validation

#11 Updated by jortel@redhat.com about 5 years ago

The celery task needs to skip files that have already been downloaded.

#12 Updated by bmbouter over 1 year ago

  • Status changed from NEW to CLOSED - WONTFIX

#13 Updated by bmbouter over 1 year ago

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

#14 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF