Project

Profile

Help

Task #1199

Story #1150: As a user, I can lazily fetch repositories

Add a generic task for a lazy-sync "active" that the plugins can dispatch

Added by bmbouter about 6 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.8.0
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Assuming that the lazy_sync key on the importer is set to "active", after each plugin builds the catalog it will need to initiate an asynchronous task which will download all the cataloged units asynchronously and save them. This task is to create this dispatchable task. When the task is dispatched it needs to save the asynchronous download as a spawned task for further tracking by the client.

This is similar to the "download_unit" task but for all units. This should not use the download_unit task because the concurrency will cause all of Pulp to wait until the repo is synchronized before any more work can continue due to per-unit reservations. This task should download all units and save them within its own task code by making calls into the plugin on a unit-by-unit basis.


Related issues

Blocks Pulp - Task #1200: Make all plugin sync importers and plugin distributors lazy awareCLOSED - WONTFIX

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision caa2d775 View on GitHub
Added by Jeremy Cline over 5 years ago

Create tasks to download lazy repository content.

This introduces two new tasks, one to download deferred content units, and one to download a repository using the lazy catalog.

closes #1199

Revision caa2d775 View on GitHub
Added by Jeremy Cline over 5 years ago

Create tasks to download lazy repository content.

This introduces two new tasks, one to download deferred content units, and one to download a repository using the lazy catalog.

closes #1199

History

#1 Updated by bmbouter about 6 years ago

  • Blocks Task #1200: Make all plugin sync importers and plugin distributors lazy aware added

#2 Updated by jcline@redhat.com almost 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jcline@redhat.com

#3 Updated by jortel@redhat.com almost 6 years ago

Download:

  • collection = downloads
  • path (unique)
  • unit_id
  • unit_type_id
  • downloaded
  • batch_id

The download-one: just inserts a download record.
The download-repo: inserts a download record for each path associated with each content unit in a repo.

A scheduled celery task runs each 30 min and queries for pending downloads ordered by object ID. if any, it groups 10 into a batch, updates the batch_id. Next it queries for batches that still have pending downloads and queues a task to download the batch with reservation. The task queries the catalog by path and concurrently downloads the files. On success, set downloaded=True.

The filesystem is checked (stat) to see if the file exists at every opportunity.

The downloads table is purged during orphan purge.

#4 Updated by jcline@redhat.com over 5 years ago

  • Status changed from ASSIGNED to POST

#5 Updated by Anonymous over 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#6 Updated by rbarlow over 5 years ago

  • Status changed from MODIFIED to 5
  • Platform Release set to 2.8.0

#7 Updated by dkliban@redhat.com over 5 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#8 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF