Task #1199
closed
Story #1150: As a user, I can lazily fetch repositories
Add a generic task for a lazy-sync "active" that the plugins can dispatch
Status:
CLOSED - CURRENTRELEASE
Description
Assuming that the lazy_sync key on the importer is set to "active", after each plugin builds the catalog it will need to initiate an asynchronous task which will download all the cataloged units asynchronously and save them. This task is to create this dispatchable task. When the task is dispatched it needs to save the asynchronous download as a spawned task for further tracking by the client.
This is similar to the "download_unit" task but for all units. This should not use the download_unit task because the concurrency will cause all of Pulp to wait until the repo is synchronized before any more work can continue due to per-unit reservations. This task should download all units and save them within its own task code by making calls into the plugin on a unit-by-unit basis.
- Blocks Task #1200: Make all plugin sync importers and plugin distributors lazy aware added
- Status changed from NEW to ASSIGNED
- Assignee set to jcline@redhat.com
Download:
- collection = downloads
- path (unique)
- unit_id
- unit_type_id
- downloaded
- batch_id
The download-one: just inserts a download record.
The download-repo: inserts a download record for each path associated with each content unit in a repo.
A scheduled celery task runs each 30 min and queries for pending downloads ordered by object ID. if any, it groups 10 into a batch, updates the batch_id. Next it queries for batches that still have pending downloads and queues a task to download the batch with reservation. The task queries the catalog by path and concurrently downloads the files. On success, set downloaded=True.
The filesystem is checked (stat) to see if the file exists at every opportunity.
The downloads table is purged during orphan purge.
- Status changed from ASSIGNED to POST
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
- Status changed from MODIFIED to 5
- Platform Release set to 2.8.0
- Status changed from 5 to CLOSED - CURRENTRELEASE
Also available in: Atom
PDF
Create tasks to download lazy repository content.
This introduces two new tasks, one to download deferred content units, and one to download a repository using the lazy catalog.
closes #1199