Task #1199
closedStory #1150: As a user, I can lazily fetch repositories
Add a generic task for a lazy-sync "active" that the plugins can dispatch
100%
Description
Assuming that the lazy_sync key on the importer is set to "active", after each plugin builds the catalog it will need to initiate an asynchronous task which will download all the cataloged units asynchronously and save them. This task is to create this dispatchable task. When the task is dispatched it needs to save the asynchronous download as a spawned task for further tracking by the client.
This is similar to the "download_unit" task but for all units. This should not use the download_unit task because the concurrency will cause all of Pulp to wait until the repo is synchronized before any more work can continue due to per-unit reservations. This task should download all units and save them within its own task code by making calls into the plugin on a unit-by-unit basis.
Related issues
Updated by bmbouter over 9 years ago
- Blocks Task #1200: Make all plugin sync importers and plugin distributors lazy aware added
Updated by jcline@redhat.com about 9 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to jcline@redhat.com
Updated by jortel@redhat.com about 9 years ago
Download:
- collection = downloads
- path (unique)
- unit_id
- unit_type_id
- downloaded
- batch_id
The download-one: just inserts a download record.
The download-repo: inserts a download record for each path associated with each content unit in a repo.
A scheduled celery task runs each 30 min and queries for pending downloads ordered by object ID. if any, it groups 10 into a batch, updates the batch_id. Next it queries for batches that still have pending downloads and queues a task to download the batch with reservation. The task queries the catalog by path and concurrently downloads the files. On success, set downloaded=True.
The filesystem is checked (stat) to see if the file exists at every opportunity.
The downloads table is purged during orphan purge.
Updated by jcline@redhat.com almost 9 years ago
- Status changed from ASSIGNED to POST
Added by Jeremy Cline almost 9 years ago
Added by Jeremy Cline almost 9 years ago
Revision caa2d775 | View on GitHub
Create tasks to download lazy repository content.
This introduces two new tasks, one to download deferred content units, and one to download a repository using the lazy catalog.
closes #1199
Updated by Anonymous almost 9 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset pulp|caa2d7752c715d4b9ea350680996e1a1a3e2c664.
Updated by rbarlow almost 9 years ago
- Status changed from MODIFIED to 5
- Platform Release set to 2.8.0
Updated by dkliban@redhat.com over 8 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
Create tasks to download lazy repository content.
This introduces two new tasks, one to download deferred content units, and one to download a repository using the lazy catalog.
closes #1199