Story #1150: As a user, I can lazily fetch repositories
Add a generic task for a lazy-sync "active" that the plugins can dispatch
Assuming that the lazy_sync key on the importer is set to "active", after each plugin builds the catalog it will need to initiate an asynchronous task which will download all the cataloged units asynchronously and save them. This task is to create this dispatchable task. When the task is dispatched it needs to save the asynchronous download as a spawned task for further tracking by the client.
This is similar to the "download_unit" task but for all units. This should not use the download_unit task because the concurrency will cause all of Pulp to wait until the repo is synchronized before any more work can continue due to per-unit reservations. This task should download all units and save them within its own task code by making calls into the plugin on a unit-by-unit basis.
Updated by email@example.com almost 7 years ago
- collection = downloads
- path (unique)
The download-one: just inserts a download record.
The download-repo: inserts a download record for each path associated with each content unit in a repo.
A scheduled celery task runs each 30 min and queries for pending downloads ordered by object ID. if any, it groups 10 into a batch, updates the batch_id. Next it queries for batches that still have pending downloads and queues a task to download the batch with reservation. The task queries the catalog by path and concurrently downloads the files. On success, set downloaded=True.
The filesystem is checked (stat) to see if the file exists at every opportunity.
The downloads table is purged during orphan purge.