Project

Profile

Help

Task #1199

closed

Story #1150: As a user, I can lazily fetch repositories

Add a generic task for a lazy-sync "active" that the plugins can dispatch

Added by bmbouter over 8 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.8.0
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Assuming that the lazy_sync key on the importer is set to "active", after each plugin builds the catalog it will need to initiate an asynchronous task which will download all the cataloged units asynchronously and save them. This task is to create this dispatchable task. When the task is dispatched it needs to save the asynchronous download as a spawned task for further tracking by the client.

This is similar to the "download_unit" task but for all units. This should not use the download_unit task because the concurrency will cause all of Pulp to wait until the repo is synchronized before any more work can continue due to per-unit reservations. This task should download all units and save them within its own task code by making calls into the plugin on a unit-by-unit basis.


Related issues

Blocks Pulp - Task #1200: Make all plugin sync importers and plugin distributors lazy awareCLOSED - WONTFIX

Actions
Actions #1

Updated by bmbouter over 8 years ago

  • Blocks Task #1200: Make all plugin sync importers and plugin distributors lazy aware added
Actions #2

Updated by jcline@redhat.com over 8 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jcline@redhat.com
Actions #3

Updated by jortel@redhat.com over 8 years ago

Download:

  • collection = downloads
  • path (unique)
  • unit_id
  • unit_type_id
  • downloaded
  • batch_id

The download-one: just inserts a download record.
The download-repo: inserts a download record for each path associated with each content unit in a repo.

A scheduled celery task runs each 30 min and queries for pending downloads ordered by object ID. if any, it groups 10 into a batch, updates the batch_id. Next it queries for batches that still have pending downloads and queues a task to download the batch with reservation. The task queries the catalog by path and concurrently downloads the files. On success, set downloaded=True.

The filesystem is checked (stat) to see if the file exists at every opportunity.

The downloads table is purged during orphan purge.

Actions #4

Updated by jcline@redhat.com about 8 years ago

  • Status changed from ASSIGNED to POST

Added by Jeremy Cline about 8 years ago

Revision caa2d775 | View on GitHub

Create tasks to download lazy repository content.

This introduces two new tasks, one to download deferred content units, and one to download a repository using the lazy catalog.

closes #1199

Added by Jeremy Cline about 8 years ago

Revision caa2d775 | View on GitHub

Create tasks to download lazy repository content.

This introduces two new tasks, one to download deferred content units, and one to download a repository using the lazy catalog.

closes #1199

Actions #5

Updated by Anonymous about 8 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #6

Updated by rbarlow about 8 years ago

  • Status changed from MODIFIED to 5
  • Platform Release set to 2.8.0
Actions #7

Updated by dkliban@redhat.com almost 8 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #8

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF