Project

Profile

Help

Story #1150

closed

As a user, I can lazily fetch repositories

Added by bmbouter over 9 years ago. Updated over 5 years ago.

Status:
CLOSED - NOTABUG
Priority:
High
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
(Total: 0:00 h)
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

As a user, I can choose to sync a repository lazily. When lazy syncing is enabled for a specific repo, any associated metadata will be synced and published before any units are downloaded. If there is no metadata (ie: Python) then the sync+publish will be a no-op. Lazy syncing will be an optional feature and will not modify the normal sync workflow at all. Lazy sync will be configurable as a per-repo setting. Lazy sync will have two modes:

  • lazy sync passive - This mode creates the lazy repo and will not fetch content units until an actual client requests them.
  • lazy sync active - This mode creates the lazy repo and fetches content units using an asynchronous task. Missing content units are fetched lazily as in passive mode during the asynchronous sync so the repo can be used right away.

This is being introduced as a platform feature so all content types will have support. This is not being integrated with nodes. This is a tracker story and tracks the required work as child tasks/stories/refactors.


Sub-issues 42 (0 open42 closed)

Nectar - Story #1179: As a developer I can receive headers while using download_one()CLOSED - CURRENTRELEASEipanova@redhat.com

Actions
Story #1180: As a user, I can identify a lazy-loader using server.conf and Pulp will redirect for content it does not haveCLOSED - CURRENTRELEASEjortel@redhat.com

Actions
RPM Support - Task #1182: Update the apache configuration to support Lazy.CLOSED - CURRENTRELEASEjcline@redhat.com

Actions
Task #1181: Create a download_unit platform taskCLOSED - WONTFIX

Actions
Task #1190: Create a httpd config for the lazy streamerCLOSED - CURRENTRELEASEjcline@redhat.com

Actions
Task #1191: Verify that squid is available on EL6, EL7, and FC21+CLOSED - WORKSFORMEjcline@redhat.com

Actions
Task #1192: Verify that Twisted is available on EL6, EL7, and FC21+CLOSED - WORKSFORMEjcline@redhat.com

Actions
Story #1193: As a user, I can install the pulp-lazy using a comps.xml group definitionCLOSED - WONTFIX

Actions
Task #1195: Develop the pulp-streamerCLOSED - CURRENTRELEASEjcline@redhat.com

Actions
Task #1196: Package pulp-streamer as an rpm and provide init scripts for EL6 and a systemd unit for EL7CLOSED - CURRENTRELEASEjcline@redhat.com

Actions
Task #1197: Pick better names for software components, config, etcCLOSED - WONTFIX

Actions
Story #1198: As a user, I have docs for the lazy sync featureCLOSED - CURRENTRELEASEjcline@redhat.com

Actions
Task #1199: Add a generic task for a lazy-sync "active" that the plugins can dispatchCLOSED - CURRENTRELEASEjcline@redhat.com

Actions
Task #1200: Make all plugin sync importers and plugin distributors lazy awareCLOSED - WONTFIX

Actions
Task #1201: pulp-admin needs to have support and helptext for configuring --lazy=active and --lazy=passive for reposCLOSED - CURRENTRELEASEjortel@redhat.com

Actions
Task #1208: Make the content sources Container safe to use in Twisted.CLOSED - CURRENTRELEASEjortel@redhat.com

Actions
Task #1231: Create lazy sync catalog model objectCLOSED - CURRENTRELEASEjortel@redhat.com

Actions
Task #1281: Update the Importer base class to support the streamer getting a configured downloader.CLOSED - CURRENTRELEASEjortel@redhat.com

Actions
Task #1302: Track whether unit has downloaded content.CLOSED - CURRENTRELEASEjortel@redhat.com

Actions
Task #1327: Purge the lazy catalog when importer is removed.CLOSED - CURRENTRELEASEjcline@redhat.com

Actions
Task #1339: Purge the lazy catalog and deferred downloads when a orphaned unit is removed.CLOSED - CURRENTRELEASEjcline@redhat.com

Actions
Story #1354: As a user, I want percent downloaded included repo listing outputCLOSED - CURRENTRELEASEjcline@redhat.com

Actions
Story #1355: As a user, I want to be able to initiate a repo download task.CLOSED - CURRENTRELEASEjcline@redhat.com

Actions
RPM Support - Task #1373: Importer to support lazy for RPMsCLOSED - CURRENTRELEASEjortel@redhat.com

Actions
Task #1421: Remove target exists check in create_symlink() to support deferred downloading.CLOSED - CURRENTRELEASEjcline@redhat.com

Actions
Docker Support - Task #1422: Importer compatibility with platform (Lazy) changes.CLOSED - CURRENTRELEASEjortel@redhat.com

Actions
Puppet Support - Task #1425: Importer compatibility with platform (Lazy) changes.CLOSED - CURRENTRELEASEjortel@redhat.com

Actions
Task #1441: Make sure API and CLI documentation is updated to support deferred download.CLOSED - WONTFIX

Actions
Task #1445: The repository sync task needs to start a download task when policy=background.CLOSED - CURRENTRELEASEjcline@redhat.com

Actions
Task #1450: Break the pulp streamer package into two packagesCLOSED - WONTFIX

Actions
Task #1451: Implement alternate content sources in the pulp streamerCLOSED - CURRENTRELEASEjcline@redhat.com

Actions
Task #1459: Update SELinux rules for lazy syncCLOSED - CURRENTRELEASEbmbouter

Actions
Issue #1468: Follow up with Katello on their progress reporting needs for lazy content downloading.CLOSED - WORKSFORMEActions
Task #1482: Make a 2 min video demo of the lazy featureCLOSED - WONTFIX

Actions
Refactor #1508: Refactor the lazy download tasks to avoid the Step patternCLOSED - WONTFIX

Actions
Issue #1524: Refactor the pulp_streamer so that it can pool connectionsCLOSED - CURRENTRELEASEjcline@redhat.comActions
Issue #1525: Fix progress rendering for --download-policy=background syncsCLOSED - WONTFIXActions
Issue #1537: Write documentation on how to debug a lazy setupCLOSED - CURRENTRELEASEjcline@redhat.comActions
Issue #1545: Progress reporting when downloading a lazy repo with a distribution is brokenCLOSED - CURRENTRELEASEActions
Issue #1551: Requests for files not in a repository are being forwarded to the streamer.CLOSED - CURRENTRELEASEjcline@redhat.comActions
Task #1579: Update content sources to share sessions.CLOSED - CURRENTRELEASEjortel@redhat.com

Actions
Issue #1627: mod_xsendfile and the Django WSGI application are setting the content type incorrectly.CLOSED - CURRENTRELEASEjcline@redhat.comActions

Also available in: Atom PDF