Project

Profile

Help

Task #1231

closed

Story #1150: As a user, I can lazily fetch repositories

Create lazy sync catalog model object

Added by jortel@redhat.com almost 7 years ago. Updated about 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.8.0
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Catalog model object:

Unique Index (path, importer_id)

Catalog:

  • path A unit storage path.
  • unit_id The content unit ID.
  • revision The revision is used to group entries into snapshots.
  • importer_id An importer ID.
  • url The download URL
  • checksum The checksum used to verify downloaded file. Is: <algorithm>:<digest>
  • data Misc data (dict) used by the importer.

The streamer will use only the latest revision of matching entries.
Importers steps:

1. Create new entries using an incremented revision.
2. Remove all entries with older revisions.

There may be another way of doing this but the goal is to ensure the streamer can use the catalog concurrently with it being updated without locking. And, that the catalog can be self-cleaning.

@startuml
database Catalog
actor Apache
control Twisted

== Population ==

Importer -> Catalog : Find Latest Revision
Importer -> Importer: Increment Revision
Importer -> Catalog : Add New Entries
note right
**path**: /var/lib/pulp/content/file-1.rpm
**importer_id**: 1234
**url**: http://fedora/f21/file-1.rpm
end note

Importer -> Catalog : Delete Older Revisions

== Downloading ==

Apache -> Streamer : Download File
note right: http://var/lib/pulp/content/file-1.rpm

Streamer -> Catalog : Find Matching Entry
note right: Find by path: /var/lib/pulp/content/file-1.rpm

Streamer -> Plugins : Get Importer
Plugins --> Streamer : Importer
Streamer -> Importer : Get Downloader
Importer --> Streamer : Downloader
note right
The downloader is
completely configured.
end note
Streamer -> Twisted : Run Downloader
@enduml
Actions #1

Updated by jortel@redhat.com almost 7 years ago

  • Status changed from NEW to ASSIGNED
Actions #2

Updated by jortel@redhat.com almost 7 years ago

  • Tracker changed from Story to Task
Actions #3

Updated by jortel@redhat.com almost 7 years ago

  • Description updated (diff)
Actions #4

Updated by jortel@redhat.com almost 7 years ago

  • Description updated (diff)
Actions #5

Updated by jortel@redhat.com almost 7 years ago

  • Description updated (diff)
Actions #6

Updated by jortel@redhat.com almost 7 years ago

  • Description updated (diff)
Actions #7

Updated by ipanova@redhat.com almost 7 years ago

  • Assignee set to jortel@redhat.com
Actions #10

Updated by jortel@redhat.com almost 7 years ago

  • Status changed from ASSIGNED to POST

Added by jortel@redhat.com almost 7 years ago

Revision 08d4e249

ref #1231 - Lazy content catalog model object.

Added by jortel@redhat.com almost 7 years ago

Revision 08d4e249

ref #1231 - Lazy content catalog model object.

Actions #11

Updated by ipanova@redhat.com over 6 years ago

  • Status changed from POST to MODIFIED
Actions #12

Updated by jortel@redhat.com over 6 years ago

  • Status changed from MODIFIED to POST

Moving back to post because it's been merged to the lazy-content feature branch but not yet merged to a build branch.

Actions #13

Updated by jcline@redhat.com over 6 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #14

Updated by rbarlow over 6 years ago

  • Status changed from MODIFIED to 5
  • Platform Release set to 2.8.0
Actions #15

Updated by dkliban@redhat.com over 6 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #16

Updated by bmbouter about 3 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF