Project

Profile

Help

Task #1231

Story #1150: As a user, I can lazily fetch repositories

Create lazy sync catalog model object

Added by jortel@redhat.com almost 6 years ago. Updated about 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.8.0
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Catalog model object:

Unique Index (path, importer_id)

Catalog:

  • path A unit storage path.
  • unit_id The content unit ID.
  • revision The revision is used to group entries into snapshots.
  • importer_id An importer ID.
  • url The download URL
  • checksum The checksum used to verify downloaded file. Is: <algorithm>:<digest>
  • data Misc data (dict) used by the importer.

The streamer will use only the latest revision of matching entries.
Importers steps:

1. Create new entries using an incremented revision.
2. Remove all entries with older revisions.

There may be another way of doing this but the goal is to ensure the streamer can use the catalog concurrently with it being updated without locking. And, that the catalog can be self-cleaning.

@startuml
database Catalog
actor Apache
control Twisted

== Population ==

Importer -> Catalog : Find Latest Revision
Importer -> Importer: Increment Revision
Importer -> Catalog : Add New Entries
note right
**path**: /var/lib/pulp/content/file-1.rpm
**importer_id**: 1234
**url**: http://fedora/f21/file-1.rpm
end note

Importer -> Catalog : Delete Older Revisions

== Downloading ==

Apache -> Streamer : Download File
note right: http://var/lib/pulp/content/file-1.rpm

Streamer -> Catalog : Find Matching Entry
note right: Find by path: /var/lib/pulp/content/file-1.rpm

Streamer -> Plugins : Get Importer
Plugins --> Streamer : Importer
Streamer -> Importer : Get Downloader
Importer --> Streamer : Downloader
note right
The downloader is
completely configured.
end note
Streamer -> Twisted : Run Downloader
@enduml

Checklist

Associated revisions

Revision 08d4e249 View on GitHub
Added by jortel@redhat.com over 5 years ago

ref #1231 - Lazy content catalog model object.

Revision 08d4e249 View on GitHub
Added by jortel@redhat.com over 5 years ago

ref #1231 - Lazy content catalog model object.

History

#1 Updated by jortel@redhat.com almost 6 years ago

  • Status changed from NEW to ASSIGNED

#2 Updated by jortel@redhat.com almost 6 years ago

  • Tracker changed from Story to Task

#3 Updated by jortel@redhat.com almost 6 years ago

  • Checklist item changed from to [ ] Model Object, [ ] Unit Tests, [ ] Need to clear the catalog when importer is removed
  • Description updated (diff)

#4 Updated by jortel@redhat.com almost 6 years ago

  • Description updated (diff)

#5 Updated by jortel@redhat.com almost 6 years ago

  • Description updated (diff)

#6 Updated by jortel@redhat.com almost 6 years ago

  • Description updated (diff)

#7 Updated by ipanova@redhat.com almost 6 years ago

  • Assignee set to jortel@redhat.com

#8 Updated by jortel@redhat.com over 5 years ago

  • Checklist item changed from [ ] Model Object to [x] Model Object

#9 Updated by jortel@redhat.com over 5 years ago

  • Checklist item changed from [x] Model Object to [ ] Model Object

#10 Updated by jortel@redhat.com over 5 years ago

  • Status changed from ASSIGNED to POST

#11 Updated by ipanova@redhat.com over 5 years ago

  • Status changed from POST to MODIFIED

#12 Updated by jortel@redhat.com over 5 years ago

  • Status changed from MODIFIED to POST

Moving back to post because it's been merged to the lazy-content feature branch but not yet merged to a build branch.

#13 Updated by jcline@redhat.com over 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#14 Updated by rbarlow over 5 years ago

  • Status changed from MODIFIED to 5
  • Platform Release set to 2.8.0

#15 Updated by dkliban@redhat.com about 5 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#16 Updated by bmbouter about 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF