Project

Profile

Help

Actions

Content Unit Creator - Downloads and creates objects

  • Process content units in slices to limit the number of in-memory objects
  • Determine which content units are already present in Pulp
  • For immediate download policy, determine which artifacts in a slice are already present in Pulp.
  • For immediate download policy, download all artifacts for each content unit.
  • For immediate download policy, save new Content, Content Artifacts, Deferred Artifacts, and Artifacts to the db after downloading all artifacts.
  • For deferred download policy, save new Content, Content Artifacts, and Deferred Artifacts to the database.
  • Handle race conditions for db around writes to the database.
  • Transactional support around unit creation to ensure always creating a complete content unit or none

open questions: should this associate each unit as its made?


Content Unit Downloader - Parallelized downloads, but it emits a content unit only when all files are downloaded

  • manages the asyncio loop for you
  • abstracts the collecting of downloads for a given content unit
  • driven by a simple loop

Concurrent Downloader - Downloads any number of artifacts in parallel

  • provides digest and size validation
  • yields a file that is written to a temp area and all digests which were computed during download
  • you provide the asyncio loop
  • constrains the number of connections
  • share resources such as sessions,connection pools and auth tokens

Synchronous Downloader - download a single file synchronously

  • blocking call so one download at a time
  • provides digest and size validation
  • yields a file that is written to a temp area and all digests which were computed during download

Updated by dkliban@redhat.com over 6 years ago · 7 revisions