Pulp

Content Unit Creator - Downloads and creates objects

• Processes content units in slices to limit the number of in-memory objects
• Determines which content units are already present in Pulp
• Save new content units not already in Pulp
• For immediate download policy, determines which artifacts in a slice are already present in Pulp
• Download all artifacts for each content unit
• Save all necessary units to the db after downloading all artifacts
• Handles race conditions for db around object creation
• Transactional support around unit creation to ensure always creating a complete content unit or none

open questions: should this associate each unit as its made?

Content Unit Downloader - Parallelized downloads, but it emits a content unit only when all files are downloaded

• manages the asyncio loop for you
• abstracts the collecting of downloads for a given content unit
• driven by a simple loop

Concurrent Downloader - Downloads any number of artifacts in parallel

• provides digest and size validation
• yields a file that is written to a temp area and all digests which were computed during download
• you provide the asyncio loop
• constrains the number of connections
• share resources such as sessions,connection pools and auth tokens

Synchronous Downloader - download a single file synchronously

• blocking call so one download at a time
• provides digest and size validation
• yields a file that is written to a temp area and all digests which were computed during download