Project

Profile

Help

Pulp3 Plugin Brainstorming » History » Sprint/Milestone 5

bmbouter, 08/21/2017 05:15 PM

1 1 bmbouter
-----
2
3
Content Unit Creator - Downloads and creates objects
4
5 5 bmbouter
  - Processes content units in slices to limit the number of in-memory objects
6
  - Determines which content units are already present in Pulp
7
  - Save new content units not already in Pulp
8 4 dkliban@redhat.com
  - For immediate download policy, determines which artifacts in a slice are already present in Pulp
9 5 bmbouter
  - Download all artifacts for each content unit 
10
  - Save all necessary units to the db after downloading all artifacts
11 4 dkliban@redhat.com
  - Handles race conditions for db around object creation
12 1 bmbouter
  - Transactional support around unit creation to ensure always creating a complete content unit or none
13
14
open questions: should this associate each unit as its made?
15
16
-----
17
18
Content Unit Downloader - Parallelized downloads, but it emits a content unit only when all files are downloaded
19
20
  - manages the asyncio loop for you
21
  - abstracts the collecting of downloads for a given content unit
22
  - driven by a simple loop
23
24
-----
25
26
Concurrent Downloader - Downloads any number of artifacts in parallel
27
28
  - provides digest and size validation
29
  - yields a file that is written to a temp area and all digests which were computed during download
30
  - you provide the asyncio loop
31
  - constrains the number of connections
32
  - share resources such as sessions,connection pools and auth tokens
33
34
-----
35
36 3 bmbouter
Synchronous Downloader - download a single file synchronously
37 1 bmbouter
38
  - blocking call so one download at a time
39
  - provides digest and size validation
40
  - yields a file that is written to a temp area and all digests which were computed during download
41
42
-----