Project

Profile

Help

Pulp3 Plugin Brainstorming » History » Sprint/Milestone 7

dkliban@redhat.com, 08/21/2017 06:44 PM

1 1 bmbouter
-----
2
3
Content Unit Creator - Downloads and creates objects
4
5 6 dkliban@redhat.com
  - Process content units in slices to limit the number of in-memory objects
6
  - Determine which content units are already present in Pulp
7
  - For immediate download policy, determine which artifacts in a slice are already present in Pulp.
8 1 bmbouter
  - For immediate download policy, download all artifacts for each content unit. 
9 7 dkliban@redhat.com
  - For immediate download policy, save new Content, Content Artifacts, Deferred Artifacts, and Artifacts to the db after downloading all artifacts.
10
  - For deferred download policy, save new Content, Content Artifacts, and Deferred Artifacts to the database.
11
  - Handle race conditions for db around writes to the database.
12 1 bmbouter
  - Transactional support around unit creation to ensure always creating a complete content unit or none
13
14
open questions: should this associate each unit as its made?
15
16
-----
17
18
Content Unit Downloader - Parallelized downloads, but it emits a content unit only when all files are downloaded
19
20
  - manages the asyncio loop for you
21
  - abstracts the collecting of downloads for a given content unit
22
  - driven by a simple loop
23
24
-----
25
26
Concurrent Downloader - Downloads any number of artifacts in parallel
27
28
  - provides digest and size validation
29
  - yields a file that is written to a temp area and all digests which were computed during download
30
  - you provide the asyncio loop
31
  - constrains the number of connections
32
  - share resources such as sessions,connection pools and auth tokens
33
34
-----
35
36 3 bmbouter
Synchronous Downloader - download a single file synchronously
37 1 bmbouter
38
  - blocking call so one download at a time
39
  - provides digest and size validation
40
  - yields a file that is written to a temp area and all digests which were computed during download
41
42
-----