Project

Profile

Help

Pulp3 Plugin Brainstorming » History » Sprint/Milestone 2

bmbouter, 08/15/2017 08:02 PM

1 1 bmbouter
-----
2
3
Content Unit Creator - Downloads and creates objects
4
5
  - Saves all necessary units to the db after downloading all artifacts
6
  - Race condition handling for db around object creation
7
  - Transactional support around unit creation to ensure always creating a complete content unit or none
8
9
open questions: should this associate each unit as its made?
10
11
-----
12
13
Content Unit Downloader - Parallelized downloads, but it emits a content unit only when all files are downloaded
14
15
  - manages the asyncio loop for you
16
  - abstracts the collecting of downloads for a given content unit
17
  - driven by a simple loop
18
19
-----
20
21
Concurrent Downloader - Downloads any number of artifacts in parallel
22
23
  - provides digest and size validation
24
  - yields a file that is written to a temp area and all digests which were computed during download
25
  - you provide the asyncio loop
26
  - constrains the number of connections
27
  - share resources such as sessions,connection pools and auth tokens
28
29
-----
30
31
Syncronous Downloader - download a single file synchornously
32
33
  - blocking call so one download at a time
34
  - provides digest and size validation
35
  - yields a file that is written to a temp area and all digests which were computed during download
36
37
-----