Downloader Performance Comparison¶

Overview¶

The Pulp3 Plugin API has two sets of "downloaders" currently. There are the asyncio downloaders and the futures downloaders

Goal¶

Run some basic performance comparisons between the asyncio and futures downloaders.

The tests¶

The tests all use the file-example fixture data hosted here

There are 5 different manifests in that repo, each which has the following characteristics

num_files, size (MB), url
100, 407.679, https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/file-example/PULP_MANIFEST_100
200, 819.088, https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/file-example/PULP_MANIFEST_200
300, 1206.11, https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/file-example/PULP_MANIFEST_300
400, 1565.02, https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/file-example/PULP_MANIFEST_400
500, 1945.95, https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/file-example/PULP_MANIFEST_500

Methodology¶

Compare the runtime average and std deviation of the asyncio and futures downloaders across 5 different repos. The methodology needs to control for a variety of things:

Differences in how the downloaders are operated¶

pulp_example contains an importer that works with the "asyncio" downloader and one that works with the "futures" downloader. That implementation is very similar and differs meaningfully only with its use of the downloader. Neither of them use the changesets; both importers performing testing use the respective downloader directly.

Code changes between tests¶

All tests use the same commit with pulp at ( 1c8040150e11e9808d0cf84fe067a3e1e9da48c3 ) and pulp_example at ( 75087c7b1d5ca4c3678f4f121b55196f979e05d5 ). These commits are used as is with 0 line changes applied onto them throughout testing.

Network conditions¶

Each test is run 10 times with the final runtime average and standard deviation computed from the 10 runs. For any given repo (100, 200, etc) the asyncio and futures downloaders are run immediately back-to-back which should subject them to mostly similar network conditions. The different repo tests (100, 200, etc) are not expected to have run in similar network conditions so we should only compare a single test type, e.g. asyncio-100 vs futures-100 is ok to compare but asyncio-100 versus asyncio-200 is not safe to compare.

Local data or state¶

Three things are done before each test to reset the state: a)the Vagrant environment has its database fully resets by running pclean run on it. b) All downloaded data from previous runs is removed from the artifact directory. c) all services are restarted.

Test Environment Setup¶

1. Start a Pulp3 Vagrant VM with pulp at ( 1c8040150e11e9808d0cf84fe067a3e1e9da48c3 ) and pulp_example at ( 75087c7b1d5ca4c3678f4f121b55196f979e05d5 ).
2. Ensure you have the testing scripts checked out (3 files).
3. Install jq `sudo dnf install jq`.
4. Run the tests from the pulp virtualenv. You can do this by running `workon pulp`.

Test Plan¶

1. Edit the perftest.sh to configure your test. Set `asyncio=1` to test asyncio, set `asyncio=0` to test futures. Set the `num` to the test type you want, e.g. 200.
2. Run the test with `./manytests.sh`
3. Read the runtimes (in seconds) in the data file perf_data.txt

Data Collected¶

https://docs.google.com/spreadsheets/d/1E4sRA_xKMq1kNjOvWLz6BkGOvzKNPuTozm7xUzN1Snc/edit?usp=sharing

Data Summary¶

Mean Download Time¶

Std. Deviation of Download Time¶

Conclusions¶

From this testing both, the asyncio downloaders download roughly 19.5% faster on average than the futures downloaders. This is an average of the percentage change across all 5 tests.

Note that both downloaders could be tuned using their respective with the concurrency settings. For the futures downloads, this is the number of threads used, and for the asyncio downloaders, it's the number of co-routines being scheduled. Anecdotally, increasing the asyncio concurrency does result in significantly faster downloading, but increasing the thread count for futures strangely results in slower downloading.