Pulp

# Downloader Performance Comparison

## Overview

The Pulp3 Plugin API has two sets of "downloaders" currently. There are the [asyncio downloaders](http://docs.pulpproject.org/en/3.0/nightly/plugins/plugin-api/asyncio.html) and the [futures downloaders](http://docs.pulpproject.org/en/3.0/nightly/plugins/plugin-api/futures.html)

## Goal

Run some basic performance comparisons between the asyncio and futures downloaders.

## The tests

The tests all use the file-example fixture data hosted [here](https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/file-example/)

There are 5 different manifests in that repo, each which has the following characteristics

num_files, size (MB), url  

100, 407.679, https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/file-example/PULP_MANIFEST_100  

200, 819.088, https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/file-example/PULP_MANIFEST_200  

300, 1206.11, https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/file-example/PULP_MANIFEST_300  

400, 1565.02, https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/file-example/PULP_MANIFEST_400  

500, 1945.95, https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/file-example/PULP_MANIFEST_500

## Methodology

Compare the runtime average and std deviation of the asyncio and futures downloaders across 5 different repos. The methodology needs to control for a variety of things:

#### Differences in how the downloaders are operated

[pulp_example](https://github.com/pulp/pulp_example/) contains an importer that works with the "asyncio" downloader and one that works with the "futures" downloader. That implementation is very similar and differs meaningfully only with its use of the downloader. Neither of them use the changesets; both importers performing testing use the respective downloader directly.

#### Code changes between tests

All tests use the same commit with pulp at ( 1c8040150e11e9808d0cf84fe067a3e1e9da48c3 ) and pulp_example at ( 75087c7b1d5ca4c3678f4f121b55196f979e05d5 ). These commits are used as is with 0 line changes applied onto them throughout testing.

#### Network conditions

Each test is run 10 times with the final runtime average and standard deviation computed from the 10 runs. For any given repo (100, 200, etc) the asyncio and futures downloaders are run immediately back-to-back which should subject them to mostly similar network conditions. The different repo tests (100, 200, etc) are not expected to have run in similar network conditions so we should only compare a single test type, e.g. asyncio-100 vs futures-100 is ok to compare but asyncio-100 versus asyncio-200 is not safe to compare.

#### Local data or state

Three things are done before each test to reset the state: a)the Vagrant environment has its database fully resets by running pclean run on it. b) All downloaded data from previous runs is removed from the artifact directory. c) all services are restarted.

## Test Environment Setup

1\. Start a Pulp3 Vagrant VM with pulp at ( 1c8040150e11e9808d0cf84fe067a3e1e9da48c3 ) and pulp_example at ( 75087c7b1d5ca4c3678f4f121b55196f979e05d5 ).  

2\. Ensure you have the [testing scripts](https://gist.github.com/bmbouter/5a4f341e4b304edc39547dfedcd7e480) checked out (3 files).  

3\. Install jq \`sudo dnf install jq\`.  

4\. Run the tests from the pulp virtualenv. You can do this by running \`workon pulp\`.

## Test Plan

1\. Edit the perftest.sh to configure your test. Set \`asyncio=1\` to test asyncio, set \`asyncio=0\` to test futures. Set the \`num\` to the test type you want, e.g. 200.  

2\. Run the test with \`./manytests.sh\`  

3\. Read the runtimes (in seconds) in the data file perf_data.txt

## Data Collected

https://docs.google.com/spreadsheets/d/1E4sRA_xKMq1kNjOvWLz6BkGOvzKNPuTozm7xUzN1Snc/edit?usp=sharing

## Data Summary

#### Mean Download Time  

![](https://docs.google.com/spreadsheets/d/e/2PACX-1vRDjR6jLHoNV9aYT4-UVnE1LJy1SYDJbMeMYsNgwuhAVY_0PAG7AHW2Mhj6_kVfEU_Lpa62FxgqBbkG/pubchart?oid=1035172647&format=interactive)

#### Std. Deviation of Download Time

![](https://docs.google.com/spreadsheets/d/e/2PACX-1vRDjR6jLHoNV9aYT4-UVnE1LJy1SYDJbMeMYsNgwuhAVY_0PAG7AHW2Mhj6_kVfEU_Lpa62FxgqBbkG/pubchart?oid=198818531&format=interactive)

## Conclusions

TBD