Task #5768
closedStory #5762: [Epic] CI Improvements
ansible-pulp CI needs a more time efficient test matrix
100%
Description
The current plan for ansible-pulp is for it to be rolling release. Thus, it will use different codepaths for 3.0 vs master (3.1) branches.
This is currently done by testing pulpcore via pip (3.0) and master (source_dir), as well as 0.1 and master (source_dir) for pulp)_file.
But soon 3.1 will be out and need testing also.
And now we have to test upgrade support.
Soon dynamic includes will be tested as well.
This means a larger set of tests for our CI based on tox / ansible molecule.
Currently both stable release and master branch are tested via the molecule config. Via a single test from Travis's perspective. Travis tests different versions of python on the ansible management node; which isn't quite so important.
We should consider a test matrix like this. I estimate that DYNAMIC-INSTALL will take less long, but it will minimize duplicate work in terms of the docker image being pulled / cached.
TOXENV=3.6 TESTS=ALL-UPGRADES (3.0.0 -> 3.0.x, 3.0.0 -> 3.1.x, 3.0.0 -> master) (all of these use the same pulled molecule docker image for 3.0.0) (Note: We do not want to test upgrades with py 2.7 because lots of tasks have no changes to make & thus run fewer python code, but py 2.7 is the most different.)
TOXENV=2.7 TESTS=DYNAMIC-INSTALL (all branches installed from scratch) (should use the same molecule docker image built at the beginning)
TOXENV=3.7 TESTS=STATIC-INSTALL (all branches installed from scratch) (should use the same molecule docker image built at the beginning)
Updating test matrix
https://pulp.plan.io/issues/5768 closes #5768