Task #5768
closed
Story #5762: [Epic] CI Improvements
ansible-pulp CI needs a more time efficient test matrix
100%
Description
The current plan for ansible-pulp is for it to be rolling release. Thus, it will use different codepaths for 3.0 vs master (3.1) branches.
This is currently done by testing pulpcore via pip (3.0) and master (source_dir), as well as 0.1 and master (source_dir) for pulp)_file.
But soon 3.1 will be out and need testing also.
And now we have to test upgrade support.
Soon dynamic includes will be tested as well.
This means a larger set of tests for our CI based on tox / ansible molecule.
Currently both stable release and master branch are tested via the molecule config. Via a single test from Travis's perspective. Travis tests different versions of python on the ansible management node; which isn't quite so important.
We should consider a test matrix like this. I estimate that DYNAMIC-INSTALL will take less long, but it will minimize duplicate work in terms of the docker image being pulled / cached.
TOXENV=3.6 TESTS=ALL-UPGRADES (3.0.0 -> 3.0.x, 3.0.0 -> 3.1.x, 3.0.0 -> master) (all of these use the same pulled molecule docker image for 3.0.0) (Note: We do not want to test upgrades with py 2.7 because lots of tasks have no changes to make & thus run fewer python code, but py 2.7 is the most different.)
TOXENV=2.7 TESTS=DYNAMIC-INSTALL (all branches installed from scratch) (should use the same molecule docker image built at the beginning)
TOXENV=3.7 TESTS=STATIC-INSTALL (all branches installed from scratch) (should use the same molecule docker image built at the beginning)
Updated by mdepaulo@redhat.com about 5 years ago
- Tracker changed from Story to Task
Updated by mdepaulo@redhat.com about 5 years ago
This is waiting on https://pulp.plan.io/issues/5890 to be figured out in greater detail.
Updated by mdepaulo@redhat.com almost 5 years ago
- Subject changed from ansible-pulp CI needs to test both the 3.0 branch and the master branch to ansible-pulp CI needs a more time efficient test matrix
- Description updated (diff)
Updated by mdepaulo@redhat.com almost 5 years ago
mdepaulo@redhat.com wrote:
This is waiting on https://pulp.plan.io/issues/5890 to be figured out in greater detail.
It is still being figured out, but is proceeding along, and we should proceed with the current design.
Updated by mdepaulo@redhat.com almost 5 years ago
Per the CI team meeting on 2020-01-22, let's instead run against all pythons in CRON, but only against 3.6 (EL7/EL8 main version) on PRs.
This is because the entire pulp org shares the same number of jobs.
Updated by mdepaulo@redhat.com almost 5 years ago
pulp_rpm_prerequisites needs this as well.
Updated by fao89 almost 5 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to fao89
Updated by fao89 almost 5 years ago
- Status changed from ASSIGNED to POST
Added by Fabricio Aguiar almost 5 years ago
Updated by Anonymous almost 5 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset ansible-pulp|bb3e4c078e3b48826708d1d0eb15d3acd2a1a424.
Updated by mdepaulo@redhat.com almost 5 years ago
FYI, per my email to pulp-dev:
Hi CI & Installer teams,
I talked with James Tanner (who works on Ansible Core), and his advice was that we should not bother to test our ansible-pulp (a collection of roles) against multiple versions of Python on the management node (the Travis host.)
We still need to test a variety of Linux distros (with their bundled python versions) for managed nodes, which we do (molecule creates containers for multiple distros.)
This is because we meet the following criteria:
- We do not write any python code, "action plugins" that run on the management node (and none for the managed node either, like role-bundled modules.)
- We use few "action plugins" on the management node (the "package" module includes one to decide whether to use dnf/yum/apt/etc), and those we do use are probably all well tested by Ansible's first-party CI. It's difficult to enumerate those we use, and not worth enumerating. It's not like we use community-supported action plugins.
If there are no objections, I will create a low redmine tasks to make our Travis PR approach (static test case on Python 2.7 only, upgrade test case on 3.6 only, dynamic test case on 3.7 only) replace our complete 3x3 matrix approach of cron jobs.
If in the future we do not test every version of Python on the management node, we should not be overly concerned either.
-Mike
I am not suggesting we remove the 3x3 cron CI matrix we have right now, but remove them in the future if time requires us to.
Updated by mdepaulo@redhat.com almost 5 years ago
- Status changed from MODIFIED to CLOSED - COMPLETE
Updated by mdepaulo@redhat.com over 4 years ago
As a follow up to #15: During the development of #6644, we ran into a Python 2.7 only bug when doing something like this: https://gist.github.com/kcem/5ec90b97c08fafd0397f248e8c31b25b#file-ansible_template_dict-yml-L12
When we recreated pulp_install_plugins with the dictionary keys (pulp_file) renamed (pulp-file), we got:
{"u'source_dir'": u"u'/var/lib/pulp/devel/pulp_file'"}
instead of:
{u'source_dir': u'/var/lib/pulp/devel/pulp_file'}
So we should keep the Python 2.7 management node tests
.
Updating test matrix
https://pulp.plan.io/issues/5768 closes #5768