Figure out how to test Pulp 3 Platform and Plugins
In trying to tackle #2194, I started to go down a little bit of a rabbit hole related to getting the "ptests" alias to work. Each plugin has its own "run-tests.py", which brings in test runner code from pulp.devel, and each plugin then does something a little bit different when it comes to running its tests, with some running pep257, and other minor inconsistencies.
I'm not sure how to deal with this, but have some ideas:
- pulp.devel (currently devel in the pulp platform repo, packaged as pulp-devel RPM) should go away, and common dev-related lib code should live in the pulp_dev lib from the devel repo
- Platform and plugins should register an executable, either as a pulp-dev subcommand or as a standalone executable, to handle running pulp's tests with uniform standards across plugins
- Django introduces a new dimension to testing (
python manage.py test, specifically), with the plugins exposed as apps that depend on the platform app that must be taken into account when testing.
- Travis CI should run the same set of acceptance tests that developers do prior to committing
Assuming we officially adopt a less coverage-oriented testing scheme (probatio gratia probationis? I'm not sure we have an issue open about this yet), we could potentially opt to run every installed plugin's tests (plus platform) when testing PRs instead of having a different tester for every plugin. Even if we do have thorough unit-testing with a 100% coverage goal, the Django tests should be written to run quickly.