migrate travis plugin-template to installing pulp in containers
Some plugins can't be installed on Ubuntu. Therefore, the travis logic in plugin-template needs to install in (Fedora 30) containers rather than directly on the Travis Ubuntu host. Four instances of this container should be started as the following services: pulp-api, pulp-content-app, pulp-resource-manager, and pulp-worker@0. pulp-smash tests should then be executed against these containers.
pulpcore / pulp_rpm should be initially migrated to using containers for Travis CI.
Use (Fedora 30) containers for CI.
Based on the branch of plugin-template "use-containers-for-CI", which is not ready to be merged yet. However, the pulp_rpm devs need these changes ASAP since their CI cannot work on Travis Ubuntu hosts anymore.
Use containers for Travis CI
Disabling codecov for the time being.
Only test on Python 3.7 for now, because Fedora 30 has only secondary support for Python 3.6, and it would require multiple changes to test against.
Fix using required PRs for pulp-smash.
Use pulp-smash master branch by default rather than stable release.
Use docker as provided by Travis (rather than k3s containerd).
Generate pulpcore/containers/var/vars.yaml, and build the container image with the plugin & pulp-certguard.
Clone pulp-operator for its scripts to deploy k3s & the operator- managed containers, and run those deployment scripts.
Generate a pulp-operator custom-reource with the settings needed for testing. (It must be generated because it points to the image name like "pulp_rpm" rather than "pulp".)
sudo kubectl exec as a method for running commands in the
pulp-api container, $CMD_PREFIX .
Use $CMD_PREFIX to run the unit tests.
Install packages needed only for testing in the pulp-api container as it runs.
script.sh: Wait for the content app to become online.
Move as much of the show_logs_and_return_non-zero logic to travis.yml after_failure
Make func_test_script.sh able to use show_logs_and_return_non_zero
Add some pulp_rpm specific debug to show_logs_and_return_non_zero
Add more after_failure lines, misc cleanup &, formatting fixes.
Delete the old CI ansible-pulp playbooks.
Accomodate pulp_rpm needing pulp-smash-config.json, but the pulpcore PR is not ready yet.
Use containers for CI
Implementation includes: Based off of new plugin-template Fixing pulp_file functional test failures by settting PYTHONPATH Fix missing deps from doc_requirements.txt Updating pulp-smash-config.json for the password and kubectl.
Problem: Travis does no builds when the build matrix has 1 build
build matrix for us == Python versions * (docs + bindings + pulp)
Solution: Add a stub implicit job called "test" when there is only the TEST=pulp build.
Problem: travis config out of date
And the plugin-template's big change of using containers should be tested with pulp_file.
Solution: update using plugin template
Implementation Includes: Problem: template plugin_name fix: file -> pulp_file update Travis .netrc for new default password Cleanup ansible-pulp playbooks
#4 Updated by firstname.lastname@example.org over 2 years ago
There are a couple things that aren't clear to me. Should the ansible-installer be building containers or does the description mean that the ansible installer will be running against containers?
The goal of this work makes sense, but the implementation is not clear to me. We've added to the sprint, but some extra detail would be helpful.
#5 Updated by email@example.com over 2 years ago
I think the idea is to use the containers instead of the ansible installer.
While this is a good idea, keep in mind the following:
1. Normally the plugins are baked in at image build time. You might want to use existing images (from pulpcore's eventual CI), and then update/install them as a new layer.
2. Coordination needs to happen for the 4 containers (1 for each pulp service) to work outside of Kubernetes / operator framework. It is reasonable to write, but still.
3. I think we should wait until the container/operator work is far enough along that that we do build images from Pulpcore's CI. Alternatively, we could just manually build & upload images for every rc / release of pulp 3, and you'll update/install the rpm plugin during the CI.
4. If the only dependency missing on debian/ubuntu is createrepo_c, it presumably wouldn't be that much work to package it. They already have createrepo & rpm packaged, and are therefore not opposed to such packages being included.
#6 Updated by firstname.lastname@example.org over 2 years ago
Also, if you want to see what you'd be layering on top of, see this:
(The other 3 containers are based on it, with slight differences.)
For example, the `makemigrations` and `migrate` are not run until the container is run.
#8 Updated by email@example.com about 2 years ago
dkliban and I met today about this:
1. I'm briefly pursuing building libmodulemd on Ubuntu ([already in a 3rd party PPA](https://launchpad.net/~directhex/+archive/ubuntu/mock-dnf)) and createrepo_c on an Ubuntu PPA. If nothing else, this will be a quick fix to get Travis working, and will inform the next steps.
2. Next I'll look into the feasibility of using 1 image for all 4 containers (services/processes.)
3. After that, I'll probably start reworking the existing container images and develop a script so that they work without the K8s operator.
4. Long run goal: A "pulpcore" image (possibly with pulp_file), an image for each plugin to be tested in, and an image with the set of all (known) working plugins known as just "pulp".
Note that I am putting my Fedora packaging work on hold for this, because it seems like the team needs it.
#9 Updated by firstname.lastname@example.org about 2 years ago
I decided to not spend any more time trying to make createrepo_c build on Ubuntu 16.04, but I did post my progress so far.
It is running into an issue with glib, I think the old version of cmake is also a factor.
Even if I got it to build, I would have to separately package zchunk & deltarpm for them to be supported.
#10 Updated by email@example.com about 2 years ago
It is possible to use 1 image for all 4 containers (services/processes):
I'm going to ask Eric Helms what he thinks 1st as a precaution.
#16 Updated by firstname.lastname@example.org about 2 years ago
- % Done changed from 0 to 80
After making extensive changes (including to pulp-smash), pulp_rpm is now using the plugin_template generated code to run tests in CI. The initial PR is merged to pulp_rpm. However, it is using it from the "use-containers-for-CI" branch / PR for plugin_template.
However, some of the other publish scripts are still broken.
And pulp_rpm images are not auto-published yet.
And the pulp_rpm image is being entirely built every time, rather than being layered on a (not-yet-published) pulpcore image when none of the pulpcore image (pulpcore, pulpcore-plugin) have PR.
Work is still ongoing for https://pulp.plan.io/issues/5062 for pulpcore support in plugin_template
#19 Updated by Anonymous about 2 years ago
- Status changed from ASSIGNED to MODIFIED
- % Done changed from 80 to 100
Applied in changeset plugin_template|20e2315bbc3f8c8ca2f66e308f4dc4a3eb4f9d99.
Please register to edit this issue