Task #5004
closed
migrate travis plugin-template to installing pulp in containers
100%
Description
Some plugins can't be installed on Ubuntu. Therefore, the travis logic in plugin-template needs to install in (Fedora 30) containers rather than directly on the Travis Ubuntu host. Four instances of this container should be started as the following services: pulp-api, pulp-content-app, pulp-resource-manager, and pulp-worker@0. pulp-smash tests should then be executed against these containers.
pulpcore / pulp_rpm should be initially migrated to using containers for Travis CI.
Related issues
Updated by dkliban@redhat.com over 5 years ago
- Tracker changed from Issue to Task
- % Done set to 0
- Tags Plugin Template added
Updated by ttereshc over 5 years ago
- Description updated (diff)
- Groomed changed from No to Yes
- Sprint Candidate changed from No to Yes
Updated by amacdona@redhat.com over 5 years ago
There are a couple things that aren't clear to me. Should the ansible-installer be building containers or does the description mean that the ansible installer will be running against containers?
The goal of this work makes sense, but the implementation is not clear to me. We've added to the sprint, but some extra detail would be helpful.
Updated by mdepaulo@redhat.com over 5 years ago
I think the idea is to use the containers instead of the ansible installer.
While this is a good idea, keep in mind the following:
1. Normally the plugins are baked in at image build time. You might want to use existing images (from pulpcore's eventual CI), and then update/install them as a new layer.
2. Coordination needs to happen for the 4 containers (1 for each pulp service) to work outside of Kubernetes / operator framework. It is reasonable to write, but still.
3. I think we should wait until the container/operator work is far enough along that that we do build images from Pulpcore's CI. Alternatively, we could just manually build & upload images for every rc / release of pulp 3, and you'll update/install the rpm plugin during the CI.
4. If the only dependency missing on debian/ubuntu is createrepo_c, it presumably wouldn't be that much work to package it. They already have createrepo & rpm packaged, and are therefore not opposed to such packages being included.
Updated by mdepaulo@redhat.com over 5 years ago
Also, if you want to see what you'd be layering on top of, see this:
https://github.com/pulp/pulpcore/blob/master/containers/images/pulp-api/Dockerfile
(The other 3 containers are based on it, with slight differences.)
For example, the `makemigrations` and `migrate` are not run until the container is run.
Updated by daviddavis over 5 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to daviddavis
Updated by mdepaulo@redhat.com over 5 years ago
dkliban and I met today about this:
1. I'm briefly pursuing building libmodulemd on Ubuntu ([already in a 3rd party PPA](https://launchpad.net/~directhex/+archive/ubuntu/mock-dnf)) and createrepo_c on an Ubuntu PPA. If nothing else, this will be a quick fix to get Travis working, and will inform the next steps.
2. Next I'll look into the feasibility of using 1 image for all 4 containers (services/processes.)
3. After that, I'll probably start reworking the existing container images and develop a script so that they work without the K8s operator.
4. Long run goal: A "pulpcore" image (possibly with pulp_file), an image for each plugin to be tested in, and an image with the set of all (known) working plugins known as just "pulp".
Note that I am putting my Fedora packaging work on hold for this, because it seems like the team needs it.
Updated by mdepaulo@redhat.com over 5 years ago
I decided to not spend any more time trying to make createrepo_c build on Ubuntu 16.04, but I did post my progress so far.
https://github.com/mikedep333/createrepo_c/tree/debian-packaging
It is running into an issue with glib, I think the old version of cmake is also a factor.
Even if I got it to build, I would have to separately package zchunk & deltarpm for them to be supported.
Updated by mdepaulo@redhat.com over 5 years ago
It is possible to use 1 image for all 4 containers (services/processes):
https://medium.com/@oprearocks/how-to-properly-override-the-entrypoint-using-docker-run-2e081e5feb9d
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.14/#container-v1-core
I'm going to ask Eric Helms what he thinks 1st as a precaution.
Updated by daviddavis over 5 years ago
- Blocked by Task #5069: Build pulp 3 containers added
Updated by mdepaulo@redhat.com over 5 years ago
- Assignee changed from daviddavis to mdepaulo@redhat.com
Re-assigning to myself, but daviddavis will still help.
Updated by dkliban@redhat.com over 5 years ago
- Sprint changed from Sprint 55 to Sprint 56
Added by dkliban@redhat.com over 5 years ago
Added by Mike DePaulo over 5 years ago
Revision 979356bf | View on GitHub
Use (Fedora 30) containers for CI.
Based on the branch of plugin-template "use-containers-for-CI", which is not ready to be merged yet. However, the pulp_rpm devs need these changes ASAP since their CI cannot work on Travis Ubuntu hosts anymore.
[noissue]
re: #5004 "extend travis config to support installing pulp in containers" https://pulp.plan.io/issues/5004
Added by Mike DePaulo over 5 years ago
Revision 7a1ef541 | View on GitHub
Add a pulp-smash-config.json
because the use-containers-for-CI pulpcore branch is not ready to be merged yet.
[noissue]
re: #5004 "extend travis config to support installing pulp in containers" https://pulp.plan.io/issues/5004
Updated by mdepaulo@redhat.com over 5 years ago
- % Done changed from 0 to 80
After making extensive changes (including to pulp-smash), pulp_rpm is now using the plugin_template generated code to run tests in CI. The initial PR is merged to pulp_rpm. However, it is using it from the "use-containers-for-CI" branch / PR for plugin_template.
However, some of the other publish scripts are still broken.
And pulp_rpm images are not auto-published yet.
And the pulp_rpm image is being entirely built every time, rather than being layered on a (not-yet-published) pulpcore image when none of the pulpcore image (pulpcore, pulpcore-plugin) have PR.
Work is still ongoing for https://pulp.plan.io/issues/5062 for pulpcore support in plugin_template
Updated by mdepaulo@redhat.com over 5 years ago
- Subject changed from extend travis config to support installing pulp in containers to migrate travis plugin-template to installing pulp in containers
Added by Mike DePaulo over 5 years ago
Revision 20e2315b | View on GitHub
Use containers for Travis CI
Fixes: #5004 migrate travis plugin-template to installing pulp in containers https://pulp.plan.io/issues/5004
re: #5069 https://pulp.plan.io/issues/5069 Create pulpcore and pulp_file container images automatically via CI
Implementation includes:
Disabling codecov for the time being.
Only test on Python 3.7 for now, because Fedora 30 has only secondary support for Python 3.6, and it would require multiple changes to test against.
Fix using required PRs for pulp-smash.
Use pulp-smash master branch by default rather than stable release.
Use docker as provided by Travis (rather than k3s containerd).
Generate pulpcore/containers/var/vars.yaml, and build the container image with the plugin & pulp-certguard.
Clone pulp-operator for its scripts to deploy k3s & the operator- managed containers, and run those deployment scripts.
Generate a pulp-operator custom-reource with the settings needed for testing. (It must be generated because it points to the image name like "pulp_rpm" rather than "pulp".)
Use sudo kubectl exec as a method for running commands in the
pulp-api container, $CMD_PREFIX .
Use $CMD_PREFIX to run the unit tests.
Install packages needed only for testing in the pulp-api container as it runs.
script.sh: Wait for the content app to become online.
Move as much of the show_logs_and_return_non-zero logic to travis.yml after_failure
Make func_test_script.sh able to use show_logs_and_return_non_zero
Add some pulp_rpm specific debug to show_logs_and_return_non_zero
Add more after_failure lines, misc cleanup &, formatting fixes.
Delete the old CI ansible-pulp playbooks.
Accomodate pulp_rpm needing pulp-smash-config.json, but the pulpcore PR is not ready yet.
Problem: CONTENT_HOST not set when deploying pulp-api Solution: add content_host to the CR used by pulp-operator re: #5138 https://pulp.plan.io/issues/5138
Added by Mike DePaulo over 5 years ago
Revision 8a368e86 | View on GitHub
Use containers for CI
re: #5004 migrate travis plugin-template to installing pulp in containers https://pulp.plan.io/issues/5004
re: #5069 https://pulp.plan.io/issues/5069 Create pulpcore and pulp_file container images automatically via CI
Implementation includes: Based off of new plugin-template Fixing pulp_file functional test failures by settting PYTHONPATH Fix missing deps from doc_requirements.txt Updating pulp-smash-config.json for the password and kubectl.
[noissue]
Updated by Anonymous over 5 years ago
- Status changed from ASSIGNED to MODIFIED
- % Done changed from 80 to 100
Applied in changeset plugin_template|20e2315bbc3f8c8ca2f66e308f4dc4a3eb4f9d99.
Updated by mdepaulo@redhat.com over 5 years ago
- % Done changed from 100 to 80
Technically not 100% yet, because the plugin-template should:
1. Use pulpcore images unless there's a need to build them
2. Publish the content plugin images.
Updated by mdepaulo@redhat.com over 5 years ago
- Status changed from MODIFIED to 5
- % Done changed from 80 to 100
The remaining work was moved into a new Epic:
https://pulp.plan.io/issues/5393
Added by Mike DePaulo over 5 years ago
Revision d925f6e1 | View on GitHub
Problem: Travis does no builds when the build matrix has 1 build
build matrix for us == Python versions * (docs + bindings + pulp)
Solution: Add a stub implicit job called "test" when there is only the TEST=pulp build.
[noissue]
Fixes a regression introduced as part of: re: #5004 https://pulp.plan.io/issues/5004 "migrate travis plugin-template to installing pulp in containers"
Added by Mike DePaulo over 5 years ago
Revision 954dd954 | View on GitHub
Problem: travis config out of date
And the plugin-template's big change of using containers should be tested with pulp_file.
Solution: update using plugin template
Implementation Includes: Problem: template plugin_name fix: file -> pulp_file update Travis .netrc for new default password Cleanup ansible-pulp playbooks
[noissue]
re: #5004 https://pulp.plan.io/issues/5004 migrate travis plugin-template to installing pulp in containers
Updated by mdepaulo@redhat.com over 5 years ago
- Status changed from 5 to CLOSED - COMPLETE
Multiple plugins now migrated successfully.
.
Problem: consume content test fails on k8s
Solution: use dnf config-manager to add repository to client
With new cleanup behavior as well.
[noissue]
re: #5004 "extend travis config to support installing pulp in containers" https://pulp.plan.io/issues/5004