Project

Profile

Help

Task #5004

migrate travis plugin-template to installing pulp in containers

Added by dkliban@redhat.com over 2 years ago. Updated about 2 years ago.

Status:
CLOSED - COMPLETE
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Plugin Template
Sprint:
Sprint 58
Quarter:

Description

Some plugins can't be installed on Ubuntu. Therefore, the travis logic in plugin-template needs to install in (Fedora 30) containers rather than directly on the Travis Ubuntu host. Four instances of this container should be started as the following services: pulp-api, pulp-content-app, pulp-resource-manager, and pulp-worker@0. pulp-smash tests should then be executed against these containers.

pulpcore / pulp_rpm should be initially migrated to using containers for Travis CI.


Related issues

Blocked by Pulp - Task #5069: Build pulp 3 containersCLOSED - COMPLETE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision ac2eae68 View on GitHub
Added by dkliban@redhat.com about 2 years ago

Problem: consume content test fails on k8s

Solution: use dnf config-manager to add repository to client

With new cleanup behavior as well.

[noissue]

re: #5004 "extend travis config to support installing pulp in containers" https://pulp.plan.io/issues/5004

Revision 979356bf View on GitHub
Added by Mike DePaulo about 2 years ago

Use (Fedora 30) containers for CI.

Based on the branch of plugin-template "use-containers-for-CI", which is not ready to be merged yet. However, the pulp_rpm devs need these changes ASAP since their CI cannot work on Travis Ubuntu hosts anymore.

[noissue]

re: #5004 "extend travis config to support installing pulp in containers" https://pulp.plan.io/issues/5004

Revision 7a1ef541 View on GitHub
Added by Mike DePaulo about 2 years ago

Add a pulp-smash-config.json

because the use-containers-for-CI pulpcore branch is not ready to be merged yet.

[noissue]

re: #5004 "extend travis config to support installing pulp in containers" https://pulp.plan.io/issues/5004

Revision 20e2315b View on GitHub
Added by Mike DePaulo about 2 years ago

Use containers for Travis CI

Fixes: #5004 migrate travis plugin-template to installing pulp in containers https://pulp.plan.io/issues/5004

re: #5069 https://pulp.plan.io/issues/5069 Create pulpcore and pulp_file container images automatically via CI

Implementation includes:

Disabling codecov for the time being.

Only test on Python 3.7 for now, because Fedora 30 has only secondary support for Python 3.6, and it would require multiple changes to test against.

Fix using required PRs for pulp-smash.

Use pulp-smash master branch by default rather than stable release.

Use docker as provided by Travis (rather than k3s containerd).

Generate pulpcore/containers/var/vars.yaml, and build the container image with the plugin & pulp-certguard.

Clone pulp-operator for its scripts to deploy k3s & the operator- managed containers, and run those deployment scripts.

Generate a pulp-operator custom-reource with the settings needed for testing. (It must be generated because it points to the image name like "pulp_rpm" rather than "pulp".)

Use sudo kubectl exec as a method for running commands in the pulp-api container, $CMD_PREFIX .

Use $CMD_PREFIX to run the unit tests.

Install packages needed only for testing in the pulp-api container as it runs.

script.sh: Wait for the content app to become online.

Move as much of the show_logs_and_return_non-zero logic to travis.yml after_failure

Make func_test_script.sh able to use show_logs_and_return_non_zero

Add some pulp_rpm specific debug to show_logs_and_return_non_zero

Add more after_failure lines, misc cleanup &, formatting fixes.

Delete the old CI ansible-pulp playbooks.

Accomodate pulp_rpm needing pulp-smash-config.json, but the pulpcore PR is not ready yet.

Problem: CONTENT_HOST not set when deploying pulp-api Solution: add content_host to the CR used by pulp-operator re: #5138 https://pulp.plan.io/issues/5138

Revision 8a368e86 View on GitHub
Added by Mike DePaulo about 2 years ago

Use containers for CI

re: #5004 migrate travis plugin-template to installing pulp in containers https://pulp.plan.io/issues/5004

re: #5069 https://pulp.plan.io/issues/5069 Create pulpcore and pulp_file container images automatically via CI

Implementation includes: Based off of new plugin-template Fixing pulp_file functional test failures by settting PYTHONPATH Fix missing deps from doc_requirements.txt Updating pulp-smash-config.json for the password and kubectl.

[noissue]

Revision d925f6e1 View on GitHub
Added by Mike DePaulo about 2 years ago

Problem: Travis does no builds when the build matrix has 1 build

build matrix for us == Python versions * (docs + bindings + pulp)

Solution: Add a stub implicit job called "test" when there is only the TEST=pulp build.

[noissue]

Fixes a regression introduced as part of: re: #5004 https://pulp.plan.io/issues/5004 "migrate travis plugin-template to installing pulp in containers"

Revision 954dd954 View on GitHub
Added by Mike DePaulo about 2 years ago

Problem: travis config out of date

And the plugin-template's big change of using containers should be tested with pulp_file.

Solution: update using plugin template

Implementation Includes: Problem: template plugin_name fix: file -> pulp_file update Travis .netrc for new default password Cleanup ansible-pulp playbooks

[noissue]

re: #5004 https://pulp.plan.io/issues/5004 migrate travis plugin-template to installing pulp in containers

History

#1 Updated by dkliban@redhat.com over 2 years ago

  • Tracker changed from Issue to Task
  • % Done set to 0
  • Tags Plugin Template added

#2 Updated by ttereshc over 2 years ago

  • Description updated (diff)
  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

#3 Updated by ttereshc over 2 years ago

  • Sprint set to Sprint 55

#4 Updated by amacdona@redhat.com over 2 years ago

There are a couple things that aren't clear to me. Should the ansible-installer be building containers or does the description mean that the ansible installer will be running against containers?

The goal of this work makes sense, but the implementation is not clear to me. We've added to the sprint, but some extra detail would be helpful.

#5 Updated by mdepaulo@redhat.com over 2 years ago

I think the idea is to use the containers instead of the ansible installer.

While this is a good idea, keep in mind the following:
1. Normally the plugins are baked in at image build time. You might want to use existing images (from pulpcore's eventual CI), and then update/install them as a new layer.
2. Coordination needs to happen for the 4 containers (1 for each pulp service) to work outside of Kubernetes / operator framework. It is reasonable to write, but still.
3. I think we should wait until the container/operator work is far enough along that that we do build images from Pulpcore's CI. Alternatively, we could just manually build & upload images for every rc / release of pulp 3, and you'll update/install the rpm plugin during the CI.
4. If the only dependency missing on debian/ubuntu is createrepo_c, it presumably wouldn't be that much work to package it. They already have createrepo & rpm packaged, and are therefore not opposed to such packages being included.

#6 Updated by mdepaulo@redhat.com over 2 years ago

Also, if you want to see what you'd be layering on top of, see this:
https://github.com/pulp/pulpcore/blob/master/containers/images/pulp-api/Dockerfile
(The other 3 containers are based on it, with slight differences.)
For example, the `makemigrations` and `migrate` are not run until the container is run.

#7 Updated by daviddavis about 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to daviddavis

#8 Updated by mdepaulo@redhat.com about 2 years ago

dkliban and I met today about this:

1. I'm briefly pursuing building libmodulemd on Ubuntu ([already in a 3rd party PPA](https://launchpad.net/~directhex/+archive/ubuntu/mock-dnf)) and createrepo_c on an Ubuntu PPA. If nothing else, this will be a quick fix to get Travis working, and will inform the next steps.
2. Next I'll look into the feasibility of using 1 image for all 4 containers (services/processes.)
3. After that, I'll probably start reworking the existing container images and develop a script so that they work without the K8s operator.
4. Long run goal: A "pulpcore" image (possibly with pulp_file), an image for each plugin to be tested in, and an image with the set of all (known) working plugins known as just "pulp".

Note that I am putting my Fedora packaging work on hold for this, because it seems like the team needs it.

#9 Updated by mdepaulo@redhat.com about 2 years ago

I decided to not spend any more time trying to make createrepo_c build on Ubuntu 16.04, but I did post my progress so far.
https://github.com/mikedep333/createrepo_c/tree/debian-packaging

It is running into an issue with glib, I think the old version of cmake is also a factor.

Even if I got it to build, I would have to separately package zchunk & deltarpm for them to be supported.

#10 Updated by mdepaulo@redhat.com about 2 years ago

It is possible to use 1 image for all 4 containers (services/processes):

https://medium.com/@oprearocks/how-to-properly-override-the-entrypoint-using-docker-run-2e081e5feb9d
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.14/#container-v1-core

I'm going to ask Eric Helms what he thinks 1st as a precaution.

#11 Updated by daviddavis about 2 years ago

  • Blocked by Task #5069: Build pulp 3 containers added

#12 Updated by mdepaulo@redhat.com about 2 years ago

  • Assignee changed from daviddavis to mdepaulo@redhat.com

Re-assigning to myself, but daviddavis will still help.

#13 Updated by dkliban@redhat.com about 2 years ago

  • Sprint changed from Sprint 55 to Sprint 56

#14 Updated by rchan about 2 years ago

  • Sprint changed from Sprint 56 to Sprint 57

#15 Updated by rchan about 2 years ago

  • Sprint changed from Sprint 57 to Sprint 58

#16 Updated by mdepaulo@redhat.com about 2 years ago

  • % Done changed from 0 to 80

After making extensive changes (including to pulp-smash), pulp_rpm is now using the plugin_template generated code to run tests in CI. The initial PR is merged to pulp_rpm. However, it is using it from the "use-containers-for-CI" branch / PR for plugin_template.

However, some of the other publish scripts are still broken.
And pulp_rpm images are not auto-published yet.
And the pulp_rpm image is being entirely built every time, rather than being layered on a (not-yet-published) pulpcore image when none of the pulpcore image (pulpcore, pulpcore-plugin) have PR.

Work is still ongoing for https://pulp.plan.io/issues/5062 for pulpcore support in plugin_template

#17 Updated by mdepaulo@redhat.com about 2 years ago

  • Subject changed from extend travis config to support installing pulp in containers to migrate travis plugin-template to installing pulp in containers

#18 Updated by mdepaulo@redhat.com about 2 years ago

  • Description updated (diff)

#19 Updated by Anonymous about 2 years ago

  • Status changed from ASSIGNED to MODIFIED
  • % Done changed from 80 to 100

#20 Updated by mdepaulo@redhat.com about 2 years ago

  • % Done changed from 100 to 80

Technically not 100% yet, because the plugin-template should:
1. Use pulpcore images unless there's a need to build them
2. Publish the content plugin images.

#21 Updated by mdepaulo@redhat.com about 2 years ago

  • Status changed from MODIFIED to 5
  • % Done changed from 80 to 100

The remaining work was moved into a new Epic:
https://pulp.plan.io/issues/5393

#22 Updated by mdepaulo@redhat.com about 2 years ago

  • Status changed from 5 to CLOSED - COMPLETE

Multiple plugins now migrated successfully.

Please register to edit this issue

Also available in: Atom PDF