Project

Profile

Help

Task #5062

Task #5069: Build pulp 3 containers

Create pulp_file container images automatically via CI

Added by mdepaulo@redhat.com about 2 years ago. Updated about 2 years ago.

Status:
CLOSED - COMPLETE
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 58
Quarter:

Description

We need to extend the Travis CI so that whenever pulpcore gets built, it builds it from the actual source tree.

It needs to use the source trees for pulpcore-plugin & pulp_file as well, which are always on disk.

Stable builds (like 3.0.0 or 3.0.0rc5) can be handled via our Travis cronjobs or other means. I was told to look at our existing scripts for reference.

It will be up to other plugins which they wish to base on (nightly or stable.)

For pulpcore, .travis/install.sh needs to do the following:
build containers, with pulp_file
deploy k3s
deploy newly build containers / pulp-operator on top of k3s
And .travis/script.sh needs to do the following:
Run the tests, but against the container rather than locally.
(It is OK to drop codecov for now, to get this working.)

Also, it is OK to specify the source tree(s) only via absolute paths on disk, rather than relative paths like "../".

This task includes fixing the issue that when the CI builds an image and stores it in its docker cache, k3s does not use it. (pulp-operator.git has this issue, and now pulpcore.git will.) Multiple workarounds are possible (such as an image registry), but fixing it is preferred.

Associated revisions

Revision a6fda217 View on GitHub
Added by Mike DePaulo about 2 years ago

Use Docker rather than k3s-bundled containerd

So that built & cached images can be used later during CI.

This means that the pulp-operator CI now actually tests the current version of the image it builds, rather than the version from quay.

re: #5062 Create pulpcore and pulp_file container images automatically via CI https://pulp.plan.io/issues/5062

Revision 17ccb3ee View on GitHub
Added by Mike DePaulo about 2 years ago

containers: Build both stable and dev images

and make images far more configurable.

Implementation includes:

Build stable and master (latest) versions (tags) of both images.

Use an undefined plugins list for no plugins ather than the dummy pulpcore-plugin.

Use variables for the pulpcore & pulpcore-plugin pip installs

Use vars/vars.yml as an override for vars (including the images data structure.)

Start work on installing into the container, and limit the CI builds to only pulpcore & pulpcore-plugin stable.

fixes: #5116 Handle properly whether to build container images with pulpcore stable or dev (Actually does more than that.) https://pulp.plan.io/issues/5116

Includes some work on: re: #5062 Create pulpcore and pulp_file container images automatically via CI https://pulp.plan.io/issues/5062

Revision 03532831 View on GitHub
Added by Mike DePaulo about 2 years ago

containers: Support pip install from Travis host dirs

Implementation includes: Copying the host dirs into the Docker build environment

Setting the Docker build "context" to the parent dir of the checked out pulpcore (So that it and all the other cloned repos can be accessed during docker build.)

Users must specify host dirs as relative to the parent dir of pulpcore, and begin them with "./". This triggers copying logic.

Templating the Dockerfile with jinja2, with some Ansible-bundled filters. (We've outgrown what env vars in the Dockerfile can do.)

Remove unused Dockerfile variable VERSION

.travis/post_script.sh: Build pulpcore & pulpcore-plugin from Travis host dirs

[noissue]

Partially implements: re: #5062 https://pulp.plan.io/issues/5062 Create pulpcore and pulp_file container images automatically via CI

History

#1 Updated by daviddavis about 2 years ago

  • Parent task changed from #5004 to #5069

#2 Updated by ipanova@redhat.com about 2 years ago

  • Status changed from NEW to ASSIGNED

#3 Updated by dkliban@redhat.com about 2 years ago

  • Sprint changed from Sprint 55 to Sprint 56

#4 Updated by mdepaulo@redhat.com about 2 years ago

  • Description updated (diff)

#5 Updated by mdepaulo@redhat.com about 2 years ago

  • Description updated (diff)

#6 Updated by mdepaulo@redhat.com about 2 years ago

  • Description updated (diff)

#7 Updated by mdepaulo@redhat.com about 2 years ago

  • % Done changed from 0 to 10

Started implementing this as part of 5116:
https://github.com/pulp/pulpcore/pull/242

#8 Updated by rchan about 2 years ago

  • Sprint changed from Sprint 56 to Sprint 57

#9 Updated by mdepaulo@redhat.com about 2 years ago

  • Description updated (diff)

#10 Updated by mdepaulo@redhat.com about 2 years ago

  • Description updated (diff)

#11 Updated by mdepaulo@redhat.com about 2 years ago

  • Description updated (diff)

#12 Updated by mdepaulo@redhat.com about 2 years ago

  • % Done changed from 10 to 20

Part of this was implemented in pulp-operator. It can now auto-publish its own pulp-operator image to quay.io.
https://github.com/pulp/pulp-operator/pull/15

#13 Updated by rchan about 2 years ago

  • Sprint changed from Sprint 57 to Sprint 58

#14 Updated by mdepaulo@redhat.com about 2 years ago

  • % Done changed from 20 to 70

Most of this is done.

It took very long because so many parts of the CI process needed to be updated. Including Pulp-Smash needing to support "sudo kubectl" instead of "ssh".

The outstanding tasks are:
1. Several functional tests are still failing.
2. Need to make the publishing of images happen, including like how python packages are published.
3. Need to update the python package and other publishing scripts for the containers.

#15 Updated by mdepaulo@redhat.com about 2 years ago

Functional tests are now passing (thanks dkliban!)

It looks like there is no need to update the python package and other publishing scripts for the containers.

Need to make the publishing of images happen, including like how python packages are published.

#16 Updated by mdepaulo@redhat.com about 2 years ago

  • % Done changed from 70 to 100

The remaining work was moved into a new Epic:
https://pulp.plan.io/issues/5393

#17 Updated by mdepaulo@redhat.com about 2 years ago

  • Subject changed from Create pulpcore and pulp_file container images automatically via CI to Create pulp_file container images automatically via CI
  • Description updated (diff)

#18 Updated by mdepaulo@redhat.com about 2 years ago

  • Status changed from ASSIGNED to 5

#19 Updated by mdepaulo@redhat.com about 2 years ago

  • Status changed from 5 to CLOSED - COMPLETE

Please register to edit this issue

Also available in: Atom PDF