Testing Coverage for 2.18.1 will consist of the following
- Migration/Upgrade Testing
- 2.17.0 --> 2.18.1
- 2.17.0 --> 2.18.1
- 2.18.1 Functional Testing Suite
Updating Jenkins Pulp Upgrade Permutations after 2.18.1 GA
Following the GA of 2.18.1, adding the 2.18 upgrade to 2.19.
For the current time, 2.17 is not being removed.
Updated the `README` to remove a specific phrase about Node Pool.
Update pulp role to define defaults for repo_priorty and evalute run-time
As an addendum to PR #644 , the repo_priorty variable in the pulp role
must be defined within the scope of the role (not the pulp_server
playbook) as other playbooks use that role.
Making changes to:
- Declare a default of '99', which is the same as no priority
- Declare a fips priority of '55' for fips pulp installs
- Evaluate the proper priority within the pulp role for the pulp.repos from fips
- Remove the pre_task declaration from the pulp_server playbook
Retested on fips and non-fips with the pulp_server playbook as well
as other playbooks dependent on the pulp role.
Adding the PRs that had to be resolved in 2.18.1b1 to get to the functional test stage this week.
One PR, PR159, is still outstanding and will be indivdually re-checked next week, 2019-02-04
- File 2.18.1_FunctionalTesting_FIPS_2.18.1.b1_20190201.log 2.18.1_FunctionalTesting_FIPS_2.18.1.b1_20190201.log added
- File 2.18.1_FunctionalTesting_nonFIPS_2.18.1.b1_20190204.log 2.18.1_FunctionalTesting_nonFIPS_2.18.1.b1_20190204.log added
- There is a change required in 2.18.1 to have recursive_copy working as we expect
- There are failures in the build ENV that are being worked by PulpQE
- Resolved by pinning pytest to version 4.0.1
- There are ENV failures with CRUD that are not seen locally -- added as a task after 2.18.1b1
- Solved by: https://github.com/pulp/pulp-ci/pull/638
- Uploading logs ran locally. Combination of Jenkins + Local logs has passing FIPS + Non-FIPS, sans the issue in #4375
- non-FIPS (CRUD Failing + #4375): https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Pulp2.18/job/pulp-2.18-dev-rhel7/22/
- non-FIPS Local re-run with all passing on 2.18.1.b1 attached
- FIPS attached (single #4375 failure)
After getting the merged fixes for #4375, issues were seen failing two tests in the test harness:
@dalley is working on a fix to complete testing of the beta.
@zhunting and @pcreech are building and re-run will happen.
- Checklist item Functional :: non-FIPS wiith #4375 :: 2.18.1 set to Done
- Checklist item Functional :: FIPS wiith #4375 :: 2.18.1 set to Done
- File Screenshot from 2019-02-06 07-45-25.png Screenshot from 2019-02-06 07-45-25.png added
- File Screenshot from 2019-02-06 07-51-18.png Screenshot from 2019-02-06 07-51-18.png added
2.18.1b1 passed with the updates @dalley posted on 2019-02-06.
Note that the Jenkin's failure with OpenSuse is due to a flakey test be re-investigated once QE moves to the new cloud in #4385
- 2.18.1-FIPS: https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Pulp2.18/job/pulp-2.18-dev-rhel7-fips/21/
- 2.18.1-nonFIPS: https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Pulp2.18/job/pulp-2.18-dev-rhel7/24/
Since Jenkin's links are mutable, attached are screenshots of the results.
After the posting on 2019-02-06 of an issue related to amqp not restarting properly in #4387, a change to our ansible installer was made to prefer pulp-created RPMs over EPEL (due to amqp failures). This resulted in overnight runs for FIPS and non-FIPS passed without issue.
There will be a discussion about the validity of this approach for test builds.
The known issue, python-amqp, was proven and identified. This should allow completion of the beta phase.
A proper resolution and/or advisement for installation of Pulp 2.18.1b1 for python-amqp will be forth-coming.
Screenshot attached of all green runs with the modified approach to non-FIPS and non-modified FIPS runs below.
- File PreGA-2171-2181_nonFips_upgrade.log PreGA-2171-2181_nonFips_upgrade.log added
- File PreGA-2180-2181_nonFips_upgrade.log PreGA-2180-2181_nonFips_upgrade.log added
- Subject changed from 2.18.1 BETA Testing to 2.18.1 Testing
Final Beta Bits Testing before GA¶
Testing with the final beta bits worked with known work-arounds.
If EPEL repos are present (or any repo providing a python2- packages), the following packages will need to be excluded from upgrade with Pulp 2.18.1:
exclude=python2-kombu python2-celery python2-amqp python2-vine
This will resolve the issue seen in #4387.
All other functional and upgrade tests worked along with a subset upgrade.
- File 2.18.1GA-HotFixUpgradeTest 2.18.1GA-HotFixUpgradeTest added
- Status changed from CLOSED - COMPLETE to MODIFIED
A solution initially implemented was to tell the end user on an upgrade if the celery stack they has was too new and to downgrade.
The output of a failed yum update may have looked like the following:
"--> Processing Conflict: pulp-server-2.18.1-1.el7.noarch conflicts python2-celery >= 4.1", "--> Finished Dependency Resolution", "Error: pulp-server conflicts with python2-celery-4.2.1-3.el7.noarch", "Error: Package: pulp-server-2.18.1-1.el7.noarch (pulp)", " Requires: python2-celery < 4.1", " Installed: python2-celery-4.2.1-3.el7.noarch (@epel)", " python2-celery = 4.2.1-3.el7", " Available: python2-celery-4.0.2-6.el7.noarch (pulp)", " python2-celery = 4.0.2-6.el7", "**********************************************************************", "yum can be configured to try to resolve such errors by temporarily enabling", "disabled repos and searching for missing dependencies.", "To enable this functionality please set 'notify_only=0' in /etc/yum/pluginconf.d/search-disabled-repos.conf", "**********************************************************************", "", " You could try using --skip-broken to work around the problem", " You could try running: rpm -Va --nofiles --nodigest" ] } to retry, use: --limit @/root/code/pulp-ci/ci/ansible/pulp_server_upgrade.retry
It was re-decided the ultimate solution was an upgrade that just worked and didn't require the ,customer to jump through any special hoops to upgrade.
The new solution was to make all celery stack RPMs EPOCH'd in NEVRA. This allows them to supercede any other sources and be preferred. This also means the Pulp Dev team will be required to maintain security and update of these packages.
A listing of some of these packages can be seen in a passing upgrade with the new solution:
"Updated:", " libmodulemd.x86_64 0:1.7.0-1.el7 ", " pulp-admin-client.noarch 0:2.18.1-2.git.111.74e1961.el7 ", " pulp-docker-admin-extensions.noarch 0:3.2.2-1.el7 ", " pulp-docker-plugins.noarch 0:3.2.2-1.el7 ", " pulp-ostree-admin-extensions.noarch 0:1.4.0-1.el7 ", " pulp-ostree-plugins.noarch 0:1.4.0-1.el7 ", " pulp-puppet-admin-extensions.noarch 0:2.18.1-1.el7 ", " pulp-puppet-plugins.noarch 0:2.18.1-1.el7 ", " pulp-puppet-tools.noarch 0:2.18.1-1.el7 ", " pulp-rpm-admin-extensions.noarch 0:2.18.1-1.el7 ", " pulp-rpm-plugins.noarch 0:2.18.1-1.el7 ", " pulp-selinux.noarch 0:2.18.1-2.git.111.74e1961.el7 ", " pulp-server.noarch 0:2.18.1-2.git.111.74e1961.el7 ", " python-crane.noarch 0:3.3.0-1.el7 ", " python-gofer.noarch 0:2.12.5-1.el7 ", " python-gofer-qpid.noarch 0:2.12.5-1.el7 ", " python-pulp-bindings.noarch 0:2.18.1-2.git.111.74e1961.el7 ", " python-pulp-client-lib.noarch 0:2.18.1-2.git.111.74e1961.el7 ", " python-pulp-common.noarch 0:2.18.1-2.git.111.74e1961.el7 ", " python-pulp-docker-common.noarch 0:3.2.2-1.el7 ", " python-pulp-oid_validation.noarch 0:2.18.1-2.git.111.74e1961.el7 ", " python-pulp-ostree-common.noarch 0:1.4.0-1.el7 ", " python-pulp-puppet-common.noarch 0:2.18.1-1.el7 ", " python-pulp-repoauth.noarch 0:2.18.1-2.git.111.74e1961.el7 ", " python-pulp-rpm-common.noarch 0:2.18.1-1.el7 ", " python-pulp-streamer.noarch 0:2.18.1-2.git.111.74e1961.el7 ", " python2-amqp.noarch 10:2.2.2-5.git.111.74e1961.el7 ", " python2-billiard.x86_64 10:126.96.36.199-4.git.110.4fbf91e.el7 ", " python2-celery.noarch 10:4.0.2-7.git.111.74e1961.el7 ", " python2-kombu.noarch 10:4.0.2-11.git.111.74e1961.el7 ", " python2-vine.noarch 10:1.1.3-5.git.110.4fbf91e.el7 ", "", "Complete!"
These bits were tested in beta and are now being pushed to the GA location.
- File pulp-upgrade-2.17-stable-2.18-stable-rhel7-fips.log pulp-upgrade-2.17-stable-2.18-stable-rhel7-fips.log added
- File pulp-upgrade-2.17-stable-2.18-stable-rhel7.log pulp-upgrade-2.17-stable-2.18-stable-rhel7.log added
GA Upgrade Test¶
Both RHEL non-fips and RHEL FIPs upgrade testing passed without issue on Jenkins.
Please register to edit this issue