Pulp

Notes¶

• There is a change required in 2.18.1 to have recursive_copy working as we expect
• There are failures in the build ENV that are being worked by PulpQE
  • Resolved by pinning pytest to version 4.0.1
  • https://github.com/pulp/pulp-ci/pull/638
• There are ENV failures with CRUD that are not seen locally -- added as a task after 2.18.1b1
  • Solved by: https://github.com/pulp/pulp-ci/pull/638
• Uploading logs ran locally. Combination of Jenkins + Local logs has passing FIPS + Non-FIPS, sans the issue in #4375
  • non-FIPS (CRUD Failing + #4375): https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Pulp2.18/job/pulp-2.18-dev-rhel7/22/
  • non-FIPS Local re-run with all passing on 2.18.1.b1 attached
  • FIPS attached (single #4375 failure)

Final Update¶

2.18.1b1 passed with the updates dalley posted on 2019-02-06.

Note that the Jenkin's failure with OpenSuse is due to a flakey test be re-investigated once QE moves to the new cloud in #4385

• 2.18.1-FIPS: https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Pulp2.18/job/pulp-2.18-dev-rhel7-fips/21/
• 2.18.1-nonFIPS: https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Pulp2.18/job/pulp-2.18-dev-rhel7/24/

Since Jenkin's links are mutable, attached are screenshots of the results.

Notes¶

After the posting on 2019-02-06 of an issue related to amqp not restarting properly in #4387, a change to our ansible installer was made to prefer pulp-created RPMs over EPEL (due to amqp failures). This resulted in overnight runs for FIPS and non-FIPS passed without issue.

There will be a discussion about the validity of this approach for test builds.

The known issue, python-amqp, was proven and identified. This should allow completion of the beta phase.

A proper resolution and/or advisement for installation of Pulp 2.18.1b1 for python-amqp will be forth-coming.

Screenshot attached of all green runs with the modified approach to non-FIPS and non-modified FIPS runs below.

• https://github.com/pulp/pulp-ci/pull/641

Final Beta Bits Testing before GA¶

Testing with the final beta bits worked with known work-arounds.

If EPEL repos are present (or any repo providing a python2- packages), the following packages will need to be excluded from upgrade with Pulp 2.18.1:

exclude=python2-kombu python2-celery python2-amqp python2-vine

This will resolve the issue seen in #4387.

All other functional and upgrade tests worked along with a subset upgrade.

Hot-Fix Test¶

A solution initially implemented was to tell the end user on an upgrade if the celery stack they has was too new and to downgrade.

The output of a failed yum update may have looked like the following:

        "--> Processing Conflict: pulp-server-2.18.1-1.el7.noarch conflicts python2-celery >= 4.1",
        "--> Finished Dependency Resolution",
        "Error: pulp-server conflicts with python2-celery-4.2.1-3.el7.noarch",
        "Error: Package: pulp-server-2.18.1-1.el7.noarch (pulp)",
        "           Requires: python2-celery < 4.1",
        "           Installed: python2-celery-4.2.1-3.el7.noarch (@epel)",
        "               python2-celery = 4.2.1-3.el7",
        "           Available: python2-celery-4.0.2-6.el7.noarch (pulp)",
        "               python2-celery = 4.0.2-6.el7",
        "**********************************************************************",
        "yum can be configured to try to resolve such errors by temporarily enabling",
        "disabled repos and searching for missing dependencies.",
        "To enable this functionality please set 'notify_only=0' in /etc/yum/pluginconf.d/search-disabled-repos.conf", 
        "**********************************************************************",
        "", 
        " You could try using --skip-broken to work around the problem",
        " You could try running: rpm -Va --nofiles --nodigest"
    ]
}
        to retry, use: --limit @/root/code/pulp-ci/ci/ansible/pulp_server_upgrade.retry

It was re-decided the ultimate solution was an upgrade that just worked and didn't require the ,customer to jump through any special hoops to upgrade.

The new solution was to make all celery stack RPMs EPOCH'd in NEVRA. This allows them to supercede any other sources and be preferred. This also means the Pulp Dev team will be required to maintain security and update of these packages.

A listing of some of these packages can be seen in a passing upgrade with the new solution:

        "Updated:", 
        "  libmodulemd.x86_64 0:1.7.0-1.el7                                              ", 
        "  pulp-admin-client.noarch 0:2.18.1-2.git.111.74e1961.el7                       ", 
        "  pulp-docker-admin-extensions.noarch 0:3.2.2-1.el7                             ", 
        "  pulp-docker-plugins.noarch 0:3.2.2-1.el7                                      ", 
        "  pulp-ostree-admin-extensions.noarch 0:1.4.0-1.el7                             ", 
        "  pulp-ostree-plugins.noarch 0:1.4.0-1.el7                                      ", 
        "  pulp-puppet-admin-extensions.noarch 0:2.18.1-1.el7                            ", 
        "  pulp-puppet-plugins.noarch 0:2.18.1-1.el7                                     ", 
        "  pulp-puppet-tools.noarch 0:2.18.1-1.el7                                       ", 
        "  pulp-rpm-admin-extensions.noarch 0:2.18.1-1.el7                               ", 
        "  pulp-rpm-plugins.noarch 0:2.18.1-1.el7                                        ", 
        "  pulp-selinux.noarch 0:2.18.1-2.git.111.74e1961.el7                            ", 
        "  pulp-server.noarch 0:2.18.1-2.git.111.74e1961.el7                             ", 
        "  python-crane.noarch 0:3.3.0-1.el7                                             ", 
        "  python-gofer.noarch 0:2.12.5-1.el7                                            ", 
        "  python-gofer-qpid.noarch 0:2.12.5-1.el7                                       ", 
        "  python-pulp-bindings.noarch 0:2.18.1-2.git.111.74e1961.el7                    ", 
        "  python-pulp-client-lib.noarch 0:2.18.1-2.git.111.74e1961.el7                  ", 
        "  python-pulp-common.noarch 0:2.18.1-2.git.111.74e1961.el7                      ", 
        "  python-pulp-docker-common.noarch 0:3.2.2-1.el7                                ", 
        "  python-pulp-oid_validation.noarch 0:2.18.1-2.git.111.74e1961.el7              ", 
        "  python-pulp-ostree-common.noarch 0:1.4.0-1.el7                                ", 
        "  python-pulp-puppet-common.noarch 0:2.18.1-1.el7                               ", 
        "  python-pulp-repoauth.noarch 0:2.18.1-2.git.111.74e1961.el7                    ", 
        "  python-pulp-rpm-common.noarch 0:2.18.1-1.el7                                  ", 
        "  python-pulp-streamer.noarch 0:2.18.1-2.git.111.74e1961.el7                    ", 
        "  python2-amqp.noarch 10:2.2.2-5.git.111.74e1961.el7                            ", 
        "  python2-billiard.x86_64 10:3.5.0.3-4.git.110.4fbf91e.el7                      ", 
        "  python2-celery.noarch 10:4.0.2-7.git.111.74e1961.el7                          ", 
        "  python2-kombu.noarch 10:4.0.2-11.git.111.74e1961.el7                          ", 
        "  python2-vine.noarch 10:1.1.3-5.git.110.4fbf91e.el7                            ", 
        "", 
        "Complete!"

These bits were tested in beta and are now being pushed to the GA location.

GA Upgrade Test¶

Both RHEL non-fips and RHEL FIPs upgrade testing passed without issue on Jenkins.