Project

Profile

Help

Test #4359

closed

2.18.1 Testing

Added by bherring almost 6 years ago. Updated over 5 years ago.

Status:
CLOSED - COMPLETE
Priority:
Urgent
Assignee:
Category:
-
Sprint/Milestone:
-
Version:
Platform Release:
Tags:
Pulp 2
Sprint:
Quarter:

Description

Notes

Testing Coverage for 2.18.1 will consist of the following

  • Migration/Upgrade Testing
    • 2.17.0 --> 2.18.1
      • FIPS
      • Non-FIPS
  • 2.18.1 Functional Testing Suite
    • FIPS
    • Non-FIPS

Files

2.18.1_UpgradeTesting_FIPS_2.17.0_2.18.1_2019012091.log (1.9 MB) 2.18.1_UpgradeTesting_FIPS_2.17.0_2.18.1_2019012091.log bherring, 01/29/2019 08:23 PM
2.18.1_UpgradeTesting_FIPS_2.18.0_2.18.1_2019012091.log (2.23 MB) 2.18.1_UpgradeTesting_FIPS_2.18.0_2.18.1_2019012091.log bherring, 01/29/2019 08:23 PM
2.18.1_UpgradeTesting_nonFIPS_2.17.0_2.18.1_2019012091.log (1.64 MB) 2.18.1_UpgradeTesting_nonFIPS_2.17.0_2.18.1_2019012091.log bherring, 01/30/2019 06:13 PM
2.18.1_UpgradeTesting_nonFIPS_2.18.0_2.18.1_2019012091.log (1.69 MB) 2.18.1_UpgradeTesting_nonFIPS_2.18.0_2.18.1_2019012091.log bherring, 01/30/2019 06:13 PM
2.18.1_FunctionalTesting_FIPS_2.18.1.b1_20190201.log (1.05 MB) 2.18.1_FunctionalTesting_FIPS_2.18.1.b1_20190201.log All pass + re-run minus #4375 failure (to be pushed to QE later) bherring, 02/04/2019 02:57 PM
2.18.1_FunctionalTesting_nonFIPS_2.18.1.b1_20190204.log (453 KB) 2.18.1_FunctionalTesting_nonFIPS_2.18.1.b1_20190204.log Jenkins + local re-runs minus #4375 (to be pushed to QE later) bherring, 02/04/2019 03:08 PM
Screenshot from 2019-02-06 07-45-25.png (129 KB) Screenshot from 2019-02-06 07-45-25.png bherring, 02/06/2019 01:51 PM
Screenshot from 2019-02-06 07-51-18.png (241 KB) Screenshot from 2019-02-06 07-51-18.png bherring, 02/06/2019 01:51 PM
Screenshot from 2019-02-07 07-33-30.png (51.9 KB) Screenshot from 2019-02-07 07-33-30.png All green runs 2019-02-07 bherring, 02/07/2019 01:44 PM
PreGA-2171-2181_nonFips_upgrade.log (33.4 KB) PreGA-2171-2181_nonFips_upgrade.log upgrade testing log bherring, 02/20/2019 08:47 PM
PreGA-2180-2181_nonFips_upgrade.log (713 KB) PreGA-2180-2181_nonFips_upgrade.log upgrade testing log bherring, 02/20/2019 08:47 PM
2.18.1GA-HotFixUpgradeTest (449 KB) 2.18.1GA-HotFixUpgradeTest bherring, 02/25/2019 01:22 PM
pulp-upgrade-2.17-stable-2.18-stable-rhel7-fips.log (95.4 KB) pulp-upgrade-2.17-stable-2.18-stable-rhel7-fips.log bherring, 02/26/2019 01:28 PM
pulp-upgrade-2.17-stable-2.18-stable-rhel7.log (90.6 KB) pulp-upgrade-2.17-stable-2.18-stable-rhel7.log bherring, 02/26/2019 01:28 PM

Related issues

Related to Pulp - Test #4357: Pulp 2.19 master pulled epel django instead of pulp 2 djangoCLOSED - COMPLETEbherringActions
Related to RPM Support - Test #4351: modules.yaml reference in repomd.xml does not use selected checksumCLOSED - COMPLETEragbalakActions
Related to RPM Support - Test #4269: Recursive and conservative recursive copyCLOSED - COMPLETEkersomActions
Related to RPM Support - Issue #4375: Recursive copy doesn't solve rich dependencies correctlyCLOSED - CURRENTRELEASEdalleyActions
Related to Pulp - Issue #4387: Epel is shipping a new version of celery which doesn't work with PulpCLOSED - CURRENTRELEASEActions
Actions #1

Updated by bherring almost 6 years ago

Actions #3

Updated by bherring almost 6 years ago

  • Related to Test #4357: Pulp 2.19 master pulled epel django instead of pulp 2 django added
Actions #4

Updated by bherring almost 6 years ago

  • Related to Test #4351: modules.yaml reference in repomd.xml does not use selected checksum added
Actions #5

Updated by bherring almost 6 years ago

  • Related to Issue #4296: Stages API could deadlock when "discovering" content due to minsize added
Actions #6

Updated by bherring almost 6 years ago

  • Related to deleted (Issue #4296: Stages API could deadlock when "discovering" content due to minsize)
Actions #7

Updated by bherring almost 6 years ago

  • Related to Test #4269: Recursive and conservative recursive copy added
Actions #8

Updated by bherring almost 6 years ago

Actions #9

Updated by bherring almost 6 years ago

Actions #11

Updated by bherring almost 6 years ago

Notes

Adding the PRs that had to be resolved in 2.18.1b1 to get to the functional test stage this week.

One PR, PR159, is still outstanding and will be indivdually re-checked next week, 2019-02-04

Actions #12

Updated by bherring almost 6 years ago

Notes

Actions #13

Updated by bherring almost 6 years ago

Actions #14

Updated by bherring almost 6 years ago

  • Related to Issue #4375: Recursive copy doesn't solve rich dependencies correctly added
Actions #15

Updated by bherring almost 6 years ago

2.18.1b1 with #4375

After getting the merged fixes for #4375, issues were seen failing two tests in the test harness:

dalley is working on a fix to complete testing of the beta.

zhunting and pcreech are building and re-run will happen.

Actions #16

Updated by bherring almost 6 years ago

Final Update

2.18.1b1 passed with the updates dalley posted on 2019-02-06.

Note that the Jenkin's failure with OpenSuse is due to a flakey test be re-investigated once QE moves to the new cloud in #4385

Since Jenkin's links are mutable, attached are screenshots of the results.

Actions #17

Updated by bherring almost 6 years ago

  • Status changed from ASSIGNED to MODIFIED
Actions #18

Updated by bherring almost 6 years ago

  • Status changed from MODIFIED to CLOSED - COMPLETE
Actions #19

Updated by bherring almost 6 years ago

  • Related to Issue #4387: Epel is shipping a new version of celery which doesn't work with Pulp added
Actions #20

Updated by bherring almost 6 years ago

Notes

After the posting on 2019-02-06 of an issue related to amqp not restarting properly in #4387, a change to our ansible installer was made to prefer pulp-created RPMs over EPEL (due to amqp failures). This resulted in overnight runs for FIPS and non-FIPS passed without issue.

There will be a discussion about the validity of this approach for test builds.

The known issue, python-amqp, was proven and identified. This should allow completion of the beta phase.

A proper resolution and/or advisement for installation of Pulp 2.18.1b1 for python-amqp will be forth-coming.

Screenshot attached of all green runs with the modified approach to non-FIPS and non-modified FIPS runs below.

Actions #21

Updated by bherring over 5 years ago

  • Subject changed from 2.18.1 RC Testing to 2.18.1 BETA Testing
Actions #22

Updated by bherring over 5 years ago

Final Beta Bits Testing before GA

Testing with the final beta bits worked with known work-arounds.

If EPEL repos are present (or any repo providing a python2- packages), the following packages will need to be excluded from upgrade with Pulp 2.18.1:

exclude=python2-kombu python2-celery python2-amqp python2-vine

This will resolve the issue seen in #4387.

All other functional and upgrade tests worked along with a subset upgrade.

Actions #23

Updated by bherring over 5 years ago

Hot-Fix Test

A solution initially implemented was to tell the end user on an upgrade if the celery stack they has was too new and to downgrade.

The output of a failed yum update may have looked like the following:

        "--> Processing Conflict: pulp-server-2.18.1-1.el7.noarch conflicts python2-celery >= 4.1",
        "--> Finished Dependency Resolution",
        "Error: pulp-server conflicts with python2-celery-4.2.1-3.el7.noarch",
        "Error: Package: pulp-server-2.18.1-1.el7.noarch (pulp)",
        "           Requires: python2-celery < 4.1",
        "           Installed: python2-celery-4.2.1-3.el7.noarch (@epel)",
        "               python2-celery = 4.2.1-3.el7",
        "           Available: python2-celery-4.0.2-6.el7.noarch (pulp)",
        "               python2-celery = 4.0.2-6.el7",
        "**********************************************************************",
        "yum can be configured to try to resolve such errors by temporarily enabling",
        "disabled repos and searching for missing dependencies.",
        "To enable this functionality please set 'notify_only=0' in /etc/yum/pluginconf.d/search-disabled-repos.conf", 
        "**********************************************************************",
        "", 
        " You could try using --skip-broken to work around the problem",
        " You could try running: rpm -Va --nofiles --nodigest"
    ]
}
        to retry, use: --limit @/root/code/pulp-ci/ci/ansible/pulp_server_upgrade.retry

It was re-decided the ultimate solution was an upgrade that just worked and didn't require the ,customer to jump through any special hoops to upgrade.

The new solution was to make all celery stack RPMs EPOCH'd in NEVRA. This allows them to supercede any other sources and be preferred. This also means the Pulp Dev team will be required to maintain security and update of these packages.

A listing of some of these packages can be seen in a passing upgrade with the new solution:

        "Updated:", 
        "  libmodulemd.x86_64 0:1.7.0-1.el7                                              ", 
        "  pulp-admin-client.noarch 0:2.18.1-2.git.111.74e1961.el7                       ", 
        "  pulp-docker-admin-extensions.noarch 0:3.2.2-1.el7                             ", 
        "  pulp-docker-plugins.noarch 0:3.2.2-1.el7                                      ", 
        "  pulp-ostree-admin-extensions.noarch 0:1.4.0-1.el7                             ", 
        "  pulp-ostree-plugins.noarch 0:1.4.0-1.el7                                      ", 
        "  pulp-puppet-admin-extensions.noarch 0:2.18.1-1.el7                            ", 
        "  pulp-puppet-plugins.noarch 0:2.18.1-1.el7                                     ", 
        "  pulp-puppet-tools.noarch 0:2.18.1-1.el7                                       ", 
        "  pulp-rpm-admin-extensions.noarch 0:2.18.1-1.el7                               ", 
        "  pulp-rpm-plugins.noarch 0:2.18.1-1.el7                                        ", 
        "  pulp-selinux.noarch 0:2.18.1-2.git.111.74e1961.el7                            ", 
        "  pulp-server.noarch 0:2.18.1-2.git.111.74e1961.el7                             ", 
        "  python-crane.noarch 0:3.3.0-1.el7                                             ", 
        "  python-gofer.noarch 0:2.12.5-1.el7                                            ", 
        "  python-gofer-qpid.noarch 0:2.12.5-1.el7                                       ", 
        "  python-pulp-bindings.noarch 0:2.18.1-2.git.111.74e1961.el7                    ", 
        "  python-pulp-client-lib.noarch 0:2.18.1-2.git.111.74e1961.el7                  ", 
        "  python-pulp-common.noarch 0:2.18.1-2.git.111.74e1961.el7                      ", 
        "  python-pulp-docker-common.noarch 0:3.2.2-1.el7                                ", 
        "  python-pulp-oid_validation.noarch 0:2.18.1-2.git.111.74e1961.el7              ", 
        "  python-pulp-ostree-common.noarch 0:1.4.0-1.el7                                ", 
        "  python-pulp-puppet-common.noarch 0:2.18.1-1.el7                               ", 
        "  python-pulp-repoauth.noarch 0:2.18.1-2.git.111.74e1961.el7                    ", 
        "  python-pulp-rpm-common.noarch 0:2.18.1-1.el7                                  ", 
        "  python-pulp-streamer.noarch 0:2.18.1-2.git.111.74e1961.el7                    ", 
        "  python2-amqp.noarch 10:2.2.2-5.git.111.74e1961.el7                            ", 
        "  python2-billiard.x86_64 10:3.5.0.3-4.git.110.4fbf91e.el7                      ", 
        "  python2-celery.noarch 10:4.0.2-7.git.111.74e1961.el7                          ", 
        "  python2-kombu.noarch 10:4.0.2-11.git.111.74e1961.el7                          ", 
        "  python2-vine.noarch 10:1.1.3-5.git.110.4fbf91e.el7                            ", 
        "", 
        "Complete!"

These bits were tested in beta and are now being pushed to the GA location.

Actions #25

Updated by bherring over 5 years ago

  • Status changed from MODIFIED to CLOSED - COMPLETE

Added by bherring over 5 years ago

Revision 9be07f44 | View on GitHub

Updating Jenkins Pulp Upgrade Permutations after 2.18.1 GA

Following the GA of 2.18.1, adding the 2.18 upgrade to 2.19.

For the current time, 2.17 is not being removed.

Updated the README to remove a specific phrase about Node Pool.

refs #4359

Added by bherring over 5 years ago

Revision ff598dda | View on GitHub

Update pulp role to define defaults for repo_priorty and evalute run-time

As an addendum to PR #644 , the repo_priorty variable in the pulp role must be defined within the scope of the role (not the pulp_server playbook) as other playbooks use that role.

Making changes to:

  • Declare a default of '99', which is the same as no priority
  • Declare a fips priority of '55' for fips pulp installs
  • Evaluate the proper priority within the pulp role for the pulp.repos from fips
  • Remove the pre_task declaration from the pulp_server playbook

Retested on fips and non-fips with the pulp_server playbook as well as other playbooks dependent on the pulp role.

refs #4359

Actions #26

Updated by bmbouter over 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF