Project

Profile

Help

Test #4359

2.18.1 Testing

Added by bherring 9 months ago. Updated 4 months ago.

Status:
CLOSED - COMPLETE
Priority:
Urgent
Assignee:
Category:
-
Sprint/Milestone:
-
Version:
Platform Release:
Tags:
Pulp 2
Sprint:

Description

Notes

Testing Coverage for 2.18.1 will consist of the following

  • Migration/Upgrade Testing
    • 2.17.0 --> 2.18.1
      • FIPS
      • Non-FIPS
  • 2.18.1 Functional Testing Suite
    • FIPS
    • Non-FIPS
2.18.1_UpgradeTesting_FIPS_2.17.0_2.18.1_2019012091.log (1.9 MB) bherring, 01/29/2019 08:23 PM 2.18.1_UpgradeTesting_FIPS_2.17.0_2.18.1_2019012091.log
2.18.1_UpgradeTesting_FIPS_2.18.0_2.18.1_2019012091.log (2.23 MB) bherring, 01/29/2019 08:23 PM 2.18.1_UpgradeTesting_FIPS_2.18.0_2.18.1_2019012091.log
2.18.1_UpgradeTesting_nonFIPS_2.17.0_2.18.1_2019012091.log (1.64 MB) bherring, 01/30/2019 06:13 PM 2.18.1_UpgradeTesting_nonFIPS_2.17.0_2.18.1_2019012091.log
2.18.1_UpgradeTesting_nonFIPS_2.18.0_2.18.1_2019012091.log (1.69 MB) bherring, 01/30/2019 06:13 PM 2.18.1_UpgradeTesting_nonFIPS_2.18.0_2.18.1_2019012091.log
2.18.1_FunctionalTesting_FIPS_2.18.1.b1_20190201.log (1.05 MB) All pass + re-run minus #4375 failure (to be pushed to QE later) bherring, 02/04/2019 02:57 PM 2.18.1_FunctionalTesting_FIPS_2.18.1.b1_20190201.log
2.18.1_FunctionalTesting_nonFIPS_2.18.1.b1_20190204.log (453 KB) Jenkins + local re-runs minus #4375 (to be pushed to QE later) bherring, 02/04/2019 03:08 PM 2.18.1_FunctionalTesting_nonFIPS_2.18.1.b1_20190204.log
Screenshot from 2019-02-06 07-45-25.png (129 KB) bherring, 02/06/2019 01:51 PM Screenshot from 2019-02-06 07-45-25.png
Screenshot from 2019-02-06 07-51-18.png (241 KB) bherring, 02/06/2019 01:51 PM Screenshot from 2019-02-06 07-51-18.png
Screenshot from 2019-02-07 07-33-30.png (51.9 KB) All green runs 2019-02-07 bherring, 02/07/2019 01:44 PM Screenshot from 2019-02-07 07-33-30.png
PreGA-2171-2181_nonFips_upgrade.log (33.4 KB) upgrade testing log bherring, 02/20/2019 08:47 PM PreGA-2171-2181_nonFips_upgrade.log
PreGA-2180-2181_nonFips_upgrade.log (713 KB) upgrade testing log bherring, 02/20/2019 08:47 PM PreGA-2180-2181_nonFips_upgrade.log
2.18.1GA-HotFixUpgradeTest (449 KB) bherring, 02/25/2019 01:22 PM 2.18.1GA-HotFixUpgradeTest
pulp-upgrade-2.17-stable-2.18-stable-rhel7-fips.log (95.4 KB) bherring, 02/26/2019 01:28 PM pulp-upgrade-2.17-stable-2.18-stable-rhel7-fips.log
pulp-upgrade-2.17-stable-2.18-stable-rhel7.log (90.6 KB) bherring, 02/26/2019 01:28 PM pulp-upgrade-2.17-stable-2.18-stable-rhel7.log
250
250
250

Checklist


Related issues

Related to Pulp - Test #4357: Pulp 2.19 master pulled epel django instead of pulp 2 django CLOSED - COMPLETE Actions
Related to RPM Support - Test #4351: modules.yaml reference in repomd.xml does not use selected checksum CLOSED - COMPLETE Actions
Related to RPM Support - Test #4269: Recursive and conservative recursive copy CLOSED - COMPLETE Actions
Related to RPM Support - Issue #4375: Recursive copy doesn't solve rich dependencies correctly CLOSED - CURRENTRELEASE Actions
Related to Pulp - Issue #4387: Epel is shipping a new version of celery which doesn't work with Pulp CLOSED - CURRENTRELEASE Actions

Associated revisions

Revision 9be07f44 View on GitHub
Added by bherring 8 months ago

Updating Jenkins Pulp Upgrade Permutations after 2.18.1 GA

Following the GA of 2.18.1, adding the 2.18 upgrade to 2.19.

For the current time, 2.17 is not being removed.

Updated the `README` to remove a specific phrase about Node Pool.

refs #4359

Revision ff598dda View on GitHub
Added by bherring 8 months ago

Update pulp role to define defaults for repo_priorty and evalute run-time

As an addendum to PR #644 , the repo_priorty variable in the pulp role
must be defined within the scope of the role (not the pulp_server
playbook) as other playbooks use that role.

Making changes to:
- Declare a default of '99', which is the same as no priority
- Declare a fips priority of '55' for fips pulp installs
- Evaluate the proper priority within the pulp role for the pulp.repos from fips
- Remove the pre_task declaration from the pulp_server playbook

Retested on fips and non-fips with the pulp_server playbook as well
as other playbooks dependent on the pulp role.

refs #4359

History

#1 Updated by bherring 9 months ago

  • Checklist item Upgrade :: FIPS :: 2.17.0 --> 2.18.1 set to Done
  • Checklist item Upgrade :: FIPS :: 2.18.0 --> 2.18.1 set to Done

#3 Updated by bherring 9 months ago

  • Related to Test #4357: Pulp 2.19 master pulled epel django instead of pulp 2 django added

#4 Updated by bherring 9 months ago

  • Related to Test #4351: modules.yaml reference in repomd.xml does not use selected checksum added

#5 Updated by bherring 9 months ago

  • Related to Issue #4296: Stages API could deadlock when "discovering" content due to minsize added

#6 Updated by bherring 9 months ago

  • Related to deleted (Issue #4296: Stages API could deadlock when "discovering" content due to minsize)

#7 Updated by bherring 9 months ago

  • Related to Test #4269: Recursive and conservative recursive copy added

#8 Updated by bherring 9 months ago

  • Checklist item Upgrade :: non-FIPS :: 2.17.0 --> 2.18.1 set to Done

#9 Updated by bherring 9 months ago

  • Checklist item Upgrade :: non-FIPS :: 2.18.0 --> 2.18.1 set to Done

#11 Updated by bherring 9 months ago

Notes

Adding the PRs that had to be resolved in 2.18.1b1 to get to the functional test stage this week.

One PR, PR159, is still outstanding and will be indivdually re-checked next week, 2019-02-04

#12 Updated by bherring 8 months ago

Notes

#13 Updated by bherring 8 months ago

  • Checklist item Functional :: non-FIPS wiith #4375 :: 2.18.1 added
  • Checklist item Functional :: FIPS wiith #4375 :: 2.18.1 added
  • Checklist item Functional :: non-FIPS :: 2.18.1 set to Done
  • Checklist item Functional :: FIPS :: 2.18.1 set to Done

#14 Updated by bherring 8 months ago

  • Related to Issue #4375: Recursive copy doesn't solve rich dependencies correctly added

#15 Updated by bherring 8 months ago

2.18.1b1 with #4375

After getting the merged fixes for #4375, issues were seen failing two tests in the test harness:

@dalley is working on a fix to complete testing of the beta.

@zhunting and @pcreech are building and re-run will happen.

#16 Updated by bherring 8 months ago

250
250

Final Update

2.18.1b1 passed with the updates @dalley posted on 2019-02-06.

Note that the Jenkin's failure with OpenSuse is due to a flakey test be re-investigated once QE moves to the new cloud in #4385

Since Jenkin's links are mutable, attached are screenshots of the results.

#17 Updated by bherring 8 months ago

  • Status changed from ASSIGNED to MODIFIED

#18 Updated by bherring 8 months ago

  • Status changed from MODIFIED to CLOSED - COMPLETE

#19 Updated by bherring 8 months ago

  • Related to Issue #4387: Epel is shipping a new version of celery which doesn't work with Pulp added

#20 Updated by bherring 8 months ago

250

Notes

After the posting on 2019-02-06 of an issue related to amqp not restarting properly in #4387, a change to our ansible installer was made to prefer pulp-created RPMs over EPEL (due to amqp failures). This resulted in overnight runs for FIPS and non-FIPS passed without issue.

There will be a discussion about the validity of this approach for test builds.

The known issue, python-amqp, was proven and identified. This should allow completion of the beta phase.

A proper resolution and/or advisement for installation of Pulp 2.18.1b1 for python-amqp will be forth-coming.

Screenshot attached of all green runs with the modified approach to non-FIPS and non-modified FIPS runs below.

#21 Updated by bherring 8 months ago

  • Subject changed from 2.18.1 RC Testing to 2.18.1 BETA Testing

#22 Updated by bherring 8 months ago

Final Beta Bits Testing before GA

Testing with the final beta bits worked with known work-arounds.

If EPEL repos are present (or any repo providing a python2- packages), the following packages will need to be excluded from upgrade with Pulp 2.18.1:

exclude=python2-kombu python2-celery python2-amqp python2-vine

This will resolve the issue seen in #4387.

All other functional and upgrade tests worked along with a subset upgrade.

#23 Updated by bherring 8 months ago

Hot-Fix Test

A solution initially implemented was to tell the end user on an upgrade if the celery stack they has was too new and to downgrade.

The output of a failed yum update may have looked like the following:

        "--> Processing Conflict: pulp-server-2.18.1-1.el7.noarch conflicts python2-celery >= 4.1",
        "--> Finished Dependency Resolution",
        "Error: pulp-server conflicts with python2-celery-4.2.1-3.el7.noarch",
        "Error: Package: pulp-server-2.18.1-1.el7.noarch (pulp)",
        "           Requires: python2-celery < 4.1",
        "           Installed: python2-celery-4.2.1-3.el7.noarch (@epel)",
        "               python2-celery = 4.2.1-3.el7",
        "           Available: python2-celery-4.0.2-6.el7.noarch (pulp)",
        "               python2-celery = 4.0.2-6.el7",
        "**********************************************************************",
        "yum can be configured to try to resolve such errors by temporarily enabling",
        "disabled repos and searching for missing dependencies.",
        "To enable this functionality please set 'notify_only=0' in /etc/yum/pluginconf.d/search-disabled-repos.conf", 
        "**********************************************************************",
        "", 
        " You could try using --skip-broken to work around the problem",
        " You could try running: rpm -Va --nofiles --nodigest" 
    ]
}
        to retry, use: --limit @/root/code/pulp-ci/ci/ansible/pulp_server_upgrade.retry

It was re-decided the ultimate solution was an upgrade that just worked and didn't require the ,customer to jump through any special hoops to upgrade.

The new solution was to make all celery stack RPMs EPOCH'd in NEVRA. This allows them to supercede any other sources and be preferred. This also means the Pulp Dev team will be required to maintain security and update of these packages.

A listing of some of these packages can be seen in a passing upgrade with the new solution:

        "Updated:", 
        "  libmodulemd.x86_64 0:1.7.0-1.el7                                              ", 
        "  pulp-admin-client.noarch 0:2.18.1-2.git.111.74e1961.el7                       ", 
        "  pulp-docker-admin-extensions.noarch 0:3.2.2-1.el7                             ", 
        "  pulp-docker-plugins.noarch 0:3.2.2-1.el7                                      ", 
        "  pulp-ostree-admin-extensions.noarch 0:1.4.0-1.el7                             ", 
        "  pulp-ostree-plugins.noarch 0:1.4.0-1.el7                                      ", 
        "  pulp-puppet-admin-extensions.noarch 0:2.18.1-1.el7                            ", 
        "  pulp-puppet-plugins.noarch 0:2.18.1-1.el7                                     ", 
        "  pulp-puppet-tools.noarch 0:2.18.1-1.el7                                       ", 
        "  pulp-rpm-admin-extensions.noarch 0:2.18.1-1.el7                               ", 
        "  pulp-rpm-plugins.noarch 0:2.18.1-1.el7                                        ", 
        "  pulp-selinux.noarch 0:2.18.1-2.git.111.74e1961.el7                            ", 
        "  pulp-server.noarch 0:2.18.1-2.git.111.74e1961.el7                             ", 
        "  python-crane.noarch 0:3.3.0-1.el7                                             ", 
        "  python-gofer.noarch 0:2.12.5-1.el7                                            ", 
        "  python-gofer-qpid.noarch 0:2.12.5-1.el7                                       ", 
        "  python-pulp-bindings.noarch 0:2.18.1-2.git.111.74e1961.el7                    ", 
        "  python-pulp-client-lib.noarch 0:2.18.1-2.git.111.74e1961.el7                  ", 
        "  python-pulp-common.noarch 0:2.18.1-2.git.111.74e1961.el7                      ", 
        "  python-pulp-docker-common.noarch 0:3.2.2-1.el7                                ", 
        "  python-pulp-oid_validation.noarch 0:2.18.1-2.git.111.74e1961.el7              ", 
        "  python-pulp-ostree-common.noarch 0:1.4.0-1.el7                                ", 
        "  python-pulp-puppet-common.noarch 0:2.18.1-1.el7                               ", 
        "  python-pulp-repoauth.noarch 0:2.18.1-2.git.111.74e1961.el7                    ", 
        "  python-pulp-rpm-common.noarch 0:2.18.1-1.el7                                  ", 
        "  python-pulp-streamer.noarch 0:2.18.1-2.git.111.74e1961.el7                    ", 
        "  python2-amqp.noarch 10:2.2.2-5.git.111.74e1961.el7                            ", 
        "  python2-billiard.x86_64 10:3.5.0.3-4.git.110.4fbf91e.el7                      ", 
        "  python2-celery.noarch 10:4.0.2-7.git.111.74e1961.el7                          ", 
        "  python2-kombu.noarch 10:4.0.2-11.git.111.74e1961.el7                          ", 
        "  python2-vine.noarch 10:1.1.3-5.git.110.4fbf91e.el7                            ", 
        "", 
        "Complete!" 

These bits were tested in beta and are now being pushed to the GA location.

#25 Updated by bherring 8 months ago

  • Status changed from MODIFIED to CLOSED - COMPLETE

#26 Updated by bmbouter 6 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF