Project

Profile

Help

Task #1415

closed

Write a pre 2.8 upgrade script

Added by bmbouter over 8 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Platform Release:
2.8.0
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Quarter:

Description

Starting with the 2.8.0 many database collections will have validation checks on them. It's likely that there are fields in existing Pulp databases which will not validate. This is true in large part because there was no validation in previous Pulp versions. Fixing the issues with a migration or mongodb queries is generally straightforward, but knowing what the database problems are is a challenge.

To identify problems, a script should be written. The script makes the following assumptions:

  • This script is designed to run on a test system that has the 2.8 code or the in-development version of 2.8 code installed.
  • This 2.8 code will have all plugins types regardless of what the user's normal database had. These plugins include: pulp_docker, pulp_ostree, pulp_puppet, pulp_python, and pulp_rpm
  • This test system must have a mongodb already running with the user's pre-upgraded data loaded into the pulp_database inside of it.
  • The test environment's server.conf will contain the config which defines how to connect to this test db just like a normal pulp setup would do.
  • The script should be run as user apache, and the first thing it will do is run pulp-manage-db.
  • This script will connect to the database using the existing pulp connection mechanisms.

After pulp-manage-db is run, for each of the following models, the script will import each mongoengine based model using mongoengine and with no modifications will save() the model again. Any ValidationErrors will be raised with save(). It's important that any ValidationError exceptions are caught and recorded by writing to a separate text file. The actual record contents also should be written to the text file using mongoengine's to_mongo() call. The text file named 'validation_errors_<timestamp>.txt' should have <timestamp> be the current datetime so that multiple runs will not collide. If any validation problems occur, the script should instruct users to look at the contents of the validation file just created and to create a representative sample of validation issues at pulp.plan.io.

Note: once a single model has reached 50 ValidationErrors, continue on to the next model. It's not productive to log a huge number of errors that are all the same.

Also, a release note should be added to the 2.8.0 release notes indicating that any users is strongly recommended to run the pre-upgrade validation script prior to upgrading to 2.8.0.


Related issues

Related to RPM Support - Issue #1430: Package* unit types fails validation after upgradeCLOSED - CURRENTRELEASEbmbouterActions
Blocks Pulp - Task #1416: Write a docker script to containerize pulp 2.8 and the validation script (task 1415)CLOSED - CURRENTRELEASEmhrivnak

Actions
Actions #1

Updated by bmbouter over 8 years ago

  • Description updated (diff)
Actions #2

Updated by bmbouter over 8 years ago

  • Description updated (diff)
Actions #3

Updated by bmbouter over 8 years ago

  • Sprint Candidate changed from No to Yes
Actions #4

Updated by mhrivnak over 8 years ago

I suggest having a reasonable limit to the number of errors that will be logged per model. Once 100 validation errors happen on a given model, it's not productive to keep trying, or to collect more data. We definitely don't want to have 250,000 validation errors and model records logged to this text file. I don't feel strongly about what the limit should be, but my gut says over 100 would be excessive.

Actions #5

Updated by bmbouter over 8 years ago

  • Blocks Task #1416: Write a docker script to containerize pulp 2.8 and the validation script (task 1415) added
Actions #6

Updated by bmbouter over 8 years ago

  • Description updated (diff)

mhrivnak wrote:

I suggest having a reasonable limit to the number of errors that will be logged per model. Once 100 validation errors happen on a given model, it's not productive to keep trying, or to collect more data. We definitely don't want to have 250,000 validation errors and model records logged to this text file. I don't feel strongly about what the limit should be, but my gut says over 100 would be excessive.

+1 to that, I've added it to the story. I'm thinking even 50 would be plenty. I've revised the story with 50.

Actions #7

Updated by bmbouter over 8 years ago

  • Description updated (diff)
Actions #8

Updated by bmbouter over 8 years ago

  • Subject changed from Write a pre 2.8.0 upgrade script to Write a pre 2.8 upgrade script
  • Description updated (diff)
Actions #9

Updated by mhrivnak over 8 years ago

  • Groomed changed from No to Yes
Actions #10

Updated by bmbouter over 8 years ago

  • Related to Issue #1430: Package* unit types fails validation after upgrade added
Actions #11

Updated by amacdona@redhat.com about 8 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to pcreech

Added by pcreech about 8 years ago

Revision 193e2af5 | View on GitHub

Validation Check Script

Actions #12

Updated by pcreech about 8 years ago

  • Status changed from ASSIGNED to MODIFIED

Added by pcreech about 8 years ago

Revision d52bde60 | View on GitHub

Update validation_check to use entry points

Updates the validation check script to take advantage of the entry points that plugins are supposed to set up.

This way we don't have to modify the check script for every possible plugin.

https://pulp.plan.io/issues/1415 re #1415

Added by pcreech about 8 years ago

Revision d52bde60 | View on GitHub

Update validation_check to use entry points

Updates the validation check script to take advantage of the entry points that plugins are supposed to set up.

This way we don't have to modify the check script for every possible plugin.

https://pulp.plan.io/issues/1415 re #1415

Actions #13

Updated by rbarlow about 8 years ago

  • Status changed from MODIFIED to 5
  • Platform Release set to 2.8.0
Actions #14

Updated by dkliban@redhat.com almost 8 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #15

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF