Project

Profile

Help

Issue #4432

Pulp upgrade from 2.7 to 2.18 logs says Database initialization failed.

Added by ymadav over 1 year ago. Updated over 1 year ago.

Status:
CLOSED - NOTABUG
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
RHEL 7
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Hi All,

I have reproduced my working environment 2.7 in my non-prod environment and created the repos and working fine without any issues.I have upgraded the pulp package from 2.7 to 2.18 version and have mongodb running with 2.6.We have dedicated servers for each components.

When i run the pulp-manage-db it throws me error saying "DuplicateKeyError: E11000 duplicate key error index: pulp_database.repos.$repo_id_-1 dup key: { : null }" and also during the restart of httpd service i see that pulp_database initialization failed.I am attaching the /var/log/messages and output of pulp-manage-db output.Can some one let me know if you have faced certain type of issue.

When i run sudo -u apache pulp-manage-db --dry-run it runs fine but without dry-run it encounters with duplicate error and fails.

error_log.docx (14.2 KB) error_log.docx ymadav, 02/19/2019 11:21 AM
Pulp_error_log.rtf (20.9 KB) Pulp_error_log.rtf ymadav, 02/19/2019 03:33 PM
managedb_2.8_firstrun (9.48 KB) managedb_2.8_firstrun upgrade file from 2.7 to 2.8 ymadav, 02/25/2019 03:30 PM
managedb_2.9_firstrun (4.14 KB) managedb_2.9_firstrun upgrade file from 2.8 to 2.9 ymadav, 02/25/2019 03:30 PM
250
250

History

#1 Updated by daviddavis over 1 year ago

First off, can you upload the error log in text? I can't read docx.

Secondly, it sounds like you have a repo (or repos?) with repo_id of null. Here's how you can clean that up:

mongo pulp_database
> db.repos.deleteMany({"repo_id": null})

Not sure what could have caused this since the pulp code doesn't allow repo_id to be null. Maybe an old bug?

#2 Updated by ymadav over 1 year ago

250

Thanks David for the reply,I see these repos id in production db as well,i have uploaded the output of those in text log file at the end.

One question if we delete null repo it will not affect the existing repos,as i have created before upgrade epel,centos repos.I don't want to lose them as this upgrade test will be proof to proceed for production.

Please let me know if i am missing anything here.

Thanks again,waiting to hear from you.

#4 Updated by daviddavis over 1 year ago

I see the problem. It looks like in 2.8, we renamed the id field to repo_id:

https://github.com/pulp/pulp/blob/2-master/server/pulp/server/db/migrations/0019_repo_collection_id.py

However, manage.py calls ensure_database_indexes:

https://github.com/pulp/pulp/blob/2-master/server/pulp/server/db/manage.py#L157

This tries to create an index on repo_id before the field is renamed. Can you try upgrading to 2.8 first and then to 2.18?

#5 Updated by ymadav over 1 year ago

Sure will give a try on this to update first to 2.8 and then to 2.18,one doubt here as we upgraded from 2.7 that will id field to repo_id will not be changed?.As i see in the servers has same info as mentioned in the links that were provided,so can you please let me know on that.

manage.py has ensure_database_indexes: info.

Thanks for the reply.

#6 Updated by daviddavis over 1 year ago

I'm not sure I fully understand your question. When you upgrade from 2.7 to 2.8, id will be renamed to repo_id for repositories.

#7 Updated by ymadav over 1 year ago

Ignore my earlier question David,I have upgraded the pulp version from 2.7 to 2.8 and see few changes in the database that were reflected, and also i did next level upgrade from 2.8 to 2.9 in that have seen few more.I am attaching those logs for reference.Attached files have info during initial migration after the update of pulp from 2.7 to 2.8 and 2.8 to 2.9. Application is working fine after upgrade in my dev environment, one question here my test DB is having 10GB when i execute pulp-manage-db command after upgrade it is taking like 5-6 mins to complete for the first time, is that behaviour is expected ?.I am asking this because we have our production db of 80GB thinking how much time it would take if it is normal behaviour

Is it ok now we can upgrade the pulp version from 2.9 to 2.18 instead of step by step.Please let me know.

Thanks for your time on this.

#8 Updated by daviddavis over 1 year ago

I think you should be able to upgrade from 2.8 to 2.18 in one step.

I am not sure about how long it will take to upgrade. These pulp versions predate my time with Pulp. Maybe someone else on the Pulp team can weigh in on which version upgrades might take a long time?

Alternatively, can you clone your production environment and then attempt to upgrade the clone? Then you could get an pretty accurate idea of how long it will take and what issues you might encounter.

#9 Updated by ymadav over 1 year ago

I started updating the dev environment version by version,now i am in Pulp 2.12 and application works fine without issues,so going version version tells me which version is blocking me here.

I can take a clone of production and do the testing but here problem is it has large chunk from NAS share,and puppet code need to be changed as well there.Once i update my dev environment to version 2.18 and mongodb to 4.0 version

i will see if the clone option works fine in my case as we have cluster of servers it is not a standalone environment.But that is good way of testing on production.

From my observation while doing upgrade from 2.11 to 2.12 there are few migrations of units that occurred in rpm plugin migrations and to complete pulp-manage-db it took 5 mins in my case.

Can you please let me know what exactly it is migrating here.

Migration to pulp_rpm.plugins.migrations version 38 complete.
Applying pulp_rpm.plugins.migrations version 39
*********************************************************************************

  • NOTE: This migration may take some time depending on the size of your Pulp content. *
    *********************************************************************************
  • Migrating RPM content...
  • Migrated units: 5637 of 56377
  • Migrated units: 11274 of 56377
  • Migrated units: 16911 of 56377
  • Migrated units: 22548 of 56377
  • Migrated units: 28185 of 56377
  • Migrated units: 33822 of 56377
  • Migrated units: 39459 of 56377
  • Migrated units: 45096 of 56377
  • Migrated units: 50733 of 56377
  • Migrated units: 56370 of 56377
  • Migrated units: 56377 of 56377
  • Migrating SRPM content...
  • Migrated units: 2 of 27
  • Migrated units: 4 of 27
  • Migrated units: 6 of 27
  • Migrated units: 8 of 27
  • Migrated units: 10 of 27
  • Migrated units: 12 of 27
  • Migrated units: 14 of 27
  • Migrated units: 16 of 27
  • Migrated units: 18 of 27
  • Migrated units: 20 of 27
  • Migrated units: 22 of 27
  • Migrated units: 24 of 27
  • Migrated units: 26 of 27
  • Migrated units: 27 of 27
    *********************************************************************************

As always thanks for your time and help on this.

#10 Updated by ttereshc over 1 year ago

FWIW, I don't know if MongoDB 4.0 is working correctly with Pulp, but definitely MongoDB from 2.4 and up to 3.6 is supported.

As for your migrations question, 0039 migration solves the problem for [S]RPMs with huge metadata by compressing it before storing in DB.
https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/migrations/0039_gzip_repodata.py

#11 Updated by CodeHeeler over 1 year ago

  • Status changed from NEW to CLOSED - NOTABUG

Let's continue the support and discussion on

#12 Updated by ymadav over 1 year ago

Thanks for the info ttereshc, we are running mongodb 2.6 so we are good with pulp 2.18 version coming to the supported version.Please confirm.

#13 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF