Issue #1738: migration failure on upgrade from 2.7 -> 2.8 - Pulp

Actions

Send by e-mail Copy link

Issue #1738

closed

migration failure on upgrade from 2.7 -> 2.8

Added by pthomas@redhat.com over 7 years ago. Updated over 4 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

High

Assignee:

amacdona@redhat.com

Category:

Sprint/Milestone:

Start date:

Due date:

Estimated time:

Severity:

3. High

Version:

Platform Release:

2.8.0

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Pulp 2

Sprint:

Quarter:

Description

pulp-manage-db failed on upgrading 2.7 -> 2.8 on El6

[root@ibm-x3550m3-09 ~]# rpm -qa |grep pulp
python-pulp-bindings-2.8.0-0.6.beta.el6.noarch
pulp-admin-client-2.8.0-0.6.beta.el6.noarch
pulp-rpm-plugins-2.8.0-0.5.beta.el6.noarch
pulp-rpm-handlers-2.8.0-0.5.beta.el6.noarch
python-pulp-repoauth-2.8.0-0.6.beta.el6.noarch
pulp-server-2.8.0-0.6.beta.el6.noarch
pulp-rpm-admin-extensions-2.8.0-0.5.beta.el6.noarch
python-pulp-agent-lib-2.8.0-0.6.beta.el6.noarch
python-pulp-docker-common-2.0.0-0.6.beta.el6.noarch
pulp-docker-plugins-2.0.0-0.6.beta.el6.noarch
pulp-agent-2.8.0-0.6.beta.el6.noarch
pulp-docker-admin-extensions-2.0.0-0.6.beta.el6.noarch
python-isodate-0.5.0-4.pulp.el6.noarch
mod_wsgi-3.4-2.pulp.el6.x86_64
python-pulp-common-2.8.0-0.6.beta.el6.noarch
python-pulp-rpm-common-2.8.0-0.5.beta.el6.noarch
python-pulp-client-lib-2.8.0-0.6.beta.el6.noarch
python-pulp-oid_validation-2.8.0-0.6.beta.el6.noarch
pulp-puppet-plugins-2.8.0-0.6.beta.el6.noarch
pulp-puppet-consumer-extensions-2.8.0-0.6.beta.el6.noarch
pulp-puppet-admin-extensions-2.8.0-0.6.beta.el6.noarch
pulp-rpm-yumplugins-2.8.0-0.5.beta.el6.noarch
python-pulp-puppet-common-2.8.0-0.6.beta.el6.noarch
pulp-rpm-consumer-extensions-2.8.0-0.5.beta.el6.noarch
pulp-selinux-2.8.0-0.6.beta.el6.noarch
pulp-consumer-client-2.8.0-0.6.beta.el6.noarch
python-kombu-3.0.33-4.pulp.el6.noarch
pulp-puppet-handlers-2.8.0-0.6.beta.el6.noarch

Steps
1. Start with a 2.7 pulp
2. Make sure you have a lot of data
3. Run yum update to upgrade to 2.8
4. Run sudo -u apache pulp-manage-db

[root@ibm-x3550m3-09 ~]# sudo -u apache pulp-manage-db
Attempting to connect to localhost:27017
Attempting to connect to localhost:27017
Write concern for Mongo connection: {}
/usr/lib/python2.6/site-packages/pulp/server/db/connection.py:159: DeprecationWarning: add_son_manipulator is deprecated
  _DATABASE.add_son_manipulator(NamespaceInjector())
Loading content types.
Loading type descriptors []
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Updating the database with types [puppet_module, drpm, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, docker_tag, distribution, rpm, srpm, docker_image]
/usr/lib/python2.6/site-packages/pulp/server/db/model/base.py:96: DeprecationWarning: ensure_index is deprecated. Use create_index instead.
  unique=unique, background=True)
/usr/lib/python2.6/site-packages/pulp/plugins/types/database.py:273: DeprecationWarning: save is deprecated. Use insert_one or replace_one instead
  content_type_collection.save(content_type)
/usr/lib/python2.6/site-packages/pulp/plugins/types/database.py:302: DeprecationWarning: ensure_index is deprecated. Use create_index instead.
  index_name = collection.ensure_index(mongo_index, unique=unique)
Exception updating unit key for type [erratum]
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/plugins/types/database.py", line 105, in update_database
    _update_unit_key(type_def)
  File "/usr/lib/python2.6/site-packages/pulp/plugins/types/database.py", line 311, in _update_unit_key
    _update_indexes(type_def, True)
  File "/usr/lib/python2.6/site-packages/pulp/plugins/types/database.py", line 302, in _update_indexes
    index_name = collection.ensure_index(mongo_index, unique=unique)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 1407, in ensure_index
    self.__create_index(keys, kwargs)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 1297, in __create_index
    sock_info, index, True, False, False, wcn)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 530, in _insert
    check_keys, manipulate, write_concern, op_id, bypass_doc_val)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 519, in _insert_one
    check_keys, acknowledged, concern, False, self.codec_options)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 458, in _legacy_write
    rqst_id, msg, max_size, acknowledged)
  File "/usr/lib64/python2.6/site-packages/pymongo/pool.py", line 264, in legacy_write
    return helpers._check_gle_response(response)
  File "/usr/lib64/python2.6/site-packages/pymongo/helpers.py", line 235, in _check_gle_response
    raise DuplicateKeyError(details["err"], code, result)
DuplicateKeyError: E11000 duplicate key error index: pulp_database.units_erratum.$errata_id_1  dup key: { : null }
Exception updating unit key for type [package_group]
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/plugins/types/database.py", line 105, in update_database
    _update_unit_key(type_def)
  File "/usr/lib/python2.6/site-packages/pulp/plugins/types/database.py", line 311, in _update_unit_key
    _update_indexes(type_def, True)
  File "/usr/lib/python2.6/site-packages/pulp/plugins/types/database.py", line 302, in _update_indexes
    index_name = collection.ensure_index(mongo_index, unique=unique)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 1407, in ensure_index
    self.__create_index(keys, kwargs)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 1297, in __create_index
    sock_info, index, True, False, False, wcn)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 530, in _insert
    check_keys, manipulate, write_concern, op_id, bypass_doc_val)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 519, in _insert_one
    check_keys, acknowledged, concern, False, self.codec_options)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 458, in _legacy_write
    rqst_id, msg, max_size, acknowledged)
  File "/usr/lib64/python2.6/site-packages/pymongo/pool.py", line 264, in legacy_write
    return helpers._check_gle_response(response)
  File "/usr/lib64/python2.6/site-packages/pymongo/helpers.py", line 235, in _check_gle_response
    raise DuplicateKeyError(details["err"], code, result)
DuplicateKeyError: E11000 duplicate key error index: pulp_database.units_package_group.$package_group_id_1_repo_id_1  dup key: { : null, : "centos5" }
Exception updating unit key for type [package_category]
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/plugins/types/database.py", line 105, in update_database
    _update_unit_key(type_def)
  File "/usr/lib/python2.6/site-packages/pulp/plugins/types/database.py", line 311, in _update_unit_key
    _update_indexes(type_def, True)
  File "/usr/lib/python2.6/site-packages/pulp/plugins/types/database.py", line 302, in _update_indexes
    index_name = collection.ensure_index(mongo_index, unique=unique)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 1407, in ensure_index
    self.__create_index(keys, kwargs)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 1297, in __create_index
    sock_info, index, True, False, False, wcn)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 530, in _insert
    check_keys, manipulate, write_concern, op_id, bypass_doc_val)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 519, in _insert_one
    check_keys, acknowledged, concern, False, self.codec_options)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 458, in _legacy_write
    rqst_id, msg, max_size, acknowledged)
  File "/usr/lib64/python2.6/site-packages/pymongo/pool.py", line 264, in legacy_write
    return helpers._check_gle_response(response)
  File "/usr/lib64/python2.6/site-packages/pymongo/helpers.py", line 235, in _check_gle_response
    raise DuplicateKeyError(details["err"], code, result)
DuplicateKeyError: E11000 duplicate key error index: pulp_database.units_package_category.$package_category_id_1_repo_id_1  dup key: { : null, : "centos5" }
UpdateFailed [erratum, package_group, package_category]
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/db/manage.py", line 193, in main
    return _auto_manage_db(options)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/manage.py", line 220, in _auto_manage_db
    old_content_types = load_content_types(dry_run=options.dry_run)
  File "/usr/lib/python2.6/site-packages/pulp/plugins/loader/api.py", line 443, in load_content_types
    database.update_database(definitions, drop_indices=drop_indices)
  File "/usr/lib/python2.6/site-packages/pulp/plugins/types/database.py", line 119, in update_database
    raise UpdateFailed(error_defs)
UpdateFailed: UpdateFailed [erratum, package_group, package_category]

Actions

Copy link

Updated by pthomas@redhat.com over 7 years ago

Description updated (diff)

Actions

Copy link

Updated by mhrivnak over 7 years ago

It looks like pulp_rpm did not get re-built as beta 6, and as a result you have a mix of beta 5 and beta 6 packages installed. dkliban is working on that now. It may or may not explain this problem, but we'll need to re-test with beta 6.

Actions

Copy link

Updated by mhrivnak over 7 years ago

Status changed from NEW to ASSIGNED
Assignee set to amacdona@redhat.com
Platform Release set to 2.8.0
Triaged changed from No to Yes

I was able to reproduce this with a 2.7.1 database dump.

Actions

Copy link

Updated by mhrivnak over 7 years ago

We have a chicken-and-egg problem with indices.

Right now, pulp-manage-db ensures that content types are all loaded before running migrations, so that migrations can depend on the content types being present. I don't recall the details of why, but we had an important reason for doing so in the past.

We had to rename a field in some content type unit keys for pulp 2.8. When making that change, the order of these operations is critical:

1) remove the previous unit key index
2) rename the field in the unit key
3) create the new unit key index

But currently, step 3 is happening first, when content types are loaded.

When the content type loading workflow happens in pulp-manage-db, it ensures that the unique index for the unit key is in place. When this happens, for any key in the unit key that was renamed, all of its values appear to be null, and this can lead to the uniqueness errors seen above.
With pulp-manage-db, we somehow need an opportunity to do steps 1 and 2, which are currently in a pulp_rpm migration, before step 3.

Ideas?

Actions

Copy link

Updated by bmbouter over 7 years ago

I'm not sure why step (3) is happening before steps 1 and 2 which run in a migration. The uniqueness indexes and other necessary indexes are loaded after migrations are run[0] right?

[0]:https://github.com/pulp/pulp/blob/53b871f0ce6257fb506812114a54b60e856285e2/server/pulp/server/db/manage.py#L142

Actions

Copy link

Updated by mhrivnak over 7 years ago

You are correct, but it is also happening at the beginning. This call:

https://github.com/pulp/pulp/blob/53b871f0ce6257fb506812114a54b60e856285e2/server/pulp/server/db/manage.py#L220

... starts a workflow that ends up here:

https://github.com/pulp/pulp/blob/53b871f0ce6257fb506812114a54b60e856285e2/server/pulp/plugins/types/database.py#L302

... all of which happens before any migrations are run.

Added by Austin Macdonald over 7 years ago

Revision 36cb83a3 | View on GitHub

Create mongoengine indexes only after migrations

Fixes a bug that broke upgrades due to the fact that indexes were being generated for newly renamed fields before the migrations were run populating the indexes with 'null' and breaking upgrades if there are uniqueness contstraints.

closes #1738