Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #1092

closed

pulp-manage-db throws exception during upgrade from Pulp 2.6.3 to 2.7.0

Added by dkliban@redhat.com over 8 years ago. Updated over 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
amacdona@redhat.com
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.7.0 Beta
Platform Release:
2.7.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

1. Install Pulp 2.6.3 from [0]
2. Run pulp-manage-db.
3. Start all services.
4. Stop all services.
5. Install Pulp 2.7.0 from [1]
6. Run pulp-manage-db

Result:

# sudo -u apache pulp-manage-db
Mongo database for connection is version 2.6.9
Loading content types.
Loading type descriptors [python.json, iso_support.json, puppet.json, nodes.json, docker.json, rpm_support.json]
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Updating the database with types [python_package, iso, puppet_module, repository, node, docker_image, distribution, drpm, erratum, package_group, package_category, package_environment, rpm, srpm, yum_repo_metadata_file]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 18
Migration package pulp_puppet.plugins.migrations is up to date at version 2
Migration package pulp_rpm.plugins.migrations is up to date at version 21
Index with name: task_id_-1 already exists with different options
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 159, in main
    return _auto_manage_db(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 220, in _auto_manage_db
    migrate_database(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 104, in migrate_database
    ensure_database_indexes()
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 122, in ensure_database_indexes
    _ensure_indexes(model.TaskStatus)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 140, in _ensure_indexes
    model_class.ensure_indexes()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 549, in ensure_indexes
    collection = cls._get_collection()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 177, in _get_collection
    cls.ensure_indexes()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 567, in ensure_indexes
    drop_dups=drop_dups, **opts)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 916, in ensure_index
    return self.create_index(key_or_list, cache_for, **kwargs)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 823, in create_index
    **self._get_wc_override()) 
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 357, in insert
    continue_on_error, self.__uuid_subtype), safe)
  File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 920, in _send_message
    rv = self.__check_response_to_last_error(response)
  File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 863, in __check_response_to_last_error
    raise OperationFailure(details["err"], details["code"])
OperationFailure: Index with name: task_id_-1 already exists with different options

Expected result:

All migrations finish without error.

[0] https://repos.fedorapeople.org/repos/pulp/pulp/testing/automation/2.6/testing/fedora-$releasever/$basearch/
[1] https://repos.fedorapeople.org/repos/pulp/pulp/testing/automation/2.7/testing/fedora-$releasever/$basearch/

Actions
Copy link
 #1

Updated by ipanova@redhat.com over 8 years ago

Irina had exactly same traceback, because of some package missing, I cannot recall what exactly was missing, I think pinging Irina would be helpful.

Actions
Copy link
 #2

Updated by amacdona@redhat.com over 8 years ago

This is caused by a doubled index for task_id in the task_status collection.

The doubled index is a result of the way that the task_id field is defined in the mongoengine model:

    task_id = StringField(unique=True, required=True)

    ...

    meta = {'collection': 'task_status',
            'indexes': ['-task_id', '-tags', '-state'],
            'allow_inheritance': False,
            'queryset_class': CriteriaQuerySet}

By setting `unique=True` in the field definition, it automatically generates an index for the field of the order 1. It is set again later in the meta, but does not enforce uniqueness there.

Actions
Copy link
 #3

Updated by amacdona@redhat.com over 8 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to amacdona@redhat.com
  • Platform Release set to 2.7.0
Actions
Copy link
 #4

Updated by bcourt over 8 years ago

  • Triaged changed from No to Yes
Actions
Copy link
 #5

Updated by amacdona@redhat.com over 8 years ago

  • Status changed from ASSIGNED to POST

https://github.com/pulp/pulp/pull/1957

I was unable to replicate the failure of the migration, I believe is the underlying cause is the duplicate index, which this PR fixes.

Actions
Copy link
 #6

Updated by dkliban@redhat.com over 8 years ago

  • Status changed from POST to MODIFIED
Actions
Copy link
 #7

Updated by dkliban@redhat.com about 8 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #8

Updated by igulina@redhat.com about 8 years ago

  • Status changed from 5 to ASSIGNED

I updated 2.7-0.3 to 2.7-0.4

>> sudo -u apache pulp-manage-db

Mongo database for connection is version 2.6.9
Loading content types.
Loading type descriptors [iso_support.json, rpm_support.json, puppet.json, docker.json]
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Updating the database with types [iso, distribution, drpm, erratum, package_group, package_category, package_environment, rpm, srpm, yum_repo_metadata_file, puppet_module, docker_image]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 18
Migration package pulp_puppet.plugins.migrations is up to date at version 2
Migration package pulp_rpm.plugins.migrations is up to date at version 21
command SON([('createIndexes', u'task_status'), ('indexes', [{'name': u'task_id_-1', 'key': SON([('task_id', -1)]), 'unique': True, 'background': False, 'dropDups': False}])]) failed: Index with name: task_id_-1 already exists with different options
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 140, in main
    return _auto_manage_db(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 201, in _auto_manage_db
    migrate_database(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 104, in migrate_database
    ensure_database_indexes()
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 120, in ensure_database_indexes
    model.TaskStatus.ensure_indexes()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 654, in ensure_indexes
    collection = cls._get_collection()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 198, in _get_collection
    cls.ensure_indexes()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 676, in ensure_indexes
    drop_dups=drop_dups, **opts)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 1167, in ensure_index
    return self.create_index(key_or_list, cache_for, **kwargs)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 1060, in create_index
    indexes=[index])
  File "/usr/lib64/python2.7/site-packages/pymongo/database.py", line 445, in command
    uuid_subtype, compile_re, **kwargs)[0]
  File "/usr/lib64/python2.7/site-packages/pymongo/database.py", line 351, in _command
    msg, allowable_errors)
  File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 178, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
OperationFailure: command SON([('createIndexes', u'task_status'), ('indexes', [{'name': u'task_id_-1', 'key': SON([('task_id', -1)]), 'unique': True, 'background': False, 'dropDups': False}])]) failed: Index with name: task_id_-1 already exists with different options
Actions
Copy link
 #9

Updated by igulina@redhat.com about 8 years ago

There is no such an issue if to run pulp-manage-db on fresh 2.7-0.4 install.

Please notice that this BZ blocks a big part of pulp functionality, - the error

An internal error occurred on the Pulp server:

RequestException: ............ failed with 500 - command
SON([('createIndexes', u'task_status'), ('indexes', [{'name': u'task_id_-1',
'key': SON([('task_id', -1)]), 'unique': True, 'background': False, 'dropDups':
False}])]) failed: Index with name: task_id_-1 already exists with different
options

appers every time when a task is created. If a user updates pulp, s/he will be in troubles. I would set it a high priority.

Actions
Copy link
 #10

Updated by amacdona@redhat.com about 8 years ago

I spun up a vm running pulp 2.6 and then upgraded it to 2.7. The migration proceeded without issue.

It looks the failing upgrade was from one 2.7-beta to the other, which is not supported.

Actions
Copy link
 #11

Updated by igulina@redhat.com about 8 years ago

  • Status changed from ASSIGNED to 6
>> rpm -qa pulp-server
pulp-server-2.6.2-0.2.beta.el7.noarch

>> sudo -u apache pulp-manage-db
Mongo database for connection is version 2.6.9
Loading content types.
Loading type descriptors [docker.json]
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Updating the database with types [docker_image]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 12
Database migrations complete.

>> for i in httpd pulp_workers pulp_resource_manager pulp_celerybeat ; do systemctl stop $i; done

>>  for i in httpd pulp_workers pulp_resource_manager pulp_celerybeat ; do systemctl status $i; done
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled)
   Active: inactive (dead) since Tue 2015-07-07 08:12:50 EDT; 6s ago
  Process: 17844 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
  Process: 13971 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
  Process: 17675 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=0/SUCCESS)
 Main PID: 17675 (code=exited, status=0/SUCCESS)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service

Jul 07 08:11:25 XXX systemd[1]: Starti...
Jul 07 08:11:26 XXX systemd[1]: Starte...
Jul 07 08:11:28 XXX pulp[17679]: pulp....
Jul 07 08:11:28 XXX pulp[17677]: pulp....
......
Jul 07 08:12:49 XXX systemd[1]: Stoppi...
Jul 07 08:12:50 XXX systemd[1]: Stoppe...
Hint: Some lines were ellipsized, use -l to show in full.
pulp_workers.service - Pulp Celery Workers
   Loaded: loaded (/usr/lib/systemd/system/pulp_workers.service; enabled)
   Active: inactive (dead) since Tue 2015-07-07 08:12:51 EDT; 6s ago
  Process: 17854 ExecStop=/usr/libexec/pulp-manage-workers stop (code=exited, status=0/SUCCESS)
  Process: 17709 ExecStart=/usr/libexec/pulp-manage-workers start (code=exited, status=0/SUCCESS)
 Main PID: 17709 (code=exited, status=0/SUCCESS)

Jul 07 08:11:27 XXX systemd[1]: Starti...
Jul 07 08:11:28 XXX systemd[1]: Starte...
Jul 07 08:12:50 XXX systemd[1]: Stoppi...
Jul 07 08:12:51 XXX systemd[1]: Stoppe...
Hint: Some lines were ellipsized, use -l to show in full.
pulp_resource_manager.service - Pulp Resource Manager
   Loaded: loaded (/usr/lib/systemd/system/pulp_resource_manager.service; enabled)
   Active: inactive (dead) since Tue 2015-07-07 08:12:52 EDT; 4s ago
  Process: 17798 ExecStart=/usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%%h -Q resource_manager -c 1 --events --umask 18 (code=exited, status=0/SUCCESS)
 Main PID: 17798 (code=exited, status=0/SUCCESS)

Jul 07 08:12:52 XXX pulp[17798]: kombu...
Jul 07 08:12:52 XXX pulp[17798]: kombu...
...
Jul 07 08:12:52 XXX celery[17798]: res...
Jul 07 08:12:52 XXX systemd[1]: Stoppe...
Hint: Some lines were ellipsized, use -l to show in full.
pulp_celerybeat.service - Pulp's Celerybeat
   Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service; enabled)
   Active: inactive (dead) since Tue 2015-07-07 08:12:52 EDT; 4s ago
  Process: 17803 ExecStart=/usr/bin/celery beat --scheduler=pulp.server.async.scheduler.Scheduler (code=exited, status=0/SUCCESS)
 Main PID: 17803 (code=exited, status=0/SUCCESS)

Jul 07 08:12:52 XXX systemd[1]: Stoppi...
Jul 07 08:12:52 XXX celery[17803]: cel...
....
Jul 07 08:12:52 XXX systemd[1]: Stoppe...
Hint: Some lines were ellipsized, use -l to show in full.

>> yum update
...
>> rpm -qa pulp-server
pulp-server-2.7.0-0.4.beta.el7.noarch

>> sudo -u apache pulp-manage-db
Mongo database for connection is version 2.6.9
Loading content types.
Loading type descriptors [docker.json]
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Updating the database with types [docker_image]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Applying pulp.server.db.migrations version 13
Migration to pulp.server.db.migrations version 13 complete.
Applying pulp.server.db.migrations version 14
Migration to pulp.server.db.migrations version 14 complete.
Applying pulp.server.db.migrations version 15
Loading type descriptors [docker.json]
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Updating the database with types [docker_image]
Migration to pulp.server.db.migrations version 15 complete.
Applying pulp.server.db.migrations version 16
Dropping the uniqueness index that included the owner_type & owner_id
Migration to pulp.server.db.migrations version 16 complete.
Applying pulp.server.db.migrations version 17
Migration to pulp.server.db.migrations version 17 complete.
Applying pulp.server.db.migrations version 18
Deleted the archived_calls collection.
Migration to pulp.server.db.migrations version 18 complete.
Database migrations complete.
Actions
Copy link
 #12

Updated by amacdona@redhat.com almost 8 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions
Copy link
 #14

Updated by bmbouter over 4 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF