Project

Profile

Help

Issue #4561

closed

E11000 duplicate key error during pulp-manage-db command

Added by siberijah over 3 years ago. Updated about 3 years ago.

Status:
CLOSED - NOTABUG
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
CentOS 7
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Hello.
I have a server that needs to be moved to another place. I installed pulp, moved databases from old server to new.
As the next step I tried to login to pulp-admin and get repositories.

[root@pulp-master ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

There was an internal server error while trying to access the Pulp application.
One possible cause is that the database needs to be migrated to the latest
version. If this is the case, run pulp-manage-db and restart the services. More
information may be found in Apache's log.

As far as I see - first I need to execute pulp-manage-db command. So I did so. And got error E11000 duplicate key error

[root@pulp-master ~]# sudo -u apache pulp-manage-db
Attempting to connect to localhost:27017
Attempting to connect to localhost:27017
Write concern for Mongo connection: {}
Loading content types.
Loading type descriptors []
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Loading unit model: modulemd_defaults = pulp_rpm.plugins.db.models:ModulemdDefaults
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: package_langpacks = pulp_rpm.plugins.db.models:PackageLangpacks
Loading unit model: modulemd = pulp_rpm.plugins.db.models:Modulemd
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest_list = pulp_docker.plugins.models:ManifestList
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading auxiliary model: erratum_pkglists = pulp_rpm.plugins.db.models:ErratumPkglist
Updating the database with types []
Found the following type definitions that were not present in the update collection [puppet_module, docker_tag, modulemd_defaults, package_langpacks, docker_manifest, docker_blob, erratum, srpm, package_group, package_category, iso, package_environment, drpm, distribution, modulemd, rpm, yum_repo_metadata_file, docker_image, docker_manifest_list]
Updating the database with types [puppet_module, docker_tag, modulemd_defaults, package_langpacks, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, drpm, distribution, modulemd, rpm, srpm, docker_image, docker_manifest_list]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
E11000 duplicate key error index: pulp_database.repos.$repo_id_-1  dup key: { : null }
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 239, in main
    return _auto_manage_db(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 306, in _auto_manage_db
    migrate_database(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 74, in migrate_database
    ensure_database_indexes()
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 166, in ensure_database_indexes
    model.Repository.ensure_indexes()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 738, in ensure_indexes
    collection = cls._get_collection()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 210, in _get_collection
    cls.ensure_indexes()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 766, in ensure_indexes
    collection.create_index(fields, background=background, **opts)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 1380, in create_index
    self.__create_index(keys, kwargs)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 1290, in __create_index
    sock_info, cmd, read_preference=ReadPreference.PRIMARY)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 205, in _command
    read_concern=read_concern)
  File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 211, in command
    read_concern)
  File "/usr/lib64/python2.7/site-packages/pymongo/network.py", line 100, in command
    helpers._check_command_response(response_doc, msg, allowable_errors)
  File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 189, in _check_command_response
    raise DuplicateKeyError(errmsg, code, response)
DuplicateKeyError: E11000 duplicate key error index: pulp_database.repos.$repo_id_-1  dup key: { : null }
Actions #1

Updated by siberijah over 3 years ago

UPD:
I executed:

> db.repos.remove({"repo_id": null})
WriteResult({ "nRemoved" : 73 })

and also I cleaned up all duplicates and all migrations has been done.

Some examples:

mongo pulp_database --eval 'db.units_package_category.remove({repo_id: "virtualbox_rhel-6.5_x86_64"})'
and
> db.repo_distributors.remove({"zenoss_deps": null})
and this
> db.units_package_group.remove({"repo_id": "centos_6-5_base"})
etc...
Actions #2

Updated by siberijah over 3 years ago

Update:

All commands from my previous message just cleaning up all data inside collection.

For example if I execute db.repos.remove({"repo_id-1": null}) it will empty all repos collection

So guys, how to move PULP to another server?

Some more info: I was trying to migrate pup 2.4 to pulp 2.18 and mongo 2.4 to 2.6.
I also tried to migrate mongo 2.4 to 3.4 but failed, and mongo 2.4 to 2.4.... the same issue - dup keys.

Actions #3

Updated by daviddavis about 3 years ago

  • Project changed from 31 to Pulp
Actions #4

Updated by daviddavis about 3 years ago

  • Tags Pulp 2 added
Actions #5

Updated by dkliban@redhat.com about 3 years ago

  • Status changed from NEW to CLOSED - NOTABUG

Please ask a question about this on the pulp user mailing list[0] or on IRC[1]. Other users will probably be able to help you more.

[0] https://www.redhat.com/mailman/listinfo/pulp-list
[1] https://pulpproject.org/help/#irc

Also available in: Atom PDF