Issue #2591


pulp not starting properly, apache pulp-manage-db throwing errors

Added by chuckuf about 7 years ago. Updated about 4 years ago.

Start date:
Due date:
Estimated time:
3. High
Platform Release:
CentOS 7
Sprint Candidate:
Pulp 2


I updated to pulp 2.12 and pulp is not starting properly.

[root@pulpcos ~]# pulp-admin status
                          Status of the server

There was an internal server error while trying to access the Pulp application.
One possible cause is that the database needs to be migrated to the latest
version. If this is the case, run pulp-manage-db and restart the services. More
information may be found in Apache's log.

[root@pulpcos ~]# for i in pulp_resource_manager pulp_workers pulp_celerybeat; do service $i stop; done
Redirecting to /bin/systemctl stop  pulp_resource_manager.service
Redirecting to /bin/systemctl stop  pulp_workers.service
Redirecting to /bin/systemctl stop  pulp_celerybeat.service

[root@pulpcos ~]# sudo -u apache pulp-manage-db
Attempting to connect to localhost:27017
Attempting to connect to localhost:27017
Write concern for Mongo connection: {}
Loading content types.
Loading type descriptors []
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: package_langpacks = pulp_rpm.plugins.db.models:PackageLangpacks
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Updating the database with types []
Found the following type definitions that were not present in the update collection [puppet_module, docker_tag, docker_manifest, docker_blob, erratum, distribution, yum_repo_metadata_file, package_group, package_category, iso, package_environment, drpm, package_langpacks, rpm, srpm, docker_image]
Updating the database with types [puppet_module, drpm, package_langpacks, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, docker_tag, distribution, rpm, srpm, docker_image]
Content types loaded.
Ensuring the admin role and user are in place.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/db/", line 227, in main
    return _auto_manage_db(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/", line 288, in _auto_manage_db
  File "/usr/lib/python2.7/site-packages/pulp/server/db/migrations/lib/", line 99, in ensure_admin
    if self.get_admins():
  File "/usr/lib/python2.7/site-packages/pulp/server/db/migrations/lib/", line 122, in get_admins
    super_users = UserManager.find_users_belonging_to_role(SUPER_USER_ROLE)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/migrations/lib/", line 144, in find_users_belonging_to_role
    if role_id in user['roles']:
KeyError: 'roles'

[root@pulpcos ~]# rpm -qi pulp-server
Name        : pulp-server
Version     : 2.12.0
Release     : 1.el7

[root@pulpcos ~]# rpm -qi python
Name        : python
Version     : 2.7.5
Release     : 48.el7
Actions #1

Updated by bizhang about 7 years ago

chuckuf, what version of Pulp are you upgrading from?

Actions #2

Updated by bizhang about 7 years ago

Removing blocker status for now. QE does not see this issue on their upgrade tests. We can readd this field to block the 2.12.1 GA release when more information about how to reproduce this issue is obtained.

Actions #3

Updated by bmbouter about 7 years ago

  • Description updated (diff)
Actions #4

Updated by chuckuf about 7 years ago

Updated pulp-server-2.10.0-1.el7.noarch @pulp-2-stable
Update 2.12.0-1.el7.noarch @pulp-2-stable

Actions #5

Updated by about 7 years ago

I did an upgrade from 2.11 -> 2.12 -> 2.12.1 beta. I made sure I had some roles & users in pulp as well. Both upgrade from 2.11 -> 2.12 and 2.12 -> 2,12 beta worked with no error.

Actions #6

Updated by chuckuf about 7 years ago

I would like to try to rollback to upgrade to see if I can get it working again, but I cannot find pulp-server-2.10.0-1 repos anywhere. All I can find is pulp-server-2.10.3. Anyone know where I can find pulp-server-2.10.0-1 repos? I downgraded to pulp-server-2.10.3, but the same error persists.

Actions #7

Updated by bmbouter about 7 years ago

Unfortunately, we only retain the latest Z-stream. From the traceback you experience I don't think having the 2.10.0-1 files will help you since I see there were no code changes in the /usr/lib/python2.7/site-packages/pulp/server/db/migrations/lib/ file. Your DB in the users table is corrupted somehow. Here is a record from my dev environment which is newer, but it should be the same on 2.10+.

> db.users.find().pretty()
        "_id" : ObjectId("58a617dee779890854d270b5"),
        "login" : "admin",
        "name" : "admin",
        "password" : "0bwgQLkAUDU=,jFqcfpnp+r0qR/fHWc63fL3G4ic8YcomS66kedmuHek=",
        "roles" : [
        "_ns" : "users"

If you compare the documents in your 'user' collection to ^, you should be able to see some differences which you could manually fix. I suspect you are missing the 'roles' part. You could also restore that document from a backup.

Any insights or observations you have into how your db got into this state would be helpful.

Actions #8

Updated by about 7 years ago

I did an upgrade from 2.10.3 to 2.12 with different roles and users created. The upgrade ran without any errors for me.

Actions #9

Updated by chuckuf about 7 years ago

Thanks for the info. I will look into the matter and post my findings. It looks like I got my Friday night planned. :)

Actions #10

Updated by chuckuf about 7 years ago

bmbouter that was the issue There was a user without a role. I am so grateful for the help. Here are my findings.


"_id" : ObjectId("5724febce72b15065c0eccad"),
"login" : "admin",
"name" : "admin",
"password" : "zYU9sADDZv0=,btl2etOzZRS4u0Ike+wDi3s4nN7/yLdSw1A9qTK//oQ=",
"roles" : [
"_ns" : "users"
"_id" : ObjectId("582bf87dfca49f63f42e30ed"),
"login" : "sync",
"name" : "Sync Account",
"password" : "QDgIa31+bDs=,JDNivY4dmF52S8gq2FWG6xm5uh62mAw9PVNB2cBjEVM=",
"_ns" : "users"

I removed it via

db.users.remove({_id: ObjectId("582bf87dfca49f63f42e30ed")});

I was able then able to run sudo -u apache /usr/bin/pulp-manage-db with no issues.

[root@pulpcos tmp]# pulp-admin status
Status of the server

Api Version: 2
Database Connection:
Connected: True
Known Workers:
_id: scheduler@pulpcos

_ns:            workers
Last Heartbeat: 2017-02-17T23:24:13Z
_id:            resource_manager@pulpcos
_ns:            workers
Last Heartbeat: 2017-02-17T23:24:12Z
_id:            reserved_resource_worker-1@pulpcos
_ns:            workers
Last Heartbeat: 2017-02-17T23:24:13Z
_id:            reserved_resource_worker-0@pulpcos
_ns:            workers
Last Heartbeat: 2017-02-17T23:24:13Z
Messaging Connection:  
Connected: True
Platform Version: 2.12

As far as what caused this, I cannot be certain. Although I would most likely say human error. I was messing around with the users recently trying to figure out how to create a user with the minimal permissions needed to sync a repo.

Actions #11

Updated by bmbouter about 7 years ago

  • Status changed from NEW to CLOSED - WORKSFORME

I'm so glad this got worked our for you. Since it sounds like the issue was in your specific environment I'm going to close as WORKFORME. If someone else reproduces the symptoms please reopen this issue.

Actions #12

Updated by bmbouter almost 7 years ago

  • Category deleted (15)
  • Status changed from CLOSED - WORKSFORME to NEW
  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High

Reopening after another user report via pulp-list:

Actions #13

Updated by kfiresmith almost 7 years ago

First off huge thanks to @Michael Hrivnak and Brian Bouterse for helping me out on a Saturday!

The query for anyone else who hits this is:
db.users.find({roles: {'$exists': false}})

{ "_id" : ObjectId("57fd2e00c969f7767cefac56"), "login" : "", "name" : "Jim Bob", "password" : "*****=", "_ns" : "users" }

db.users.remove({_id: ObjectId("57fd2e00c969f7767cefac56")});

WriteResult({ "nRemoved" : 1 })

Actions #14

Updated by mhrivnak almost 7 years ago

The problem seems to be that a user existed in the DB without a "roles" attribute. A workaround is to do this in a mongo shell:

db.users.update({roles: {'$exists': false}}, {'$set': {'roles': []}}, {multi: true})

That command will find any such user record and add a "roles" attribute to it, with an empty array. This is the correct state for a user that has no associated roles.

We still aren't sure how a user object can get into this state. kfiresmith says in his case, the user object was created in the past by an older version of Pulp, he thinks perhaps 2.9.

Actions #15

Updated by kfiresmith almost 7 years ago

This might be overkill, but I'm wondering if some kind of "pre-flight check" that thumbs through the DB might be in order as a safety feature, where various object aspects that would bring a migration unit down can be found and alerted upon before initiating an upgrade.

Actions #16

Updated by mhrivnak almost 7 years ago

I created a user with pulp-admin on Pulp 2.10.3, and it worked as expected.

$ pulp-admin auth user create --login me

Here is what I found in the database:

{ "_id" : ObjectId("58cda0bbaf235b00074d2939"), "login" : "me", "name" : "me", "password" : "ptYcmP0reII=,1TXwGZQ+EGtVcU8s2cCTkZ5Z+McOqfuDUJicyEJnkpg=", "roles" : [ ], "_ns" : "users" }

The "roles" attribute is there with a correct default value of an empty array.

Actions #17

Updated by bmbouter almost 7 years ago

What if the database was already missing the roles attribute and it only became a problem at upgrade to 2.12? I haven't checked to see if some new validation was added between 2.10 and 2.12, but I want to share this theory.

@kodiak, would you be able to share an example of the troublesome user mongo document pre-upgrade?

Actions #18

Updated by bizhang almost 7 years ago

  • Status changed from NEW to CLOSED - WONTFIX

Closing this as won't fix since there is a workaround. We can reopen if a reproducer is discovererd

Actions #19

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF