Pulp

chuckuf, what version of Pulp are you upgrading from?

Updated pulp-server-2.10.0-1.el7.noarch @pulp-2-stable
Update 2.12.0-1.el7.noarch @pulp-2-stable

Update
I did an upgrade from 2.11 -> 2.12 -> 2.12.1 beta. I made sure I had some roles & users in pulp as well. Both upgrade from 2.11 -> 2.12 and 2.12 -> 2,12 beta worked with no error.

I would like to try to rollback to upgrade to see if I can get it working again, but I cannot find pulp-server-2.10.0-1 repos anywhere. All I can find is pulp-server-2.10.3. Anyone know where I can find pulp-server-2.10.0-1 repos? I downgraded to pulp-server-2.10.3, but the same error persists.

Unfortunately, we only retain the latest Z-stream. From the traceback you experience I don't think having the 2.10.0-1 files will help you since I see there were no code changes in the /usr/lib/python2.7/site-packages/pulp/server/db/migrations/lib/managers.py file. Your DB in the users table is corrupted somehow. Here is a record from my dev environment which is newer, but it should be the same on 2.10+.

> db.users.find().pretty()
{
        "_id" : ObjectId("58a617dee779890854d270b5"),
        "login" : "admin",
        "name" : "admin",
        "password" : "0bwgQLkAUDU=,jFqcfpnp+r0qR/fHWc63fL3G4ic8YcomS66kedmuHek=",
        "roles" : [
                "super-users"
        ],
        "_ns" : "users"
}

If you compare the documents in your 'user' collection to ^, you should be able to see some differences which you could manually fix. I suspect you are missing the 'roles' part. You could also restore that document from a backup.

Any insights or observations you have into how your db got into this state would be helpful.

I did an upgrade from 2.10.3 to 2.12 with different roles and users created. The upgrade ran without any errors for me.

Thanks for the info. I will look into the matter and post my findings. It looks like I got my Friday night planned. :)

bmbouter that was the issue There was a user without a role. I am so grateful for the help. Here are my findings.

db.users.find().pretty()

{
"_id" : ObjectId("5724febce72b15065c0eccad"),
"login" : "admin",
"name" : "admin",
"password" : "zYU9sADDZv0=,btl2etOzZRS4u0Ike+wDi3s4nN7/yLdSw1A9qTK//oQ=",
"roles" : [
"super-users"
],
"_ns" : "users"
}
{
"_id" : ObjectId("582bf87dfca49f63f42e30ed"),
"login" : "sync",
"name" : "Sync Account",
"password" : "QDgIa31+bDs=,JDNivY4dmF52S8gq2FWG6xm5uh62mAw9PVNB2cBjEVM=",
"_ns" : "users"
}

I removed it via

db.users.remove({_id: ObjectId("582bf87dfca49f63f42e30ed")});

I was able then able to run sudo -u apache /usr/bin/pulp-manage-db with no issues.

[root@pulpcos tmp]# pulp-admin status
--------------------------------------------------------------------
Status of the server
--------------------------------------------------------------------

Api Version: 2
Database Connection:
Connected: True
Known Workers:
_id: scheduler@pulpcos

_ns:            workers
Last Heartbeat: 2017-02-17T23:24:13Z
_id:            resource_manager@pulpcos

_ns:            workers
Last Heartbeat: 2017-02-17T23:24:12Z
_id:            reserved_resource_worker-1@pulpcos

_ns:            workers
Last Heartbeat: 2017-02-17T23:24:13Z
_id:            reserved_resource_worker-0@pulpcos

_ns:            workers
Last Heartbeat: 2017-02-17T23:24:13Z
Messaging Connection:  
Connected: True
Versions:              
Platform Version: 2.12

As far as what caused this, I cannot be certain. Although I would most likely say human error. I was messing around with the users recently trying to figure out how to create a user with the minimal permissions needed to sync a repo.

First off huge thanks to @Michael Hrivnak and Brian Bouterse for helping me out on a Saturday!

The query for anyone else who hits this is:
db.users.find({roles: {'$exists': false}})

{ "_id" : ObjectId("57fd2e00c969f7767cefac56"), "login" : "jimbob@COLLEGE.EDU", "name" : "Jim Bob", "password" : "*****=", "_ns" : "users" }

db.users.remove({_id: ObjectId("57fd2e00c969f7767cefac56")});

WriteResult({ "nRemoved" : 1 })

The problem seems to be that a user existed in the DB without a "roles" attribute. A workaround is to do this in a mongo shell:

db.users.update({roles: {'$exists': false}}, {'$set': {'roles': []}}, {multi: true})

That command will find any such user record and add a "roles" attribute to it, with an empty array. This is the correct state for a user that has no associated roles.

We still aren't sure how a user object can get into this state. kfiresmith says in his case, the user object was created in the past by an older version of Pulp, he thinks perhaps 2.9.

This might be overkill, but I'm wondering if some kind of "pre-flight check" that thumbs through the DB might be in order as a safety feature, where various object aspects that would bring a migration unit down can be found and alerted upon before initiating an upgrade.

I created a user with pulp-admin on Pulp 2.10.3, and it worked as expected.

$ pulp-admin auth user create --login me

Here is what I found in the database:

{ "_id" : ObjectId("58cda0bbaf235b00074d2939"), "login" : "me", "name" : "me", "password" : "ptYcmP0reII=,1TXwGZQ+EGtVcU8s2cCTkZ5Z+McOqfuDUJicyEJnkpg=", "roles" : [ ], "_ns" : "users" }

The "roles" attribute is there with a correct default value of an empty array.

What if the database was already missing the roles attribute and it only became a problem at upgrade to 2.12? I haven't checked to see if some new validation was added between 2.10 and 2.12, but I want to share this theory.

@kodiak, would you be able to share an example of the troublesome user mongo document pre-upgrade?