Pulp

In your case there is a data quality problem in your database that needs to be corrected. Two of your repositories have a 'null' as their repo id. Pulp needs all repos to have a unique value for repo_id. This was likely created due to a Pulp bug, but to upgrade your database to newer Pulp code you'll need to resolve this.

The easy way is to remove one or more of the offending repositories and then re-run the validation tool. You may try to look at the detail view of a repository to determine if the repo_id field is missing or null.

I've taken a look at the collection inside of pulp_database. None of the repositories have the field name repo_id. Used: db.repos.find( { repo_id: { $exists: true }} )

Example repository:

{
        "_id" : ObjectId("553145f6dfb891098394f162"),
        "_ns" : "repos",
        "content_unit_counts" : {
                "distribution" : 1,
                "erratum" : 4807,
                "package_category" : 15,
                "package_group" : 212,
                "rpm" : 5023
        },
        "description" : null,
        "display_name" : "SL66-os-i386",
        "id" : "SL66-os-i386",
        "last_unit_added" : ISODate("2015-06-13T18:18:54.064Z"),
        "last_unit_removed" : null,
        "notes" : {
                "_repo-type" : "rpm-repo"
        },
        "scratchpad" : {
                "checksum_type" : "sha256"
        }
}

I also took a look for duplicate values for _id or id and found none:

> db.repos.aggregate([ { $group: {  _id: { _id: "$_id" }, uniqueIds: { $addToSet: "$_id" }, count: { $sum: 1 }}}, { $match: { count: { $gt: 1 }}} ])
{ "result" : [ ], "ok" : 1 }
> db.repos.aggregate([ { $group: {  _id: { id: "$id" }, uniqueIds: { $addToSet: "$_id" }, count: { $sum: 1 }}}, { $match: { count: { $gt: 1 }}} ])
{ "result" : [ ], "ok" : 1 }

Do you think I should add the field "repo_id" to all repositories?

Thanks for the input!

It looks like the upgrade itself is failing before logging is initialized (line 4).

I am curious what version of docker you are running, and if it came with Scientific Linux or if you installed it yourself? I have some debugging ideas, it might be easiest to talk this through in IRC, #pulp on freenode, feel free to ping me (asmacdo).

@asmacdo makes a great point. The logging is failing to be initialized. _logger.critical is None. Why is that?

Can you list your indexes on the repos collection and verify the name of the field we are looking for?

I think the field i named 'id'. Not to be confused with '_id'. Once the field name is known based on your index config we need to find the duplicates. Your commands look better than what I could produce, but the index must have a duplicate. I think it's going to have two documents in the repos collection that have 'id' unset. Maybe search using not $exists?

Post more ideas or questions and I'll see them.

I've deleted the checksum_type fields and testing came back clean. I'll begin upgrading our pulp system.

Thank you for all of your help!