Project

Profile

Help

Issue #8084

"'NoneType' object is not subscriptable" error during migration

Added by ggainey 9 months ago. Updated 9 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 89
Quarter:

Description

Migrating a large number of repositories almost succeeded, until failing with a "description": "'NoneType' object is not subscriptable" error at the migrate-distributions step. At a guess, one of the 274 repositories has something unexpected, but I have not yet tracked it down to the exact one. I do have a snapshot of the Pulp2 mongo database to use when needed.

The Giant List Of Repos in this big test can be found here :

https://github.com/ggainey/pulp_startup/blob/main/8044_create_index/all_sync_pulp2.sh

The Mongo db is 7.6GB and can be made available on request.

The task-output of the failed task:

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 7781_update_ordering]$ http :/pulp/api/v3/tasks/89f2388d-6c44-4307-af8a-97afb4e940ba/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: Keep-Alive
Content-Length: 7412
Content-Type: application/json
Correlation-ID: b56675d8977a4c58b60ce861f417e6e3
Date: Sat, 09 Jan 2021 01:16:00 GMT
Keep-Alive: timeout=5, max=10000
Server: gunicorn/20.0.4
Vary: Accept,Cookie
X-Frame-Options: SAMEORIGIN

{
    "child_tasks": [],
    "created_resources": [
        "/pulp/api/v3/task-groups/22082f8a-60fd-4afd-84aa-8bc9cb43a3ed/"
    ],
    "error": {
        "description": "'NoneType' object is not subscriptable",
        "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 975, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 696, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 719, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/tasks/migrate.py\", line 141, in migrate_from_pulp2\n    migrate_content(plan, skip_corrupted=skip_corrupted)\n  File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/migration.py\", line 47, in migrate_content\n    plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)\n  File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/rpm/migrator.py\", line 145, in migrate_content_to_pulp3\n    loop.run_until_complete(dm.create())\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py\", line 90, in create\n    await pipeline\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py\", line 182, in run\n    self.migrate_to_pulp3(cmodel, ctype)\n  File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py\", line 277, in migrate_to_pulp3\n    pulp3content, extra_info = pulp_2to3_detail_content.create_pulp3_content()\n  File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/rpm/pulp_2to3_models.py\", line 775, in create_pulp3_content\n    return (DistributionTree(**treeinfo_serialized[\"distribution_tree\"]),\n"
    },
    "finished_at": "2021-01-09T01:04:42.648930Z",
    "logging_cid": "315467a7170848f29974bcb3d5273961",
    "name": "pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
    "parent_task": null,
    "progress_reports": [
        {
            "code": "premigrating.content.general",
            "done": 0,
            "message": "Pre-migrating Pulp 2 SRPM content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 0
        },
        {
            "code": "premigrating.content.detail",
            "done": 0,
            "message": "Pre-migrating Pulp 2 SRPM content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 0
        },
        {
            "code": "premigrating.content.general",
            "done": 254,
            "message": "Pre-migrating Pulp 2 MODULEMD_DEFAULTS content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 254
        },
        {
            "code": "premigrating.content.general",
            "done": 23,
            "message": "Pre-migrating Pulp 2 DISTRIBUTION content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 23
        },
        {
            "code": "premigrating.content.detail",
            "done": 23,
            "message": "Pre-migrating Pulp 2 DISTRIBUTION content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 23
        },
        {
            "code": "processing.repositories",
            "done": 127,
            "message": "Processing Pulp 2 repositories, importers, distributors",
            "state": "completed",
            "suffix": null,
            "total": 127
        },
        {
            "code": "premigrating.content.detail",
            "done": 254,
            "message": "Pre-migrating Pulp 2 MODULEMD_DEFAULTS content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 254
        },
        {
            "code": "premigrating.content.general",
            "done": 75,
            "message": "Pre-migrating Pulp 2 YUM_REPO_METADATA_FILE content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 75
        },
        {
            "code": "premigrating.content.detail",
            "done": 75,
            "message": "Pre-migrating Pulp 2 YUM_REPO_METADATA_FILE content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 75
        },
        {
            "code": "premigrating.content.detail",
            "done": 112495,
            "message": "Pre-migrating Pulp 2 ERRATUM content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 112495
        },
        {
            "code": "premigrating.content.general",
            "done": 371,
            "message": "Pre-migrating Pulp 2 MODULEMD content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 371
        },
        {
            "code": "premigrating.content.detail",
            "done": 371,
            "message": "Pre-migrating Pulp 2 MODULEMD content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 371
        },
        {
            "code": "premigrating.content.general",
            "done": 4745,
            "message": "Pre-migrating Pulp 2 PACKAGE_GROUP content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 4745
        },
        {
            "code": "premigrating.content.detail",
            "done": 4745,
            "message": "Pre-migrating Pulp 2 PACKAGE_GROUP content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 4745
        },
        {
            "code": "premigrating.content.general",
            "done": 112495,
            "message": "Pre-migrating Pulp 2 ERRATUM content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 112495
        },
        {
            "code": "premigrating.content.general",
            "done": 34,
            "message": "Pre-migrating Pulp 2 PACKAGE_LANGPACKS content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 34
        },
        {
            "code": "premigrating.content.detail",
            "done": 34,
            "message": "Pre-migrating Pulp 2 PACKAGE_LANGPACKS content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 34
        },
        {
            "code": "premigrating.content.general",
            "done": 396,
            "message": "Pre-migrating Pulp 2 PACKAGE_CATEGORY content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 396
        },
        {
            "code": "premigrating.content.detail",
            "done": 396,
            "message": "Pre-migrating Pulp 2 PACKAGE_CATEGORY content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 396
        },
        {
            "code": "premigrating.content.general",
            "done": 180,
            "message": "Pre-migrating Pulp 2 PACKAGE_ENVIRONMENT content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 180
        },
        {
            "code": "premigrating.content.detail",
            "done": 180,
            "message": "Pre-migrating Pulp 2 PACKAGE_ENVIRONMENT content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 180
        },
        {
            "code": "creating.repositories",
            "done": 127,
            "message": "Creating repositories in Pulp 3",
            "state": "completed",
            "suffix": null,
            "total": 127
        },
        {
            "code": "migrating.importers",
            "done": 127,
            "message": "Migrating importers to Pulp 3",
            "state": "completed",
            "suffix": null,
            "total": 127
        },
        {
            "code": "migrating.rpm.content",
            "done": 259738,
            "message": "Migrating rpm content to Pulp 3 rpm",
            "state": "completed",
            "suffix": null,
            "total": 259738
        },
        {
            "code": "migrating.content",
            "done": 0,
            "message": "Migrating content to Pulp 3",
            "state": "failed",
            "suffix": null,
            "total": 0
        },
        {
            "code": "premigrating.content.general",
            "done": 259738,
            "message": "Pre-migrating Pulp 2 RPM content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 259738
        },
        {
            "code": "premigrating.content.detail",
            "done": 259738,
            "message": "Pre-migrating Pulp 2 RPM content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 259738
        },
        {
            "code": "migrating.rpm.content",
            "done": 0,
            "message": "Migrating rpm content to Pulp 3 srpm",
            "state": "completed",
            "suffix": null,
            "total": 0
        },
        {
            "code": "migrating.rpm.content",
            "done": 0,
            "message": "Migrating rpm content to Pulp 3 distribution",
            "state": "failed",
            "suffix": null,
            "total": 23
        }
    ],
    "pulp_created": "2021-01-08T20:40:04.242552Z",
    "pulp_href": "/pulp/api/v3/tasks/89f2388d-6c44-4307-af8a-97afb4e940ba/",
    "reserved_resources_record": [
        "pulp_2to3_migration"
    ],
    "started_at": "2021-01-08T20:40:04.441480Z",
    "state": "failed",
    "task_group": "/pulp/api/v3/task-groups/22082f8a-60fd-4afd-84aa-8bc9cb43a3ed/",
    "worker": "/pulp/api/v3/workers/2d6a7e75-85d7-471b-9ce0-aa850599656d/"
}

Associated revisions

Revision c66c3cba View on GitHub
Added by ggainey 9 months ago

Taught rpm-migration to be more stable in the face of missing treeinfos.

fixes #8084

Revision c66c3cba View on GitHub
Added by ggainey 9 months ago

Taught rpm-migration to be more stable in the face of missing treeinfos.

fixes #8084

Revision c66c3cba View on GitHub
Added by ggainey 9 months ago

Taught rpm-migration to be more stable in the face of missing treeinfos.

fixes #8084

History

#1 Updated by jsherril@redhat.com 9 months ago

  • Priority changed from Normal to High
  • Tags Katello added

I hit this as well on a user database

#3 Updated by dalley 9 months ago

    def get_treeinfo_serialized(self):
        """
        Create a Pulp 3 Distribution content for saving later in a bulk operation.

        Returns:
            dict: a serialized treeinfo data

        """
        namespaces = [".treeinfo", "treeinfo"]
        for namespace in namespaces:
            treeinfo = PulpTreeInfo()
            try:
                treeinfo.load(f=os.path.join(self.pulp2content.pulp2_storage_path, namespace))
            except FileNotFoundError:
                continue
            self.filename = namespace
            treeinfo_parsed = treeinfo.parsed_sections()
            treeinfo_serialized = TreeinfoData(treeinfo_parsed).to_dict(filename=namespace)
            # Pulp 2 only knows about the top level kickstart repository
            treeinfo_serialized["repositories"] = {'.': None}
            # Pulp 2 did not support addon repositories, so we should not list them here either
            treeinfo_serialized['addons'] = {}
            # Pulp 2 only supported variants that are in the root of the repository
            variants = {}
            for name, variant in treeinfo_serialized['variants'].items():
                if variant['repository'] == '.':
                    variants[name] = variant
            treeinfo_serialized['variants'] = variants
            # Reset build_timestamp so Pulp will fetch all the addons during the next sync
            treeinfo_serialized['distribution_tree']['build_timestamp'] = 0
            return treeinfo_serialized

    def create_pulp3_content(self):
        """
        Create a Pulp 3 Distribution content for saving later in a bulk operation.

        Returns:
            (DistributionTree, dict): an in-memory DistributionTree content and serialized
                                      treeinfo data

        """
        treeinfo_serialized = self.get_treeinfo_serialized()
        return (DistributionTree(**treeinfo_serialized["distribution_tree"]),
                treeinfo_serialized)

It looks like "treeinfo_serilized" is None and therefore the subscript operation fails. Either .get_treeinfo_serialized() is falsely returning None, or this chunk of code is being called in a situation where it shouldn't be.

.get_treeinfo_serialized() should probably throw an assertion failure if it can't find what it's looking for, instead of silent None regardless of where the problem is.

Tanya do you have any ideas?

#4 Updated by dalley 9 months ago

  • Sprint set to Sprint 88

#5 Updated by ttereshc 9 months ago

  • Sprint/Milestone set to 0.9.0

#6 Updated by rchan 9 months ago

  • Sprint changed from Sprint 88 to Sprint 89

#7 Updated by ggainey 9 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ggainey

#8 Updated by ggainey 9 months ago

debugging notes:

Jan 28 03:18:22 pulp2-nightly-pulp3-source-centos7.padre-fedora.example.com rq[6382]: 
  pulp [66acc5ec9aad4796b3cea9ecf2321620]: pulp_2to3_migration.app.plugin.rpm.pulp_2to3_models:
    ERROR: >>> NO TREEINFO FOUND for 
      /var/lib/pulp/content/units/distribution/82/8a832a0d16c7d561db0fa2ac129112501b3003ce1fc677917bc548a430f9bc

pulp=> select * from pulp_2to3_migration_pulp2content where pulp2_storage_path = '/var/lib/pulp/content/units/distribution/82/8a832a0d16c7d561db0fa2ac129112501b3003ce1fc677917bc548a430f9bc'
pulp-> ;
               pulp_id                |         pulp_created          |       pulp_last_updated       |               pulp2_id            
   | pulp2_content_type_id | pulp2_last_updated |                                             pulp2_storage_path                          
                   | downloaded | pulp3_content_id | pulp2_repo_id 
--------------------------------------+-------------------------------+-------------------------------+-----------------------------------
---+-----------------------+--------------------+-----------------------------------------------------------------------------------------
-------------------+------------+------------------+---------------
 63ca3324-c132-4c8e-bb6f-cef1d64c6913 | 2021-01-27 23:41:19.161583+00 | 2021-01-27 23:41:19.161616+00 | 1da47a3a-b568-4efa-be13-59854a0ead
94 | distribution          |         1610014880 | /var/lib/pulp/content/units/distribution/82/8a832a0d16c7d561db0fa2ac129112501b3003ce1fc6
77917bc548a430f9bc | f          |                  | 
(1 row)

pulp=> select * from pulp_2to3_migration_pulp2distribution where pulp_2to3_migration_pulp2distribution.pulp2content_id = '63ca3324-c132-4c8e-bb6f-cef1d64c6913';
               pulp_id                |         pulp_created          |       pulp_last_updated       |                distribution_id    
            |          family          | variant | version |  arch  |           pulp2content_id            
--------------------------------------+-------------------------------+-------------------------------+-----------------------------------
------------+--------------------------+---------+---------+--------+--------------------------------------
 1dd43951-3243-4d97-b990-e84f9309a7cc | 2021-01-27 23:41:19.850627+00 | 2021-01-27 23:41:19.850635+00 | ks-Red Hat Enterprise Linux-BaseOS
-8.3-x86_64 | Red Hat Enterprise Linux | BaseOS  | 8.3     | x86_64 | 63ca3324-c132-4c8e-bb6f-cef1d64c6913
(1 row)

pulp=> 

#9 Updated by ggainey 9 months ago

> db.units_distribution.find({'distribution_id': 'ks-Red Hat Enterprise Linux-BaseOS-8.3-x86_64'})
{ "_id" : "1da47a3a-b568-4efa-be13-59854a0ead94", 
"pulp_user_metadata" : {  }, 
"_last_updated" : 1610014880, 
"_storage_path" : "/var/lib/pulp/content/units/distribution/82/8a832a0d16c7d561db0fa2ac129112501b3003ce1fc677917bc548a430f9bc", 
"downloaded" : false, 
"distribution_id" : "ks-Red Hat Enterprise Linux-BaseOS-8.3-x86_64", 
"family" : "Red Hat Enterprise Linux", "variant" : "BaseOS", "version" : "8.3", "arch" : "x86_64",
 "files" : [
 { "relativepath" : "images/pxeboot/initrd.img", "checksumtype" : "sha256", "checksum" : "2e974f7a2e79d7f7db24beccf9af3b244f9edcd464990c9f6d7bc2e3d9b7151b" },
 { "relativepath" : "images/pxeboot/vmlinuz", "checksumtype" : "sha256", "checksum" : "98371ec7d815c710fa21a7ab3c59a9c3aed98b7a3a2c78237c24528cd5aa3353" },
 { "relativepath" : "images/boot.iso", "checksumtype" : "sha256", "checksum" : "1b73ebfebd1f9424c806032168873b067259d8b29f4e9d39ae0e4009cce49b93" },
 { "relativepath" : "images/efiboot.img", "checksumtype" : "sha256", "checksum" : "cfa231ea085e32b116afcbcdcc87857948f0172b090427c52e45bbc80cefa2f3" }, 
{ "relativepath" : "images/install.img", "checksumtype" : "sha256", "checksum" : "4e00b6a55dd0dc4d560aada4b9c920ec1bdbf1df69c273bb4a59ba98d9e7e9ca" }
], 
"timestamp" : 1602239855, "packagedir" : "Packages", "version_sort_index" : "01-8.01-3", "_ns" : "units_distribution", "_content_type_id" : "distribution" }

#10 Updated by ggainey 9 months ago

Current hypothesis: All the repos in this example are on-demand in pulp2, and have no content sync'd. .treeinfo files never get lazy-content entries (I think). So this code fails, because it 'assumes' that it will always be able to find a .treeinfo file 'on disk' in pulp2.

#11 Updated by ggainey 9 months ago

OK< the source of the problem described here, is that both test-cases are running against a mongo-db export from a long-gone pulp2 system - nothing exists "on disk", only in the DB. Running with skip-corrupted should let the migration complete (after a lot of bitter complaints). However, there is a path involving missing-treeinfo that results in the python-exception that interrupts the migration.

I think I have a path forward that will let this scenario run to completion. We're testing something that is a never-happen-case for "actual" users of migration - but which is very very common for testers...

#12 Updated by ggainey 9 months ago

Testing:

# Sync one ks-repo into Pulp2
pulp-admin login -u admin -p admin
pulp-admin rpm repo create --repo-id rhel8-ks --feed https://cdn.redhat.com/content/dist/rhel8/8/x86_64/baseos/kickstart \
  --download-policy on_demand --generate-sqlite false --repoview false \
  --feed-ca-cert /home/vagrant/devel/pulp_startup/CDN_cert/redhat-uep.pem \
  --feed-cert /home/vagrant/devel/pulp_startup/CDN_cert/cdn.crt \
  --feed-key /home/vagrant/devel/pulp_startup/CDN_cert/cdn.key
pulp-admin rpm repo sync run --repo-id rhel8-ks
# Dump the resulting mongodb
mongodump --db pulp_database --gzip
# Reset pulp2 to 'empty' 
reset_pulp2 
# Restore just the mongodb
mongorestore --gzip .
# Set up and migrate that data into pulp3
http POST :24817/pulp/api/v3/migration-plans/ plan='{"plugins": [{"type": "rpm"}]}' 
http POST /pulp/api/v3/migration-plans/<migration-plan-UUID>/run/ skip_corrupted=true

Without a fix, this will FAIL with the "'NoneType' object is not subscriptable" error. With the fix, the task will complete successfully while complaining about missing/corrupted files.

#13 Updated by ggainey 9 months ago

  • Status changed from ASSIGNED to POST

#14 Updated by ttereshc 9 months ago

  • Sprint/Milestone changed from 0.9.0 to 0.7.0

#15 Updated by ggainey 9 months ago

  • Status changed from POST to MODIFIED

#16 Updated by pulpbot 9 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF