Project

Profile

Help

Issue #8084

"'NoneType' object is not subscriptable" error during migration

Added by ggainey 12 days ago. Updated 2 days ago.

Status:
NEW
Priority:
High
Assignee:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 89
Quarter:

Description

Migrating a large number of repositories almost succeeded, until failing with a "description": "'NoneType' object is not subscriptable" error at the migrate-distributions step. At a guess, one of the 274 repositories has something unexpected, but I have not yet tracked it down to the exact one. I do have a snapshot of the Pulp2 mongo database to use when needed.

The Giant List Of Repos in this big test can be found here :

https://github.com/ggainey/pulp_startup/blob/main/8044_create_index/all_sync_pulp2.sh

The Mongo db is 7.6GB and can be made available on request.

The task-output of the failed task:

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 7781_update_ordering]$ http :/pulp/api/v3/tasks/89f2388d-6c44-4307-af8a-97afb4e940ba/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: Keep-Alive
Content-Length: 7412
Content-Type: application/json
Correlation-ID: b56675d8977a4c58b60ce861f417e6e3
Date: Sat, 09 Jan 2021 01:16:00 GMT
Keep-Alive: timeout=5, max=10000
Server: gunicorn/20.0.4
Vary: Accept,Cookie
X-Frame-Options: SAMEORIGIN

{
    "child_tasks": [],
    "created_resources": [
        "/pulp/api/v3/task-groups/22082f8a-60fd-4afd-84aa-8bc9cb43a3ed/"
    ],
    "error": {
        "description": "'NoneType' object is not subscriptable",
        "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 975, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 696, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 719, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/tasks/migrate.py\", line 141, in migrate_from_pulp2\n    migrate_content(plan, skip_corrupted=skip_corrupted)\n  File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/migration.py\", line 47, in migrate_content\n    plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)\n  File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/rpm/migrator.py\", line 145, in migrate_content_to_pulp3\n    loop.run_until_complete(dm.create())\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py\", line 90, in create\n    await pipeline\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py\", line 182, in run\n    self.migrate_to_pulp3(cmodel, ctype)\n  File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py\", line 277, in migrate_to_pulp3\n    pulp3content, extra_info = pulp_2to3_detail_content.create_pulp3_content()\n  File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/rpm/pulp_2to3_models.py\", line 775, in create_pulp3_content\n    return (DistributionTree(**treeinfo_serialized[\"distribution_tree\"]),\n"
    },
    "finished_at": "2021-01-09T01:04:42.648930Z",
    "logging_cid": "315467a7170848f29974bcb3d5273961",
    "name": "pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
    "parent_task": null,
    "progress_reports": [
        {
            "code": "premigrating.content.general",
            "done": 0,
            "message": "Pre-migrating Pulp 2 SRPM content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 0
        },
        {
            "code": "premigrating.content.detail",
            "done": 0,
            "message": "Pre-migrating Pulp 2 SRPM content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 0
        },
        {
            "code": "premigrating.content.general",
            "done": 254,
            "message": "Pre-migrating Pulp 2 MODULEMD_DEFAULTS content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 254
        },
        {
            "code": "premigrating.content.general",
            "done": 23,
            "message": "Pre-migrating Pulp 2 DISTRIBUTION content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 23
        },
        {
            "code": "premigrating.content.detail",
            "done": 23,
            "message": "Pre-migrating Pulp 2 DISTRIBUTION content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 23
        },
        {
            "code": "processing.repositories",
            "done": 127,
            "message": "Processing Pulp 2 repositories, importers, distributors",
            "state": "completed",
            "suffix": null,
            "total": 127
        },
        {
            "code": "premigrating.content.detail",
            "done": 254,
            "message": "Pre-migrating Pulp 2 MODULEMD_DEFAULTS content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 254
        },
        {
            "code": "premigrating.content.general",
            "done": 75,
            "message": "Pre-migrating Pulp 2 YUM_REPO_METADATA_FILE content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 75
        },
        {
            "code": "premigrating.content.detail",
            "done": 75,
            "message": "Pre-migrating Pulp 2 YUM_REPO_METADATA_FILE content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 75
        },
        {
            "code": "premigrating.content.detail",
            "done": 112495,
            "message": "Pre-migrating Pulp 2 ERRATUM content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 112495
        },
        {
            "code": "premigrating.content.general",
            "done": 371,
            "message": "Pre-migrating Pulp 2 MODULEMD content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 371
        },
        {
            "code": "premigrating.content.detail",
            "done": 371,
            "message": "Pre-migrating Pulp 2 MODULEMD content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 371
        },
        {
            "code": "premigrating.content.general",
            "done": 4745,
            "message": "Pre-migrating Pulp 2 PACKAGE_GROUP content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 4745
        },
        {
            "code": "premigrating.content.detail",
            "done": 4745,
            "message": "Pre-migrating Pulp 2 PACKAGE_GROUP content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 4745
        },
        {
            "code": "premigrating.content.general",
            "done": 112495,
            "message": "Pre-migrating Pulp 2 ERRATUM content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 112495
        },
        {
            "code": "premigrating.content.general",
            "done": 34,
            "message": "Pre-migrating Pulp 2 PACKAGE_LANGPACKS content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 34
        },
        {
            "code": "premigrating.content.detail",
            "done": 34,
            "message": "Pre-migrating Pulp 2 PACKAGE_LANGPACKS content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 34
        },
        {
            "code": "premigrating.content.general",
            "done": 396,
            "message": "Pre-migrating Pulp 2 PACKAGE_CATEGORY content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 396
        },
        {
            "code": "premigrating.content.detail",
            "done": 396,
            "message": "Pre-migrating Pulp 2 PACKAGE_CATEGORY content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 396
        },
        {
            "code": "premigrating.content.general",
            "done": 180,
            "message": "Pre-migrating Pulp 2 PACKAGE_ENVIRONMENT content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 180
        },
        {
            "code": "premigrating.content.detail",
            "done": 180,
            "message": "Pre-migrating Pulp 2 PACKAGE_ENVIRONMENT content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 180
        },
        {
            "code": "creating.repositories",
            "done": 127,
            "message": "Creating repositories in Pulp 3",
            "state": "completed",
            "suffix": null,
            "total": 127
        },
        {
            "code": "migrating.importers",
            "done": 127,
            "message": "Migrating importers to Pulp 3",
            "state": "completed",
            "suffix": null,
            "total": 127
        },
        {
            "code": "migrating.rpm.content",
            "done": 259738,
            "message": "Migrating rpm content to Pulp 3 rpm",
            "state": "completed",
            "suffix": null,
            "total": 259738
        },
        {
            "code": "migrating.content",
            "done": 0,
            "message": "Migrating content to Pulp 3",
            "state": "failed",
            "suffix": null,
            "total": 0
        },
        {
            "code": "premigrating.content.general",
            "done": 259738,
            "message": "Pre-migrating Pulp 2 RPM content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 259738
        },
        {
            "code": "premigrating.content.detail",
            "done": 259738,
            "message": "Pre-migrating Pulp 2 RPM content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 259738
        },
        {
            "code": "migrating.rpm.content",
            "done": 0,
            "message": "Migrating rpm content to Pulp 3 srpm",
            "state": "completed",
            "suffix": null,
            "total": 0
        },
        {
            "code": "migrating.rpm.content",
            "done": 0,
            "message": "Migrating rpm content to Pulp 3 distribution",
            "state": "failed",
            "suffix": null,
            "total": 23
        }
    ],
    "pulp_created": "2021-01-08T20:40:04.242552Z",
    "pulp_href": "/pulp/api/v3/tasks/89f2388d-6c44-4307-af8a-97afb4e940ba/",
    "reserved_resources_record": [
        "pulp_2to3_migration"
    ],
    "started_at": "2021-01-08T20:40:04.441480Z",
    "state": "failed",
    "task_group": "/pulp/api/v3/task-groups/22082f8a-60fd-4afd-84aa-8bc9cb43a3ed/",
    "worker": "/pulp/api/v3/workers/2d6a7e75-85d7-471b-9ce0-aa850599656d/"
}

History

#1 Updated by jsherril@redhat.com 10 days ago

  • Priority changed from Normal to High
  • Tags Katello added

I hit this as well on a user database

#3 Updated by dalley 6 days ago

    def get_treeinfo_serialized(self):
        """
        Create a Pulp 3 Distribution content for saving later in a bulk operation.

        Returns:
            dict: a serialized treeinfo data

        """
        namespaces = [".treeinfo", "treeinfo"]
        for namespace in namespaces:
            treeinfo = PulpTreeInfo()
            try:
                treeinfo.load(f=os.path.join(self.pulp2content.pulp2_storage_path, namespace))
            except FileNotFoundError:
                continue
            self.filename = namespace
            treeinfo_parsed = treeinfo.parsed_sections()
            treeinfo_serialized = TreeinfoData(treeinfo_parsed).to_dict(filename=namespace)
            # Pulp 2 only knows about the top level kickstart repository
            treeinfo_serialized["repositories"] = {'.': None}
            # Pulp 2 did not support addon repositories, so we should not list them here either
            treeinfo_serialized['addons'] = {}
            # Pulp 2 only supported variants that are in the root of the repository
            variants = {}
            for name, variant in treeinfo_serialized['variants'].items():
                if variant['repository'] == '.':
                    variants[name] = variant
            treeinfo_serialized['variants'] = variants
            # Reset build_timestamp so Pulp will fetch all the addons during the next sync
            treeinfo_serialized['distribution_tree']['build_timestamp'] = 0
            return treeinfo_serialized

    def create_pulp3_content(self):
        """
        Create a Pulp 3 Distribution content for saving later in a bulk operation.

        Returns:
            (DistributionTree, dict): an in-memory DistributionTree content and serialized
                                      treeinfo data

        """
        treeinfo_serialized = self.get_treeinfo_serialized()
        return (DistributionTree(**treeinfo_serialized["distribution_tree"]),
                treeinfo_serialized)

It looks like "treeinfo_serilized" is None and therefore the subscript operation fails. Either .get_treeinfo_serialized() is falsely returning None, or this chunk of code is being called in a situation where it shouldn't be.

.get_treeinfo_serialized() should probably throw an assertion failure if it can't find what it's looking for, instead of silent None regardless of where the problem is.

Tanya do you have any ideas?

#4 Updated by dalley 5 days ago

  • Sprint set to Sprint 88

#5 Updated by ttereshc 5 days ago

  • Sprint/Milestone set to Katello, early March 2021

#6 Updated by rchan 2 days ago

  • Sprint changed from Sprint 88 to Sprint 89

Please register to edit this issue

Also available in: Atom PDF