Issue #6084: Migration plugin error when trying to migrate the docker plugin with pulpcore 3.1+ and pulp_container >1.0 - Migration Plugin - Pulp

Actions

Copy link

#1

Updated by dalley about 4 years ago

Blocks Issue #5978: Migration of multiple plugins does not work added

Actions

Copy link

#2

Updated by dalley about 4 years ago

Subject changed from Migration plugin error when trying to sync the docker plugin to Migration plugin error when trying to migrate the docker plugin

Actions

Copy link

#3

Updated by ttereshc about 4 years ago

Does it happen with 3.1? I thought Katello needs everything to work with 3.0 first. We need to check with Justin.

Actions

Copy link

#4

Updated by ipanova@redhat.com about 4 years ago

dalley this happens because of the changes happened in pulpcore related to futures, we updated the container plugin but forgot to update the migration plugin code, Here is the PR we submitted https://github.com/pulp/pulp_container/pull/38/files. I think we just need to remove the does_batch flag and that should do it.

@tanya, yes these changes landed in 3.1, we can just pin migration plugin to <3.1 https://github.com/pulp/pulp-2to3-migration/blob/master/setup.py#L6

Actions

Copy link

#5

Updated by dalley about 4 years ago

I believe there's more than one problem. Fixing does_batch does solve one of them (thanks Ipanova!) but even after that I still get:

Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: pulp: rq.worker:INFO: 21012@pulp2-nightly-pulp3-source-centos7.localhost.example.com: 113882d9-c64d-492f-bc49-8f3a48625f8a
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21014]: pulp: rq.worker:INFO: resource-manager: Job OK (36f8d9e8-a935-4d2f-970a-636ccee9ec6e)
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: pulp: pulp_2to3_migration.pulp2.connection:INFO: Attempting to connect to localhost:27017
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: pulp: asyncio:ERROR: Task exception was never retrieved
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: future: <Task finished coro=<DockerMigrator.migrate_content_to_pulp3() done, defined at /home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/docker/migrator.py:73> exception=AttributeError("'NoneType' object has no attribute '_artifacts'",)>
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: Traceback (most recent call last):
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/docker/migrator.py", line 80, in migrate_content_to_pulp3
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: await dm.create()
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py", line 83, in create
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: await pipeline
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 228, in create_pipeline
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: await asyncio.gather(*futures)
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: await self.run()
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/content_stages.py", line 94, in run
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: await self._pre_save(batch)
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/docker/migrator.py", line 273, in _pre_save
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: artifact = man._artifacts.get()
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: AttributeError: 'NoneType' object has no attribute '_artifacts'

The problem is that man is "None", because the Manifest.objects.all() queryset is empty at that point in time: https://github.com/pulp/pulp-2to3-migration/blob/master/pulp_2to3_migration/app/plugin/docker/migrator.py#L272

Not knowing anything about the docker plugin / repo structure, is it a correct assumption that a Manifest is required in this instance? Or are there valid circumstances when it would just come back as Nonethat we need to handle?

Actions

Copy link

#6

Updated by dalley about 4 years ago

I get this error even with

pulp-2to3-migration (0.0.1b2.dev0, /home/vagrant/devel/pulp-2to3-migration)
pulp-container (1.0.0)
pulp-file (0.2.0.dev0, /home/vagrant/devel/pulp_file)
pulpcore (3.0.0)

Actions

Copy link

#7

Updated by ipanova@redhat.com about 4 years ago

dalley wrote:

I believe there's more than one problem. Fixing does_batch does solve one of them (thanks Ipanova!) but even after that I still get:

Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: pulp: rq.worker:INFO: 21012@pulp2-nightly-pulp3-source-centos7.localhost.example.com: 113882d9-c64d-492f-bc49-8f3a48625f8a
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21014]: pulp: rq.worker:INFO: resource-manager: Job OK (36f8d9e8-a935-4d2f-970a-636ccee9ec6e)
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: pulp: pulp_2to3_migration.pulp2.connection:INFO: Attempting to connect to localhost:27017
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: pulp: asyncio:ERROR: Task exception was never retrieved
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: future: <Task finished coro=<DockerMigrator.migrate_content_to_pulp3() done, defined at /home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/docker/migrator.py:73> exception=AttributeError("'NoneType' object has no attribute '_artifacts'",)>
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: Traceback (most recent call last):
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/docker/migrator.py", line 80, in migrate_content_to_pulp3
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: await dm.create()
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py", line 83, in create
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: await pipeline
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 228, in create_pipeline
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: await asyncio.gather(*futures)
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: await self.run()
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/content_stages.py", line 94, in run
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: await self._pre_save(batch)
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/docker/migrator.py", line 273, in _pre_save
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: artifact = man._artifacts.get()
Feb 04 13:35:43 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[21012]: AttributeError: 'NoneType' object has no attribute '_artifacts'

The problem is that man is "None", because the Manifest.objects.all() queryset is empty at that point in time: https://github.com/pulp/pulp-2to3-migration/blob/master/pulp_2to3_migration/app/plugin/docker/migrator.py#L272

Not knowing anything about the docker plugin / repo structure, is it a correct assumption that a Manifest is required in this instance? Or are there valid circumstances when it would just come back as Nonethat we need to handle?

dalley i can confirm that does_batch does not fix the issue ( if using pulpcore 3.1). we are relying to see manifest already saved in the db because we send it first into the queue and with does_batch set to false we were able to control the order. I am afraid we will need to look into this more after the changes happened in core. https://github.com/pulp/pulp-2to3-migration/blob/master/pulp_2to3_migration/app/plugin/docker/migrator.py#L270

Actions

Copy link

#8

Updated by ipanova@redhat.com about 4 years ago

dalley wrote:

I get this error even with

pulp-2to3-migration (0.0.1b2.dev0, /home/vagrant/devel/pulp-2to3-migration)
pulp-container (1.0.0)
pulp-file (0.2.0.dev0, /home/vagrant/devel/pulp_file)
pulpcore (3.0.0)

i am surprised, we need to investigate this.

Actions

Copy link

#9

Updated by dalley about 4 years ago

Status changed from ASSIGNED to NEW
Assignee deleted (~~dalley~~)

Seems likely that it was partly a mistake on my part -- I removed the does_batch flag for pulpcore 3.1 and forgot to restore it for pulpcore 3.0 where it is needed.

However, the failure on pulpcore 3.1 due to the removal of this flag remains an issue. Removing does_batch changed the order in which things happen in a way that breaks us, so a fix will need to be found for that.

Actions

Copy link

#10

Updated by dalley about 4 years ago

Subject changed from Migration plugin error when trying to migrate the docker plugin to Migration plugin error when trying to migrate the docker plugin with pulpcore 3.1+ and pulp_container >1.0

Actions

Copy link

#11

Updated by ipanova@redhat.com about 4 years ago

Triaged changed from No to Yes

Actions

Copy link

#12

Updated by sajha about 4 years ago

Tags Katello-P2 added

Actions

Copy link

#13

Updated by sajha about 4 years ago

Note that the progress report when hitting this issue did not capture this error. The progress report looked like below for me when I saw the error in my logs.

{
    "count": 2,
    "next": null,
    "previous": null,
    "results": [
        {
            "created_resources": [
                "/pulp/api/v3/distributions/container/container/b8c10b9d-9a95-4094-a1ad-f59a714d7667/"
            ],
            "error": null,
            "finished_at": "2020-03-09T16:13:07.765973Z",
            "name": "pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
            "progress_reports": [
                {
                    "code": "processing.repositories",
                    "done": 1,
                    "message": "Processing Pulp 2 repositories, importers, distributors",
                    "state": "completed",
                    "suffix": null,
                    "total": 1
                },
                {
                    "code": "creating.repositories",
                    "done": 1,
                    "message": "Creating repositories in Pulp 3",
                    "state": "completed",
                    "suffix": null,
                    "total": 1
                },
                {
                    "code": "migrating.importers",
                    "done": 1,
                    "message": "Migrating importers to Pulp 3",
                    "state": "completed",
                    "suffix": null,
                    "total": 1
                },
                {
                    "code": "premigrating.content.general",
                    "done": 0,
                    "message": "Pre-migrating Pulp 2 DOCKER_MANIFEST_LIST content (general info)",
                    "state": "completed",
                    "suffix": null,
                    "total": 0
                },
                {
                    "code": "premigrating.content.detail",
                    "done": 0,
                    "message": "Pre-migrating Pulp 2 DOCKER_MANIFEST_LIST content (detail info)",
                    "state": "completed",
                    "suffix": null,
                    "total": 0
                },
                {
                    "code": "premigrating.content.general",
                    "done": 5,
                    "message": "Pre-migrating Pulp 2 DOCKER_BLOB content (general info)",
                    "state": "completed",
                    "suffix": null,
                    "total": 5
                },
                {
                    "code": "premigrating.content.general",
                    "done": 1,
                    "message": "Pre-migrating Pulp 2 DOCKER_TAG content (general info)",
                    "state": "completed",
                    "suffix": null,
                    "total": 1
                },
                {
                    "code": "premigrating.content.detail",
                    "done": 1,
                    "message": "Pre-migrating Pulp 2 DOCKER_TAG content (detail info)",
                    "state": "completed",
                    "suffix": null,
                    "total": 1
                },
                {
                    "code": "premigrating.content.detail",
                    "done": 5,
                    "message": "Pre-migrating Pulp 2 DOCKER_BLOB content (detail info)",
                    "state": "completed",
                    "suffix": null,
                    "total": 5
                },
                {
                    "code": "premigrating.content.general",
                    "done": 2,
                    "message": "Pre-migrating Pulp 2 DOCKER_MANIFEST content (general info)",
                    "state": "completed",
                    "suffix": null,
                    "total": 2
                },
                {
                    "code": "premigrating.content.detail",
                    "done": 2,
                    "message": "Pre-migrating Pulp 2 DOCKER_MANIFEST content (detail info)",
                    "state": "completed",
                    "suffix": null,
                    "total": 2
                },
                {
                    "code": "migrating.docker.content",
                    "done": 0,
                    "message": "Migrating docker content to Pulp 3 docker_blob",
                    "state": "completed",
                    "suffix": null,
                    "total": 5
                },
                {
                    "code": "migrating.docker.content",
                    "done": 0,
                    "message": "Migrating docker content to Pulp 3 docker_manifest",
                    "state": "completed",
                    "suffix": null,
                    "total": 2
                },
                {
                    "code": "migrating.docker.content",
                    "done": 0,
                    "message": "Migrating docker content to Pulp 3 docker_manifest_list",
                    "state": "completed",
                    "suffix": null,
                    "total": 0
                },
                {
                    "code": "migrating.docker.content",
                    "done": 1,
                    "message": "Migrating docker content to Pulp 3 docker_tag",
                    "state": "completed",
                    "suffix": null,
                    "total": 1
                },
                {
                    "code": "migrating.content",
                    "done": 8,
                    "message": "Migrating content to Pulp 3",
                    "state": "completed",
                    "suffix": null,
                    "total": 8
                },
                {
                    "code": "migrating.distributors",
                    "done": 1,
                    "message": "Migrating distributors to Pulp 3",
                    "state": "completed",
                    "suffix": null,
                    "total": 1
                }
            ],
            "pulp_created": "2020-03-09T16:13:07.178288Z",
            "pulp_href": "/pulp/api/v3/tasks/117d4345-2549-43c4-81b7-f0003ca9fb92/",
            "reserved_resources_record": [
                "pulp_2to3_migration"
            ],
            "started_at": "2020-03-09T16:13:07.406690Z",
            "state": "completed",
            "worker": "/pulp/api/v3/workers/42fad837-5ba7-4ee8-9e32-51fbff829d30/"
        },
        {
            "created_resources": [],
            "error": null,
            "finished_at": "2020-03-09T16:13:07.297644Z",
            "name": "pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
            "progress_reports": [
                {
                    "code": "processing.repositories",
                    "done": 0,
                    "message": "Processing Pulp 2 repositories, importers, distributors",
                    "state": "completed",
                    "suffix": null,
                    "total": 0
                },
                {
                    "code": "creating.repositories",
                    "done": 0,
                    "message": "Creating repositories in Pulp 3",
                    "state": "completed",
                    "suffix": null,
                    "total": 0
                },
                {
                    "code": "migrating.importers",
                    "done": 0,
                    "message": "Migrating importers to Pulp 3",
                    "state": "completed",
                    "suffix": null,
                    "total": 0
                },
                {
                    "code": "premigrating.content.general",
                    "done": 0,
                    "message": "Pre-migrating Pulp 2 ISO content (general info)",
                    "state": "completed",
                    "suffix": null,
                    "total": 0
                },
                {
                    "code": "premigrating.content.detail",
                    "done": 0,
                    "message": "Pre-migrating Pulp 2 ISO content (detail info)",
                    "state": "completed",
                    "suffix": null,
                    "total": 0
                },
                {
                    "code": "migrating.iso.content",
                    "done": 0,
                    "message": "Migrating iso content to Pulp 3 iso",
                    "state": "completed",
                    "suffix": null,
                    "total": 0
                },
                {
                    "code": "migrating.content",
                    "done": 0,
                    "message": "Migrating content to Pulp 3",
                    "state": "completed",
                    "suffix": null,
                    "total": 0
                },
                {
                    "code": "migrating.distributors",
                    "done": 0,
                    "message": "Migrating distributors to Pulp 3",
                    "state": "completed",
                    "suffix": null,
                    "total": 0
                }
            ],
            "pulp_created": "2020-03-09T16:13:07.148624Z",
            "pulp_href": "/pulp/api/v3/tasks/643f5ff5-ab44-420b-a3f9-6f09b2ad4a30/",
            "reserved_resources_record": [
                "pulp_2to3_migration"
            ],
            "started_at": "2020-03-09T16:13:07.218988Z",
            "state": "completed",
            "worker": "/pulp/api/v3/workers/42fad837-5ba7-4ee8-9e32-51fbff829d30/"
        }
    ]
}

Actions

Copy link

#14