Actions
Issue #7887
closedIf orphaned content is removed in pulp 2 between migration re-runs, FileNotFoundError is raised
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 93
Quarter:
Q1-2021
Description
It happens only if orphaned content hasn't been fully migrated with the first run.
To reproduce:
- have orphaned content in pulp 2
- run migration, stop it before the content migration is done (ensure that at least one of the orphaned content units was pre-migrated but not migrated.)
- clean orphans in pulp2
- run migration again
{
"child_tasks": [],
"created_resources": [
"/pulp/api/v3/task-groups/f6d728d2-61ea-49f2-b461-f8e0baf33b9b/"
],
"error": {
"description": "[Errno 2] No such file or directory: '/var/lib/pulp/content/units/iso/d7/269cf0f9afe9445d5e31c82cf13be3b17f75af57312a1638659ac592c221fc/1.iso'",
"traceback": " File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n rv = job.perform()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 684, in perform\n self._result = self._execute()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 690, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/tasks/migrate.py\", line 141, in migrate_from_pulp2\n migrate_content(plan, skip_corrupted=skip_corrupted)\n File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/migration.py\", line 47, in migrate_content\n plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)\n File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/iso/migrator.py\", line 64, in migrate_content_to_pulp3\n loop.run_until_complete(dm.create())\n File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n return future.result()\n File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py\", line 89, in create\n await pipeline\n File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n await asyncio.gather(*futures)\n File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py\", line 178, in run\n self.migrate_to_pulp3(cmodel, ctype)\n File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py\", line 378, in migrate_to_pulp3\n downloaded=pulp2content.downloaded\n File \"/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py\", line 128, in create_artifact\n expected_size=expected_size)\n File \"/home/vagrant/devel/pulpcore/pulpcore/app/models/content.py\", line 277, in init_and_validate\n with open(file, \"rb\") as f:\n"
},
"finished_at": "2020-11-23T11:12:54.853991Z",
"name": "pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
"parent_task": null,
"progress_reports": [
{
"code": "creating.repositories",
"done": 0,
"message": "Creating repositories in Pulp 3",
"state": "completed",
"suffix": null,
"total": 0
},
{
"code": "migrating.importers",
"done": 0,
"message": "Migrating importers to Pulp 3",
"state": "completed",
"suffix": null,
"total": 0
},
{
"code": "migrating.content",
"done": 0,
"message": "Migrating content to Pulp 3",
"state": "failed",
"suffix": null,
"total": 0
},
{
"code": "migrating.iso.content",
"done": 13,
"message": "Migrating iso content to Pulp 3 iso",
"state": "failed",
"suffix": null,
"total": 276
},
{
"code": "processing.repositories",
"done": 4,
"message": "Processing Pulp 2 repositories, importers, distributors",
"state": "completed",
"suffix": null,
"total": 4
},
{
"code": "premigrating.content.general",
"done": 0,
"message": "Pre-migrating Pulp 2 ISO content (general info)",
"state": "completed",
"suffix": null,
"total": 0
},
{
"code": "premigrating.content.detail",
"done": 0,
"message": "Pre-migrating Pulp 2 ISO content (detail info)",
"state": "completed",
"suffix": null,
"total": 0
}
],
"pulp_created": "2020-11-23T11:12:54.394083Z",
"pulp_href": "/pulp/api/v3/tasks/e1a1a18e-bebe-4f74-bde8-804432964765/",
"reserved_resources_record": [
"pulp_2to3_migration"
],
"started_at": "2020-11-23T11:12:54.538645Z",
"state": "failed",
"task_group": "/pulp/api/v3/task-groups/f6d728d2-61ea-49f2-b461-f8e0baf33b9b/",
"worker": "/pulp/api/v3/workers/74ba07cd-4830-4527-9c70-452f16fe2de5/"
}
Actions
Remove Pulp2Content records if content is no longer in Pulp2
If some content is only pre-migrated and not migrated to Pulp 3, and then removed from Pulp 2, it can cause a problem with subsequent migration run. Such content is treated as corrupted but in reality, it's no longer in Pulp 2. Now such records are removed before any pre-migration starts.
closes #7887 https://pulp.plan.io/issues/7887