Issue #5980
Updated by ttereshc almost 5 years ago
A migration plan has to be declarative.
The migration result should not be affected by previous attempts with different migration plans.
In the example below, repositories @file@ and @file2@ are correctly migrated according to the MP.
Then migration plan is changed to "migrate-all" (4 repositories, including @file@ and @file2@).
In the result only 3 repositories are present. One was not migrated because it has older timestamp than @file@ and @file2@.
<pre>
http POST :24817/pulp/api/v3/migration-plans/ plan='{"plugins": [{ "type": "iso", "repositories" : [{"name": "file","pulp2_importer_repository_id": "file", "repository_versions": [{"pulp2_repository_id": "file"}]}, {"name": "file2","pulp2_importer_repository_id": "file2", "repository_versions": [{"pulp2_repository_id": "file2"}]}]}]}'
HTTP/1.1 201 Created
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 401
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:36:28 GMT
Location: /pulp/api/v3/migration-plans/8b54478b-97aa-4457-8d60-485706e2ca2c/
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"plan": {
"plugins": [
{
"repositories": [
{
"name": "file",
"pulp2_importer_repository_id": "file",
"repository_versions": [
{
"pulp2_repository_id": "file"
}
]
},
{
"name": "file2",
"pulp2_importer_repository_id": "file2",
"repository_versions": [
{
"pulp2_repository_id": "file2"
}
]
}
],
"type": "iso"
}
]
},
"pulp_created": "2020-01-13T18:36:28.540285Z",
"pulp_href": "/pulp/api/v3/migration-plans/8b54478b-97aa-4457-8d60-485706e2ca2c/"
}
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http POST :24817/pulp/api/v3/migration-plans/8b54478b-97aa-4457-8d60-485706e2ca2c/run/
HTTP/1.1 202 Accepted
Allow: POST, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:36:39 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"task": "/pulp/api/v3/tasks/b14e8b97-0408-4106-a3d0-30850d93b60b/"
}
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http GET :24817/pulp/api/v3/pulp2repositories/
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 1054
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:36:54 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"count": 2,
"next": null,
"previous": null,
"results": [
{
"is_migrated": true,
"not_in_pulp2": false,
"pulp2_object_id": "5e1c932dc998ac367a9b79d0",
"pulp2_repo_id": "file",
"pulp3_distribution_hrefs": [],
"pulp3_publication_href": [],
"pulp3_remote_href": "/pulp/api/v3/remotes/file/file/f0bff6eb-5b5d-4c0e-9c58-f37cab8b746e/",
"pulp3_repository_version": "/pulp/api/v3/repositories/file/file/1027c158-9d45-4795-8535-2d6c7812e042/versions/1/",
"pulp_created": "2020-01-13T18:36:45.314986Z",
"pulp_href": "/pulp/api/v3/pulp2repositories/0311514a-d099-4567-8217-7fc409387de1/"
},
{
"is_migrated": true,
"not_in_pulp2": false,
"pulp2_object_id": "5e1c932ec998ac367cd60d85",
"pulp2_repo_id": "file2",
"pulp3_distribution_hrefs": [],
"pulp3_publication_href": [],
"pulp3_remote_href": "/pulp/api/v3/remotes/file/file/a707774a-e364-4ac6-b045-930e101bb5bd/",
"pulp3_repository_version": "/pulp/api/v3/repositories/file/file/5e9d0729-8d45-47a5-8839-9654a4589b51/versions/1/",
"pulp_created": "2020-01-13T18:36:45.255291Z",
"pulp_href": "/pulp/api/v3/pulp2repositories/74afef49-5c36-4361-a672-86ec59cd2065/"
}
]
}
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http POST :24817/pulp/api/v3/migration-plans/ plan='{"plugins": [{"type": "iso"}]}'
HTTP/1.1 201 Created
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 163
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:36:59 GMT
Location: /pulp/api/v3/migration-plans/6b9d4a5d-6c84-41d7-8888-28dc5ba81e50/
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"plan": {
"plugins": [
{
"type": "iso"
}
]
},
"pulp_created": "2020-01-13T18:36:59.486540Z",
"pulp_href": "/pulp/api/v3/migration-plans/6b9d4a5d-6c84-41d7-8888-28dc5ba81e50/"
}
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http POST :24817/pulp/api/v3/migration-plans/6b9d4a5d-6c84-41d7-8888-28dc5ba81e50/run/
HTTP/1.1 202 Accepted
Allow: POST, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:37:09 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"task": "/pulp/api/v3/tasks/42f32246-d8a9-4d95-9643-ec92f61f9691/"
}
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http GET :24817/pulp/api/v3/tasks/42f32246-d8a9-4d95-9643-ec92f61f9691/
HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 2131
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:37:32 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"created_resources": [
"/pulp/api/v3/repositories/file/file/1fa3ead3-61c8-4288-bafb-565a7023a7c7/versions/1/",
"/pulp/api/v3/publications/file/file/2d813243-5088-4d1a-a27e-cd45357d80fb/",
"/pulp/api/v3/distributions/file/file/888268c7-0d17-4536-8e3b-6fad4dd133f1/",
"/pulp/api/v3/publications/file/file/28012224-b540-459c-a717-b2c9984a7adc/",
"/pulp/api/v3/distributions/file/file/267bdcf4-5767-4790-9128-db3ae37684f6/",
"/pulp/api/v3/publications/file/file/08f24cbf-ee99-406c-b639-ef47f36518df/",
"/pulp/api/v3/distributions/file/file/a6afaa7b-070b-4442-9587-55d4bdc1904f/"
],
"error": null,
"finished_at": "2020-01-13T18:37:25.590227Z",
"name": "pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
"progress_reports": [
{
"code": "premigrating.repositories",
"done": 2,
"message": "Pre-migrating Pulp 2 repositories, importers, distributors",
"state": "completed",
"suffix": null,
"total": 2
},
{
"code": "creating.repositories",
"done": 1,
"message": "Creating repositories in Pulp 3",
"state": "completed",
"suffix": null,
"total": 1
},
{
"code": "migrating.importers",
"done": 1,
"message": "Migrating importers to Pulp 3",
"state": "completed",
"suffix": null,
"total": 1
},
{
"code": "premigrating.content.general",
"done": 0,
"message": "Pre-migrating Pulp 2 ISO content (general info)",
"state": "completed",
"suffix": null,
"total": 0
},
{
"code": "premigrating.content.detail",
"done": 0,
"message": "Pre-migrating Pulp 2 ISO content (detail info)",
"state": "completed",
"suffix": null,
"total": 0
},
{
"code": "migrating.content",
"done": 250,
"message": "Migrating content to Pulp 3",
"state": "completed",
"suffix": null,
"total": 250
},
{
"code": "migrating.iso.content",
"done": 250,
"message": "Migrating iso content to Pulp 3 iso",
"state": "completed",
"suffix": null,
"total": 250
},
{
"code": "migrating.distributors",
"done": 3,
"message": "Migrating distributors to Pulp 3",
"state": "completed",
"suffix": null,
"total": 3
}
],
"pulp_created": "2020-01-13T18:37:09.292815Z",
"pulp_href": "/pulp/api/v3/tasks/42f32246-d8a9-4d95-9643-ec92f61f9691/",
"reserved_resources_record": [
"pulp_2to3_migration"
],
"started_at": "2020-01-13T18:37:09.402291Z",
"state": "completed",
"worker": "/pulp/api/v3/workers/c8851912-2998-4c73-8fa8-346e1986a22d/"
}
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http GET :24817/pulp/api/v3/pulp2repositories/
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 2014
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:37:44 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"count": 3,
"next": null,
"previous": null,
"results": [
{
"is_migrated": true,
"not_in_pulp2": false,
"pulp2_object_id": "5e1c932dc998ac367a9b79d0",
"pulp2_repo_id": "file",
"pulp3_distribution_hrefs": [
"/pulp/api/v3/distributions/file/file/267bdcf4-5767-4790-9128-db3ae37684f6/"
],
"pulp3_publication_href": [
"/pulp/api/v3/publications/file/file/28012224-b540-459c-a717-b2c9984a7adc/"
],
"pulp3_remote_href": "/pulp/api/v3/remotes/file/file/f0bff6eb-5b5d-4c0e-9c58-f37cab8b746e/",
"pulp3_repository_version": "/pulp/api/v3/repositories/file/file/1027c158-9d45-4795-8535-2d6c7812e042/versions/1/",
"pulp_created": "2020-01-13T18:36:45.314986Z",
"pulp_href": "/pulp/api/v3/pulp2repositories/0311514a-d099-4567-8217-7fc409387de1/"
},
{
"is_migrated": true,
"not_in_pulp2": false,
"pulp2_object_id": "5e1c932ec998ac367cd60d85",
"pulp2_repo_id": "file2",
"pulp3_distribution_hrefs": [
"/pulp/api/v3/distributions/file/file/a6afaa7b-070b-4442-9587-55d4bdc1904f/"
],
"pulp3_publication_href": [
"/pulp/api/v3/publications/file/file/08f24cbf-ee99-406c-b639-ef47f36518df/"
],
"pulp3_remote_href": "/pulp/api/v3/remotes/file/file/a707774a-e364-4ac6-b045-930e101bb5bd/",
"pulp3_repository_version": "/pulp/api/v3/repositories/file/file/5e9d0729-8d45-47a5-8839-9654a4589b51/versions/1/",
"pulp_created": "2020-01-13T18:36:45.255291Z",
"pulp_href": "/pulp/api/v3/pulp2repositories/74afef49-5c36-4361-a672-86ec59cd2065/"
},
{
"is_migrated": true,
"not_in_pulp2": false,
"pulp2_object_id": "5e1c932ec998ac367cd60d8b",
"pulp2_repo_id": "file-large",
"pulp3_distribution_hrefs": [
"/pulp/api/v3/distributions/file/file/888268c7-0d17-4536-8e3b-6fad4dd133f1/"
],
"pulp3_publication_href": [
"/pulp/api/v3/publications/file/file/2d813243-5088-4d1a-a27e-cd45357d80fb/"
],
"pulp3_remote_href": "/pulp/api/v3/remotes/file/file/a199d649-2102-4a44-a4f1-d802b63ad51d/",
"pulp3_repository_version": "/pulp/api/v3/repositories/file/file/1fa3ead3-61c8-4288-bafb-565a7023a7c7/versions/1/",
"pulp_created": "2020-01-13T18:37:23.214863Z",
"pulp_href": "/pulp/api/v3/pulp2repositories/7a902214-433f-4826-b9e4-5b37a48e029f/"
}
]
}
</pre>
h4. Potential solution
Pre-migration step should not consider the latest timestamp of a migration.
It should check for every pulp 2 repository:
- if it needs to be migrated according to the current MP
- if it was premigrated before and then premigrate or unset is_migrated flag if there were changes on pulp 2 side
h4. Open questions
1. Should all the pulp3 repositories not mentioned in the new MP be removed? (repos are still in pulp 2, they are just not covered by a MP)
2. If no ^, what to do if a new plan has already migrated a repository but with different importer/distributor? Is it allowed?
3. If no ^, is some kind of migration "reset" endpoint needed? (e.g. delete everything you migrated for plugin N)
.