Migration Plugin

A migration plan has to be declarative.
The migration result should not be affected by previous attempts with different migration plans.

In the example below, repositories file and file2 are correctly migrated according to the MP.
Then migration plan is changed to "migrate-all" (4 repositories, including file and file2).
In the result only 3 repositories are present. One was not migrated because it has older timestamp than file and file2.

http POST :24817/pulp/api/v3/migration-plans/ plan='{"plugins": [{ "type": "iso", "repositories" :  [{"name": "file","pulp2_importer_repository_id": "file", "repository_versions": [{"pulp2_repository_id": "file"}]}, {"name": "file2","pulp2_importer_repository_id": "file2", "repository_versions": [{"pulp2_repository_id": "file2"}]}]}]}'
HTTP/1.1 201 Created
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 401
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:36:28 GMT
Location: /pulp/api/v3/migration-plans/8b54478b-97aa-4457-8d60-485706e2ca2c/
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "plan": {
        "plugins": [
            {
                "repositories": [
                    {
                        "name": "file",
                        "pulp2_importer_repository_id": "file",
                        "repository_versions": [
                            {
                                "pulp2_repository_id": "file"
                            }
                        ]
                    },
                    {
                        "name": "file2",
                        "pulp2_importer_repository_id": "file2",
                        "repository_versions": [
                            {
                                "pulp2_repository_id": "file2"
                            }
                        ]
                    }
                ],
                "type": "iso"
            }
        ]
    },
    "pulp_created": "2020-01-13T18:36:28.540285Z",
    "pulp_href": "/pulp/api/v3/migration-plans/8b54478b-97aa-4457-8d60-485706e2ca2c/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http POST :24817/pulp/api/v3/migration-plans/8b54478b-97aa-4457-8d60-485706e2ca2c/run/
HTTP/1.1 202 Accepted
Allow: POST, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:36:39 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "task": "/pulp/api/v3/tasks/b14e8b97-0408-4106-a3d0-30850d93b60b/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http GET :24817/pulp/api/v3/pulp2repositories/
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 1054
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:36:54 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "count": 2,
    "next": null,
    "previous": null,
    "results": [
        {
            "is_migrated": true,
            "not_in_pulp2": false,
            "pulp2_object_id": "5e1c932dc998ac367a9b79d0",
            "pulp2_repo_id": "file",
            "pulp3_distribution_hrefs": [],
            "pulp3_publication_href": [],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/f0bff6eb-5b5d-4c0e-9c58-f37cab8b746e/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/1027c158-9d45-4795-8535-2d6c7812e042/versions/1/",
            "pulp_created": "2020-01-13T18:36:45.314986Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/0311514a-d099-4567-8217-7fc409387de1/"
        },
        {
            "is_migrated": true,
            "not_in_pulp2": false,
            "pulp2_object_id": "5e1c932ec998ac367cd60d85",
            "pulp2_repo_id": "file2",
            "pulp3_distribution_hrefs": [],
            "pulp3_publication_href": [],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/a707774a-e364-4ac6-b045-930e101bb5bd/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/5e9d0729-8d45-47a5-8839-9654a4589b51/versions/1/",
            "pulp_created": "2020-01-13T18:36:45.255291Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/74afef49-5c36-4361-a672-86ec59cd2065/"
        }
    ]
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http POST :24817/pulp/api/v3/migration-plans/ plan='{"plugins": [{"type": "iso"}]}'
HTTP/1.1 201 Created
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 163
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:36:59 GMT
Location: /pulp/api/v3/migration-plans/6b9d4a5d-6c84-41d7-8888-28dc5ba81e50/
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "plan": {
        "plugins": [
            {
                "type": "iso"
            }
        ]
    },
    "pulp_created": "2020-01-13T18:36:59.486540Z",
    "pulp_href": "/pulp/api/v3/migration-plans/6b9d4a5d-6c84-41d7-8888-28dc5ba81e50/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http POST :24817/pulp/api/v3/migration-plans/6b9d4a5d-6c84-41d7-8888-28dc5ba81e50/run/
HTTP/1.1 202 Accepted
Allow: POST, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:37:09 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "task": "/pulp/api/v3/tasks/42f32246-d8a9-4d95-9643-ec92f61f9691/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http GET :24817/pulp/api/v3/tasks/42f32246-d8a9-4d95-9643-ec92f61f9691/
HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 2131
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:37:32 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "created_resources": [
        "/pulp/api/v3/repositories/file/file/1fa3ead3-61c8-4288-bafb-565a7023a7c7/versions/1/",
        "/pulp/api/v3/publications/file/file/2d813243-5088-4d1a-a27e-cd45357d80fb/",
        "/pulp/api/v3/distributions/file/file/888268c7-0d17-4536-8e3b-6fad4dd133f1/",
        "/pulp/api/v3/publications/file/file/28012224-b540-459c-a717-b2c9984a7adc/",
        "/pulp/api/v3/distributions/file/file/267bdcf4-5767-4790-9128-db3ae37684f6/",
        "/pulp/api/v3/publications/file/file/08f24cbf-ee99-406c-b639-ef47f36518df/",
        "/pulp/api/v3/distributions/file/file/a6afaa7b-070b-4442-9587-55d4bdc1904f/"
    ],
    "error": null,
    "finished_at": "2020-01-13T18:37:25.590227Z",
    "name": "pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
    "progress_reports": [
        {
            "code": "premigrating.repositories",
            "done": 2,
            "message": "Pre-migrating Pulp 2 repositories, importers, distributors",
            "state": "completed",
            "suffix": null,
            "total": 2
        },
        {
            "code": "creating.repositories",
            "done": 1,
            "message": "Creating repositories in Pulp 3",
            "state": "completed",
            "suffix": null,
            "total": 1
        },
        {
            "code": "migrating.importers",
            "done": 1,
            "message": "Migrating importers to Pulp 3",
            "state": "completed",
            "suffix": null,
            "total": 1
        },
        {
            "code": "premigrating.content.general",
            "done": 0,
            "message": "Pre-migrating Pulp 2 ISO content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 0
        },
        {
            "code": "premigrating.content.detail",
            "done": 0,
            "message": "Pre-migrating Pulp 2 ISO content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 0
        },
        {
            "code": "migrating.content",
            "done": 250,
            "message": "Migrating content to Pulp 3",
            "state": "completed",
            "suffix": null,
            "total": 250
        },
        {
            "code": "migrating.iso.content",
            "done": 250,
            "message": "Migrating iso content to Pulp 3 iso",
            "state": "completed",
            "suffix": null,
            "total": 250
        },
        {
            "code": "migrating.distributors",
            "done": 3,
            "message": "Migrating distributors to Pulp 3",
            "state": "completed",
            "suffix": null,
            "total": 3
        }
    ],
    "pulp_created": "2020-01-13T18:37:09.292815Z",
    "pulp_href": "/pulp/api/v3/tasks/42f32246-d8a9-4d95-9643-ec92f61f9691/",
    "reserved_resources_record": [
        "pulp_2to3_migration"
    ],
    "started_at": "2020-01-13T18:37:09.402291Z",
    "state": "completed",
    "worker": "/pulp/api/v3/workers/c8851912-2998-4c73-8fa8-346e1986a22d/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http GET :24817/pulp/api/v3/pulp2repositories/
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 2014
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:37:44 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "count": 3,
    "next": null,
    "previous": null,
    "results": [
        {
            "is_migrated": true,
            "not_in_pulp2": false,
            "pulp2_object_id": "5e1c932dc998ac367a9b79d0",
            "pulp2_repo_id": "file",
            "pulp3_distribution_hrefs": [
                "/pulp/api/v3/distributions/file/file/267bdcf4-5767-4790-9128-db3ae37684f6/"
            ],
            "pulp3_publication_href": [
                "/pulp/api/v3/publications/file/file/28012224-b540-459c-a717-b2c9984a7adc/"
            ],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/f0bff6eb-5b5d-4c0e-9c58-f37cab8b746e/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/1027c158-9d45-4795-8535-2d6c7812e042/versions/1/",
            "pulp_created": "2020-01-13T18:36:45.314986Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/0311514a-d099-4567-8217-7fc409387de1/"
        },
        {
            "is_migrated": true,
            "not_in_pulp2": false,
            "pulp2_object_id": "5e1c932ec998ac367cd60d85",
            "pulp2_repo_id": "file2",
            "pulp3_distribution_hrefs": [
                "/pulp/api/v3/distributions/file/file/a6afaa7b-070b-4442-9587-55d4bdc1904f/"
            ],
            "pulp3_publication_href": [
                "/pulp/api/v3/publications/file/file/08f24cbf-ee99-406c-b639-ef47f36518df/"
            ],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/a707774a-e364-4ac6-b045-930e101bb5bd/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/5e9d0729-8d45-47a5-8839-9654a4589b51/versions/1/",
            "pulp_created": "2020-01-13T18:36:45.255291Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/74afef49-5c36-4361-a672-86ec59cd2065/"
        },
        {
            "is_migrated": true,
            "not_in_pulp2": false,
            "pulp2_object_id": "5e1c932ec998ac367cd60d8b",
            "pulp2_repo_id": "file-large",
            "pulp3_distribution_hrefs": [
                "/pulp/api/v3/distributions/file/file/888268c7-0d17-4536-8e3b-6fad4dd133f1/"
            ],
            "pulp3_publication_href": [
                "/pulp/api/v3/publications/file/file/2d813243-5088-4d1a-a27e-cd45357d80fb/"
            ],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/a199d649-2102-4a44-a4f1-d802b63ad51d/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/1fa3ead3-61c8-4288-bafb-565a7023a7c7/versions/1/",
            "pulp_created": "2020-01-13T18:37:23.214863Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/7a902214-433f-4826-b9e4-5b37a48e029f/"
        }
    ]
}

Potential solution¶

Pre-migration step should not consider the latest timestamp of a migration.
It should check for every pulp 2 repository:
- if it needs to be migrated according to the current MP
- if it was premigrated before and then premigrate or unset is_migrated flag if there were changes on pulp 2 side

Open questions¶

1. Should all the pulp3 repositories not mentioned in the new MP be removed? (repos are still in pulp 2, they are just not covered by a MP)
2. If no ^, what to do if a new plan has already migrated a repository but with different importer/distributor? Is it allowed?
3. If no ^, is some kind of migration "reset" endpoint needed? (e.g. delete everything you migrated for plugin N)