Project

Profile

Help

Issue #5980

Older repositories are not migrated if the newer ones have already been migrated

Added by ttereshc over 1 year ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 64
Quarter:

Description

A migration plan has to be declarative.
The migration result should not be affected by previous attempts with different migration plans.

In the example below, repositories file and file2 are correctly migrated according to the MP.
Then migration plan is changed to "migrate-all" (4 repositories, including file and file2).
In the result only 3 repositories are present. One was not migrated because it has older timestamp than file and file2.

http POST :24817/pulp/api/v3/migration-plans/ plan='{"plugins": [{ "type": "iso", "repositories" :  [{"name": "file","pulp2_importer_repository_id": "file", "repository_versions": [{"pulp2_repository_id": "file"}]}, {"name": "file2","pulp2_importer_repository_id": "file2", "repository_versions": [{"pulp2_repository_id": "file2"}]}]}]}'
HTTP/1.1 201 Created
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 401
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:36:28 GMT
Location: /pulp/api/v3/migration-plans/8b54478b-97aa-4457-8d60-485706e2ca2c/
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "plan": {
        "plugins": [
            {
                "repositories": [
                    {
                        "name": "file",
                        "pulp2_importer_repository_id": "file",
                        "repository_versions": [
                            {
                                "pulp2_repository_id": "file"
                            }
                        ]
                    },
                    {
                        "name": "file2",
                        "pulp2_importer_repository_id": "file2",
                        "repository_versions": [
                            {
                                "pulp2_repository_id": "file2"
                            }
                        ]
                    }
                ],
                "type": "iso"
            }
        ]
    },
    "pulp_created": "2020-01-13T18:36:28.540285Z",
    "pulp_href": "/pulp/api/v3/migration-plans/8b54478b-97aa-4457-8d60-485706e2ca2c/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http POST :24817/pulp/api/v3/migration-plans/8b54478b-97aa-4457-8d60-485706e2ca2c/run/
HTTP/1.1 202 Accepted
Allow: POST, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:36:39 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "task": "/pulp/api/v3/tasks/b14e8b97-0408-4106-a3d0-30850d93b60b/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http GET :24817/pulp/api/v3/pulp2repositories/
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 1054
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:36:54 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "count": 2,
    "next": null,
    "previous": null,
    "results": [
        {
            "is_migrated": true,
            "not_in_pulp2": false,
            "pulp2_object_id": "5e1c932dc998ac367a9b79d0",
            "pulp2_repo_id": "file",
            "pulp3_distribution_hrefs": [],
            "pulp3_publication_href": [],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/f0bff6eb-5b5d-4c0e-9c58-f37cab8b746e/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/1027c158-9d45-4795-8535-2d6c7812e042/versions/1/",
            "pulp_created": "2020-01-13T18:36:45.314986Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/0311514a-d099-4567-8217-7fc409387de1/"
        },
        {
            "is_migrated": true,
            "not_in_pulp2": false,
            "pulp2_object_id": "5e1c932ec998ac367cd60d85",
            "pulp2_repo_id": "file2",
            "pulp3_distribution_hrefs": [],
            "pulp3_publication_href": [],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/a707774a-e364-4ac6-b045-930e101bb5bd/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/5e9d0729-8d45-47a5-8839-9654a4589b51/versions/1/",
            "pulp_created": "2020-01-13T18:36:45.255291Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/74afef49-5c36-4361-a672-86ec59cd2065/"
        }
    ]
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http POST :24817/pulp/api/v3/migration-plans/ plan='{"plugins": [{"type": "iso"}]}'
HTTP/1.1 201 Created
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 163
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:36:59 GMT
Location: /pulp/api/v3/migration-plans/6b9d4a5d-6c84-41d7-8888-28dc5ba81e50/
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "plan": {
        "plugins": [
            {
                "type": "iso"
            }
        ]
    },
    "pulp_created": "2020-01-13T18:36:59.486540Z",
    "pulp_href": "/pulp/api/v3/migration-plans/6b9d4a5d-6c84-41d7-8888-28dc5ba81e50/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http POST :24817/pulp/api/v3/migration-plans/6b9d4a5d-6c84-41d7-8888-28dc5ba81e50/run/
HTTP/1.1 202 Accepted
Allow: POST, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:37:09 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "task": "/pulp/api/v3/tasks/42f32246-d8a9-4d95-9643-ec92f61f9691/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http GET :24817/pulp/api/v3/tasks/42f32246-d8a9-4d95-9643-ec92f61f9691/
HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 2131
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:37:32 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "created_resources": [
        "/pulp/api/v3/repositories/file/file/1fa3ead3-61c8-4288-bafb-565a7023a7c7/versions/1/",
        "/pulp/api/v3/publications/file/file/2d813243-5088-4d1a-a27e-cd45357d80fb/",
        "/pulp/api/v3/distributions/file/file/888268c7-0d17-4536-8e3b-6fad4dd133f1/",
        "/pulp/api/v3/publications/file/file/28012224-b540-459c-a717-b2c9984a7adc/",
        "/pulp/api/v3/distributions/file/file/267bdcf4-5767-4790-9128-db3ae37684f6/",
        "/pulp/api/v3/publications/file/file/08f24cbf-ee99-406c-b639-ef47f36518df/",
        "/pulp/api/v3/distributions/file/file/a6afaa7b-070b-4442-9587-55d4bdc1904f/"
    ],
    "error": null,
    "finished_at": "2020-01-13T18:37:25.590227Z",
    "name": "pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
    "progress_reports": [
        {
            "code": "premigrating.repositories",
            "done": 2,
            "message": "Pre-migrating Pulp 2 repositories, importers, distributors",
            "state": "completed",
            "suffix": null,
            "total": 2
        },
        {
            "code": "creating.repositories",
            "done": 1,
            "message": "Creating repositories in Pulp 3",
            "state": "completed",
            "suffix": null,
            "total": 1
        },
        {
            "code": "migrating.importers",
            "done": 1,
            "message": "Migrating importers to Pulp 3",
            "state": "completed",
            "suffix": null,
            "total": 1
        },
        {
            "code": "premigrating.content.general",
            "done": 0,
            "message": "Pre-migrating Pulp 2 ISO content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 0
        },
        {
            "code": "premigrating.content.detail",
            "done": 0,
            "message": "Pre-migrating Pulp 2 ISO content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 0
        },
        {
            "code": "migrating.content",
            "done": 250,
            "message": "Migrating content to Pulp 3",
            "state": "completed",
            "suffix": null,
            "total": 250
        },
        {
            "code": "migrating.iso.content",
            "done": 250,
            "message": "Migrating iso content to Pulp 3 iso",
            "state": "completed",
            "suffix": null,
            "total": 250
        },
        {
            "code": "migrating.distributors",
            "done": 3,
            "message": "Migrating distributors to Pulp 3",
            "state": "completed",
            "suffix": null,
            "total": 3
        }
    ],
    "pulp_created": "2020-01-13T18:37:09.292815Z",
    "pulp_href": "/pulp/api/v3/tasks/42f32246-d8a9-4d95-9643-ec92f61f9691/",
    "reserved_resources_record": [
        "pulp_2to3_migration"
    ],
    "started_at": "2020-01-13T18:37:09.402291Z",
    "state": "completed",
    "worker": "/pulp/api/v3/workers/c8851912-2998-4c73-8fa8-346e1986a22d/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http GET :24817/pulp/api/v3/pulp2repositories/
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 2014
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:37:44 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "count": 3,
    "next": null,
    "previous": null,
    "results": [
        {
            "is_migrated": true,
            "not_in_pulp2": false,
            "pulp2_object_id": "5e1c932dc998ac367a9b79d0",
            "pulp2_repo_id": "file",
            "pulp3_distribution_hrefs": [
                "/pulp/api/v3/distributions/file/file/267bdcf4-5767-4790-9128-db3ae37684f6/"
            ],
            "pulp3_publication_href": [
                "/pulp/api/v3/publications/file/file/28012224-b540-459c-a717-b2c9984a7adc/"
            ],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/f0bff6eb-5b5d-4c0e-9c58-f37cab8b746e/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/1027c158-9d45-4795-8535-2d6c7812e042/versions/1/",
            "pulp_created": "2020-01-13T18:36:45.314986Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/0311514a-d099-4567-8217-7fc409387de1/"
        },
        {
            "is_migrated": true,
            "not_in_pulp2": false,
            "pulp2_object_id": "5e1c932ec998ac367cd60d85",
            "pulp2_repo_id": "file2",
            "pulp3_distribution_hrefs": [
                "/pulp/api/v3/distributions/file/file/a6afaa7b-070b-4442-9587-55d4bdc1904f/"
            ],
            "pulp3_publication_href": [
                "/pulp/api/v3/publications/file/file/08f24cbf-ee99-406c-b639-ef47f36518df/"
            ],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/a707774a-e364-4ac6-b045-930e101bb5bd/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/5e9d0729-8d45-47a5-8839-9654a4589b51/versions/1/",
            "pulp_created": "2020-01-13T18:36:45.255291Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/74afef49-5c36-4361-a672-86ec59cd2065/"
        },
        {
            "is_migrated": true,
            "not_in_pulp2": false,
            "pulp2_object_id": "5e1c932ec998ac367cd60d8b",
            "pulp2_repo_id": "file-large",
            "pulp3_distribution_hrefs": [
                "/pulp/api/v3/distributions/file/file/888268c7-0d17-4536-8e3b-6fad4dd133f1/"
            ],
            "pulp3_publication_href": [
                "/pulp/api/v3/publications/file/file/2d813243-5088-4d1a-a27e-cd45357d80fb/"
            ],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/a199d649-2102-4a44-a4f1-d802b63ad51d/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/1fa3ead3-61c8-4288-bafb-565a7023a7c7/versions/1/",
            "pulp_created": "2020-01-13T18:37:23.214863Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/7a902214-433f-4826-b9e4-5b37a48e029f/"
        }
    ]
}

Potential solution

Pre-migration step should not consider the latest timestamp of a migration.
It should check for every pulp 2 repository:
- if it needs to be migrated according to the current MP
- if it was premigrated before and then premigrate or unset is_migrated flag if there were changes on pulp 2 side

Open questions

1. Should all the pulp3 repositories not mentioned in the new MP be removed? (repos are still in pulp 2, they are just not covered by a MP)
2. If no ^, what to do if a new plan has already migrated a repository but with different importer/distributor? Is it allowed?
3. If no ^, is some kind of migration "reset" endpoint needed? (e.g. delete everything you migrated for plugin N)

Associated revisions

Revision 80971584 View on GitHub
Added by ttereshc over 1 year ago

Fully migrate importers and distributors on every run

Also inspect all importers, distributors in case some of them are requested to be migrated without a repository. Make sure RemoteArtifacts are created for already existing content when old Remotes are removed.

Migrations are squashed due to changes in the importer/distributor models.

closes #5980 https://pulp.plan.io/issues/5980

closes #5852 https://pulp.plan.io/issues/5852

Revision 80971584 View on GitHub
Added by ttereshc over 1 year ago

Fully migrate importers and distributors on every run

Also inspect all importers, distributors in case some of them are requested to be migrated without a repository. Make sure RemoteArtifacts are created for already existing content when old Remotes are removed.

Migrations are squashed due to changes in the importer/distributor models.

closes #5980 https://pulp.plan.io/issues/5980

closes #5852 https://pulp.plan.io/issues/5852

Revision 80971584 View on GitHub
Added by ttereshc over 1 year ago

Fully migrate importers and distributors on every run

Also inspect all importers, distributors in case some of them are requested to be migrated without a repository. Make sure RemoteArtifacts are created for already existing content when old Remotes are removed.

Migrations are squashed due to changes in the importer/distributor models.

closes #5980 https://pulp.plan.io/issues/5980

closes #5852 https://pulp.plan.io/issues/5852

History

#1 Updated by ttereshc over 1 year ago

  • Project changed from RPM Support to Pulp

#2 Updated by ttereshc over 1 year ago

  • Description updated (diff)

#3 Updated by ttereshc over 1 year ago

  • Description updated (diff)

#4 Updated by ttereshc over 1 year ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 64

#5 Updated by ttereshc over 1 year ago

  • Project changed from Pulp to Migration Plugin

#6 Updated by ttereshc over 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ttereshc

#7 Updated by ttereshc over 1 year ago

  • Status changed from ASSIGNED to POST

#8 Updated by ttereshc over 1 year ago

  • Status changed from POST to MODIFIED

#11 Updated by ttereshc over 1 year ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
  • Sprint/Milestone set to 0.1.0

#12 Updated by bmbouter over 1 year ago

  • Tags deleted (Pulp 2 to 3 Migrate)

We're removing the 'Pulp 2 to 3 Migrate' tag to allow membership of the Migration Plugin project in Redmine to be the controlling area https://pulp.plan.io/projects/migration

Please register to edit this issue

Also available in: Atom PDF