Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #5980

closed

Older repositories are not migrated if the newer ones have already been migrated

Added by ttereshc almost 5 years ago. Updated over 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
ttereshc
Sprint/Milestone:
0.1.0
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 64
Quarter:

Description

A migration plan has to be declarative.
The migration result should not be affected by previous attempts with different migration plans.

In the example below, repositories file and file2 are correctly migrated according to the MP.
Then migration plan is changed to "migrate-all" (4 repositories, including file and file2).
In the result only 3 repositories are present. One was not migrated because it has older timestamp than file and file2.

http POST :24817/pulp/api/v3/migration-plans/ plan='{"plugins": [{ "type": "iso", "repositories" :  [{"name": "file","pulp2_importer_repository_id": "file", "repository_versions": [{"pulp2_repository_id": "file"}]}, {"name": "file2","pulp2_importer_repository_id": "file2", "repository_versions": [{"pulp2_repository_id": "file2"}]}]}]}'
HTTP/1.1 201 Created
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 401
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:36:28 GMT
Location: /pulp/api/v3/migration-plans/8b54478b-97aa-4457-8d60-485706e2ca2c/
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "plan": {
        "plugins": [
            {
                "repositories": [
                    {
                        "name": "file",
                        "pulp2_importer_repository_id": "file",
                        "repository_versions": [
                            {
                                "pulp2_repository_id": "file"
                            }
                        ]
                    },
                    {
                        "name": "file2",
                        "pulp2_importer_repository_id": "file2",
                        "repository_versions": [
                            {
                                "pulp2_repository_id": "file2"
                            }
                        ]
                    }
                ],
                "type": "iso"
            }
        ]
    },
    "pulp_created": "2020-01-13T18:36:28.540285Z",
    "pulp_href": "/pulp/api/v3/migration-plans/8b54478b-97aa-4457-8d60-485706e2ca2c/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http POST :24817/pulp/api/v3/migration-plans/8b54478b-97aa-4457-8d60-485706e2ca2c/run/
HTTP/1.1 202 Accepted
Allow: POST, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:36:39 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "task": "/pulp/api/v3/tasks/b14e8b97-0408-4106-a3d0-30850d93b60b/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http GET :24817/pulp/api/v3/pulp2repositories/
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 1054
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:36:54 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "count": 2,
    "next": null,
    "previous": null,
    "results": [
        {
            "is_migrated": true,
            "not_in_pulp2": false,
            "pulp2_object_id": "5e1c932dc998ac367a9b79d0",
            "pulp2_repo_id": "file",
            "pulp3_distribution_hrefs": [],
            "pulp3_publication_href": [],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/f0bff6eb-5b5d-4c0e-9c58-f37cab8b746e/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/1027c158-9d45-4795-8535-2d6c7812e042/versions/1/",
            "pulp_created": "2020-01-13T18:36:45.314986Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/0311514a-d099-4567-8217-7fc409387de1/"
        },
        {
            "is_migrated": true,
            "not_in_pulp2": false,
            "pulp2_object_id": "5e1c932ec998ac367cd60d85",
            "pulp2_repo_id": "file2",
            "pulp3_distribution_hrefs": [],
            "pulp3_publication_href": [],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/a707774a-e364-4ac6-b045-930e101bb5bd/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/5e9d0729-8d45-47a5-8839-9654a4589b51/versions/1/",
            "pulp_created": "2020-01-13T18:36:45.255291Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/74afef49-5c36-4361-a672-86ec59cd2065/"
        }
    ]
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http POST :24817/pulp/api/v3/migration-plans/ plan='{"plugins": [{"type": "iso"}]}'
HTTP/1.1 201 Created
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 163
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:36:59 GMT
Location: /pulp/api/v3/migration-plans/6b9d4a5d-6c84-41d7-8888-28dc5ba81e50/
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "plan": {
        "plugins": [
            {
                "type": "iso"
            }
        ]
    },
    "pulp_created": "2020-01-13T18:36:59.486540Z",
    "pulp_href": "/pulp/api/v3/migration-plans/6b9d4a5d-6c84-41d7-8888-28dc5ba81e50/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http POST :24817/pulp/api/v3/migration-plans/6b9d4a5d-6c84-41d7-8888-28dc5ba81e50/run/
HTTP/1.1 202 Accepted
Allow: POST, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:37:09 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "task": "/pulp/api/v3/tasks/42f32246-d8a9-4d95-9643-ec92f61f9691/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http GET :24817/pulp/api/v3/tasks/42f32246-d8a9-4d95-9643-ec92f61f9691/
HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 2131
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:37:32 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "created_resources": [
        "/pulp/api/v3/repositories/file/file/1fa3ead3-61c8-4288-bafb-565a7023a7c7/versions/1/",
        "/pulp/api/v3/publications/file/file/2d813243-5088-4d1a-a27e-cd45357d80fb/",
        "/pulp/api/v3/distributions/file/file/888268c7-0d17-4536-8e3b-6fad4dd133f1/",
        "/pulp/api/v3/publications/file/file/28012224-b540-459c-a717-b2c9984a7adc/",
        "/pulp/api/v3/distributions/file/file/267bdcf4-5767-4790-9128-db3ae37684f6/",
        "/pulp/api/v3/publications/file/file/08f24cbf-ee99-406c-b639-ef47f36518df/",
        "/pulp/api/v3/distributions/file/file/a6afaa7b-070b-4442-9587-55d4bdc1904f/"
    ],
    "error": null,
    "finished_at": "2020-01-13T18:37:25.590227Z",
    "name": "pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
    "progress_reports": [
        {
            "code": "premigrating.repositories",
            "done": 2,
            "message": "Pre-migrating Pulp 2 repositories, importers, distributors",
            "state": "completed",
            "suffix": null,
            "total": 2
        },
        {
            "code": "creating.repositories",
            "done": 1,
            "message": "Creating repositories in Pulp 3",
            "state": "completed",
            "suffix": null,
            "total": 1
        },
        {
            "code": "migrating.importers",
            "done": 1,
            "message": "Migrating importers to Pulp 3",
            "state": "completed",
            "suffix": null,
            "total": 1
        },
        {
            "code": "premigrating.content.general",
            "done": 0,
            "message": "Pre-migrating Pulp 2 ISO content (general info)",
            "state": "completed",
            "suffix": null,
            "total": 0
        },
        {
            "code": "premigrating.content.detail",
            "done": 0,
            "message": "Pre-migrating Pulp 2 ISO content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 0
        },
        {
            "code": "migrating.content",
            "done": 250,
            "message": "Migrating content to Pulp 3",
            "state": "completed",
            "suffix": null,
            "total": 250
        },
        {
            "code": "migrating.iso.content",
            "done": 250,
            "message": "Migrating iso content to Pulp 3 iso",
            "state": "completed",
            "suffix": null,
            "total": 250
        },
        {
            "code": "migrating.distributors",
            "done": 3,
            "message": "Migrating distributors to Pulp 3",
            "state": "completed",
            "suffix": null,
            "total": 3
        }
    ],
    "pulp_created": "2020-01-13T18:37:09.292815Z",
    "pulp_href": "/pulp/api/v3/tasks/42f32246-d8a9-4d95-9643-ec92f61f9691/",
    "reserved_resources_record": [
        "pulp_2to3_migration"
    ],
    "started_at": "2020-01-13T18:37:09.402291Z",
    "state": "completed",
    "worker": "/pulp/api/v3/workers/c8851912-2998-4c73-8fa8-346e1986a22d/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ http GET :24817/pulp/api/v3/pulp2repositories/
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 2014
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:37:44 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "count": 3,
    "next": null,
    "previous": null,
    "results": [
        {
            "is_migrated": true,
            "not_in_pulp2": false,
            "pulp2_object_id": "5e1c932dc998ac367a9b79d0",
            "pulp2_repo_id": "file",
            "pulp3_distribution_hrefs": [
                "/pulp/api/v3/distributions/file/file/267bdcf4-5767-4790-9128-db3ae37684f6/"
            ],
            "pulp3_publication_href": [
                "/pulp/api/v3/publications/file/file/28012224-b540-459c-a717-b2c9984a7adc/"
            ],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/f0bff6eb-5b5d-4c0e-9c58-f37cab8b746e/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/1027c158-9d45-4795-8535-2d6c7812e042/versions/1/",
            "pulp_created": "2020-01-13T18:36:45.314986Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/0311514a-d099-4567-8217-7fc409387de1/"
        },
        {
            "is_migrated": true,
            "not_in_pulp2": false,
            "pulp2_object_id": "5e1c932ec998ac367cd60d85",
            "pulp2_repo_id": "file2",
            "pulp3_distribution_hrefs": [
                "/pulp/api/v3/distributions/file/file/a6afaa7b-070b-4442-9587-55d4bdc1904f/"
            ],
            "pulp3_publication_href": [
                "/pulp/api/v3/publications/file/file/08f24cbf-ee99-406c-b639-ef47f36518df/"
            ],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/a707774a-e364-4ac6-b045-930e101bb5bd/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/5e9d0729-8d45-47a5-8839-9654a4589b51/versions/1/",
            "pulp_created": "2020-01-13T18:36:45.255291Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/74afef49-5c36-4361-a672-86ec59cd2065/"
        },
        {
            "is_migrated": true,
            "not_in_pulp2": false,
            "pulp2_object_id": "5e1c932ec998ac367cd60d8b",
            "pulp2_repo_id": "file-large",
            "pulp3_distribution_hrefs": [
                "/pulp/api/v3/distributions/file/file/888268c7-0d17-4536-8e3b-6fad4dd133f1/"
            ],
            "pulp3_publication_href": [
                "/pulp/api/v3/publications/file/file/2d813243-5088-4d1a-a27e-cd45357d80fb/"
            ],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/a199d649-2102-4a44-a4f1-d802b63ad51d/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/1fa3ead3-61c8-4288-bafb-565a7023a7c7/versions/1/",
            "pulp_created": "2020-01-13T18:37:23.214863Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/7a902214-433f-4826-b9e4-5b37a48e029f/"
        }
    ]
}

Potential solution

Pre-migration step should not consider the latest timestamp of a migration.
It should check for every pulp 2 repository:
- if it needs to be migrated according to the current MP
- if it was premigrated before and then premigrate or unset is_migrated flag if there were changes on pulp 2 side

Open questions

1. Should all the pulp3 repositories not mentioned in the new MP be removed? (repos are still in pulp 2, they are just not covered by a MP)
2. If no ^, what to do if a new plan has already migrated a repository but with different importer/distributor? Is it allowed?
3. If no ^, is some kind of migration "reset" endpoint needed? (e.g. delete everything you migrated for plugin N)

Actions
Copy link
 #1

Updated by ttereshc almost 5 years ago

  • Project changed from RPM Support to Pulp
Actions
Copy link
 #2

Updated by ttereshc almost 5 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by ttereshc almost 5 years ago

  • Description updated (diff)
Actions
Copy link
 #4

Updated by ttereshc almost 5 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 64
Actions
Copy link
 #5

Updated by ttereshc almost 5 years ago

  • Project changed from Pulp to Migration Plugin
Actions
Copy link
 #6

Updated by ttereshc almost 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ttereshc
Actions
Copy link
 #7

Updated by ttereshc almost 5 years ago

  • Status changed from ASSIGNED to POST

Added by ttereshc almost 5 years ago

Revision 80971584 | View on GitHub

Fully migrate importers and distributors on every run

Also inspect all importers, distributors in case some of them are requested to be migrated without a repository. Make sure RemoteArtifacts are created for already existing content when old Remotes are removed.

Migrations are squashed due to changes in the importer/distributor models.

closes #5980 https://pulp.plan.io/issues/5980

closes #5852 https://pulp.plan.io/issues/5852

Added by ttereshc almost 5 years ago

Revision 80971584 | View on GitHub

Fully migrate importers and distributors on every run

Also inspect all importers, distributors in case some of them are requested to be migrated without a repository. Make sure RemoteArtifacts are created for already existing content when old Remotes are removed.

Migrations are squashed due to changes in the importer/distributor models.

closes #5980 https://pulp.plan.io/issues/5980

closes #5852 https://pulp.plan.io/issues/5852

Added by ttereshc almost 5 years ago

Revision 80971584 | View on GitHub

Fully migrate importers and distributors on every run

Also inspect all importers, distributors in case some of them are requested to be migrated without a repository. Make sure RemoteArtifacts are created for already existing content when old Remotes are removed.

Migrations are squashed due to changes in the importer/distributor models.

closes #5980 https://pulp.plan.io/issues/5980

closes #5852 https://pulp.plan.io/issues/5852

Actions
Copy link
 #8

Updated by ttereshc almost 5 years ago

  • Status changed from POST to MODIFIED
Actions
Copy link
 #11

Updated by ttereshc over 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
  • Sprint/Milestone set to 0.1.0
Actions
Copy link
 #12

Updated by bmbouter over 4 years ago

  • Tags deleted (Pulp 2 to 3 Migrate)

We're removing the 'Pulp 2 to 3 Migrate' tag to allow membership of the Migration Plugin project in Redmine to be the controlling area https://pulp.plan.io/projects/migration

Actions
Send by e-mail Copy link

Also available in: Atom PDF