Project

Profile

Help

Story #3821

As a user, I can migrate from Pulp 2 to Pulp 3

Added by dkliban@redhat.com about 2 years ago. Updated 16 days ago.

Status:
NEW
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
07/05/2019
Due date:
07/12/2019
% Done:

93%

Estimated time:
(Total: 4:00 h)
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:

Description

Description

Pulp 3 will provide a plugin that will enable users to migrate their Pulp 2 instances to Pulp 3. The package will be called pulp-2to3-migration. Plugin writers will contribute to it to provide support of migration of the content types managed by that plugin.

After installing pulp-2to3-migration on a Pulp 3 system, users will be able to use the REST API on Pulp 3 to start the migration of the Pulp 2 instance to Pulp 3. Users will be able to describe the final state of their Pulp 3 instance for a specific plugin using a Migration Plan (MP). The MP is a JSON document that describes which artifacts, content, repositories, repository versions, publication, remotes, and distributions get created in Pulp 3 from their equivalent counterparts in Pulp 2. Users will be able to use Pulp 2's REST API to generate a default MP for migrating everything. Users will then submit the MP via REST API to Pulp 3.

After the migration task start running in Pulp 3, the user will be able to use the REST API to query the mappings between Pulp 2 and Pulp 3 resources.

Features

Pulp 2 will provide a REST API that returns a default Migration Plan.

This Migration Plan will migrate all content, all repositories, all importers, and all distributors.

Pulp 3 will provide a REST API that:

- creates a Migration Plan
- executes a Migration plan in one of 3 modes (run the migration without validation, run migration if MP is validated, run validation only - dry-run).

This API will dispatch one or more tasks to perform the migration. The tasking system will limit one migration "operation" at a time, although many tasks may run concurrently for that operation.

When dry_run is set to True, the Migration Plan validated and some information about the plan is returned to the user. Some validation that will be performed will include:

  • JSON syntax
  • do the listed Pulp 2 entities exist
    If all validation passes, the user receives a list of Content types and Repositories from Pulp 2 that will be used to create content and repositories in Pulp 3.

Pulp 3 will provide multiple REST APIs for querying mappings between Pulp 2 and Pulp 3.

One for content, one for repository and related resources - importers, distributors.

Migration Plan schema

The Migration Plan (MP) will consist of the plugin type and repositories specification for each plugin. The Repositories section will describe Repositories, RepositoryVersions, Publications, and Distributions to create.

Examples

The MP below will:

  • migrate all content for ISO and Docker plugins
  • create a repository with name 'foo' in Pulp 3.
  • create a Remote in Pulp 3 based on details of an importer for the repo "my-repo-id-for-importer" in Pulp 2
  • create one repository version that will have the same content set as the ISO 'my-repo-id' repo in Pulp 2
  • implicitly create a Publication from this repository version
  • create 2 distributions based on details of distributors in Pulp 2
  • both of these distributions will serve the same publication.
 {
    "plugins": [
     { 
         "type": "iso", 
         "repositories" :  [
           {
             "name": "foo",
             "pulp2_importer_repository_id":  "my-repo-id-for-importer" 
             "repository_versions": [
                 {
                   "pulp2_repository_id": "my-repo-id",
                   "distributor_ids":  ["distributor_id1", "distributor_id2"]
                  },          
               ]
            }
        ]
     },
     {
       "type": "docker",
     }
  ]
}

The MP below will:

  • migrate all ISO content
  • implicitly create a repository with a single repository version in Pulp 3 for every repository in Pulp 2
  • the content set of the repository version will match the content of the repository in Pulp 2
  • implicitly create a Publication for all repositories that need publications
  • implicitly create a Remote for every importer associated with the repository
  • implicitly create a Distribution for every non-export distributor associated with the repository and associate either the RepositoryVersion or Publication with that Distribution
{
    "plugins": [
       { 
         "type": "iso",
       }
     ]
}

Subtasks

Story #2962: As an User I would like to migrate in-place from Pulp2 to Pulp3 NEW

Actions
Story #3822: As a user, I can migrate all content units of a specific type from Pulp 2 to Pulp 3CLOSED - CURRENTRELEASE

Actions
Story #3810: As a user, I can migrate ISO content units from Pulp 2 into Pulp 3 as File ContentCLOSED - CURRENTRELEASEttereshc

Actions
Task #4205: bootstrap pulp-2to3-migrate django appCLOSED - COMPLETEttereshc

Actions
Story #5337: As a user, I can fully migrate on_demand contentCLOSED - CURRENTRELEASEttereshc

Actions
Story #5363: As a user, I can migrate docker content units from Pulp 2 into Pulp 3CLOSED - CURRENTRELEASEipanova@redhat.com

Actions
File Support - Task #3823: Analyze repository version creation performance for pulp_fileNEW

Actions
RPM Support - Task #3824: Analyze repository version creation performance for pulp_rpmCLOSED - COMPLETE

Actions
Story #5003: As a user, I can use the REST API to dispatch a Pulp 2 to 3 migration task.CLOSED - CURRENTRELEASEttereshc

Actions
Story #5080: As a user, I can track which tasks were using a specific Migration PlanCLOSED - DUPLICATEttereshc

Actions
Task #5081: Add ability to the migration tool to connect to mongodb from Pulp 3 environment CLOSED - CURRENTRELEASEttereshc

Actions
Task #5082: As a developer, pulplift can create a dev environment for migration toolCLOSED - COMPLETEdkliban@redhat.com

Actions
Task #5083: Add model for Pulp 2 data and dump Pulp 2 generic data using this model CLOSED - CURRENTRELEASEttereshc

Actions
Story #5178: As a user, a Migration Plan is validatedCLOSED - CURRENTRELEASEipanova@redhat.com

Actions
Story #5229: As a user, I can see progress of Migration Plan executionCLOSED - CURRENTRELEASEttereshc

Actions
Story #5230: As a user, I have a default configuration for MongoDB connection CLOSED - CURRENTRELEASE

Actions
Story #5231: As a user, I can query Pulp 2->Pulp 3 mapping for migrated contentCLOSED - CURRENTRELEASEdkliban@redhat.com

Actions
Story #5307: As a user, I can migrate repositories from Pulp2 to Pulp3CLOSED - CURRENTRELEASE

Actions
Task #5306: Add models to pre-migrate pulp 2 data about repositories/importers/distributorsCLOSED - CURRENTRELEASEttereshc

Actions
Story #5340: As a user, I can migrate Pulp 2 repository and importer to Pulp 3CLOSED - CURRENTRELEASEttereshc

Actions
Story #5342: As a user, I can migrate repository into a set of repository versionsCLOSED - CURRENTRELEASEttereshc

Actions
Story #5343: As a user, I can migrate Pulp 2 distributor into publication/distribution in Pulp 3CLOSED - CURRENTRELEASEipanova@redhat.com

Actions
Story #5319: As a user I have validated resources Migration plan refers toCLOSED - CURRENTRELEASEdalley

Actions
Story #5407: As a user I have a pulp2 api that generates a MP templateCLOSED - WONTFIX

Actions
Story #5450: As a user, a Migration Plan is respected CLOSED - CURRENTRELEASEdalley

Actions
Task #5476: Let plugin migrate all its content models at once and not one by oneCLOSED - CURRENTRELEASEttereshc

Actions
Issue #5485: Migration plugin should filter only those repo types that belong to the plugin that is being migratedCLOSED - CURRENTRELEASEipanova@redhat.comActions
Refactor #5491: Stop using mappings for plugin specific items, move everyting to a plugin migrator classCLOSED - CURRENTRELEASEttereshc

Actions
Story #5549: As a user, I can migrate content onlyCLOSED - WONTFIX

Actions
Task #5632: Mark changed/removed Pulp 2 repositories/importers/distributorsCLOSED - CURRENTRELEASEttereshc

Actions
Story #5634: As a user, I can query Pulp 2->Pulp 3 mapping for migrated resourcesCLOSED - CURRENTRELEASEfao89

Actions
Task #5837: Specify pulp2_distributor_repository_ids instead of distributor_idsCLOSED - CURRENTRELEASEdalley

Actions
Task #5845: Update get_pulp3_repository_setup so repos are grouped by plugin typeCLOSED - CURRENTRELEASEdalley

Actions
Task #5849: Add an example of a migration planCLOSED - CURRENTRELEASE

Actions
Task #5852: Add ability to pre-migrate importers and distributors independently from the repo they belong toCLOSED - CURRENTRELEASE

Actions

Related issues

Related to Migration Plugin - Test #5194: [Epic] Test migration from Pulp 2 to Pulp 3NEW<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

History

#1 Updated by dkliban@redhat.com about 2 years ago

  • Tracker changed from Issue to Story
  • % Done set to 0

#2 Updated by jsherril@redhat.com almost 2 years ago

  • Tags Katello-P1 added

#3 Updated by bmbouter about 1 year ago

  • Tags deleted (Pulp 3)

#4 Updated by dkliban@redhat.com about 1 year ago

  • Description updated (diff)

#5 Updated by dkliban@redhat.com about 1 year ago

  • Description updated (diff)

#6 Updated by dkliban@redhat.com about 1 year ago

  • Description updated (diff)

#7 Updated by jsherril@redhat.com about 1 year ago

  • Tags Katello-P3 added
  • Tags deleted (Katello-P1)

#8 Updated by kersom 12 months ago

  • Related to Test #5194: [Epic] Test migration from Pulp 2 to Pulp 3 added

#9 Updated by ttereshc 9 months ago

  • Description updated (diff)

#10 Updated by ttereshc 9 months ago

  • Description updated (diff)

#11 Updated by ttereshc 9 months ago

  • Description updated (diff)

#12 Updated by ttereshc 9 months ago

  • Description updated (diff)

#13 Updated by ttereshc 9 months ago

  • Tags Pulp 2 to 3 Migrate added
  • Tags deleted (Katello-P3)

#14 Updated by ttereshc 6 months ago

  • Project changed from Pulp to Migration Plugin

#15 Updated by ttereshc 5 months ago

  • Description updated (diff)

#16 Updated by bmbouter about 2 months ago

  • Tags deleted (Pulp 2 to 3 Migrate)

We're removing the 'Pulp 2 to 3 Migrate' tag to allow membership of the Migration Plugin project in Redmine to be the controlling area https://pulp.plan.io/projects/migration

Please register to edit this issue

Also available in: Atom PDF