Project

Profile

Help

Story #3821

As a user, I can migrate from Pulp 2 to Pulp 3

Added by dkliban@redhat.com over 1 year ago. Updated 2 days ago.

Status:
NEW
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
07/05/2019
Due date:
07/12/2019
% Done:

54%

Estimated time:
(Total: 4:00 h)
Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2 to 3 Migrate
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:

Description

Description

Pulp 3 will provide a plugin that will enable users to migrate their Pulp 2 instances to Pulp 3. The package will be called pulp-2to3-migration. Plugin writers will contribute to it to provide support of migration of the content types managed by that plugin.

After installing pulp-2to3-migration on a Pulp 3 system, users will be able to use the REST API on Pulp 3 to start the migration of the Pulp 2 instance to Pulp 3. Users will be able to describe the final state of their Pulp 3 instance for a specific plugin using a Migration Plan (MP). The MP is a JSON document that describes which artifacts, content, repositories, repository versions, publication, remotes, and distributions get created in Pulp 3 from their equivalent counterparts in Pulp 2. Users will be able to use Pulp 2's REST API to generate a default MP for migrating everything. Users will then submit the MP via REST API to Pulp 3.

After the migration task start running in Pulp 3, the user will be able to use the REST API to query the mappings between Pulp 2 and Pulp 3 resources.

Features

Pulp 2 will provide a REST API that returns a default Migration Plan.

This Migration Plan will migrate all content, all repositories, all importers, and all distributors.

Pulp 3 will provide a REST API that:

- creates a Migration Plan
- executes a Migration plan in one of 3 modes (run the migration without validation, run migration if MP is validated, run validation only - dry-run).

This API will dispatch one or more tasks to perform the migration. The tasking system will limit one migration "operation" at a time, although many tasks may run concurrently for that operation.

When dry_run is set to True, the Migration Plan validated and some information about the plan is returned to the user. Some validation that will be performed will include:
  • JSON syntax
  • do the listed Pulp 2 entities exist
    If all validation passes, the user receives a list of Content types and Repositories from Pulp 2 that will be used to create content and repositories in Pulp 3.

Pulp 3 will provide multiple REST APIs for querying mappings between Pulp 2 and Pulp 3.

One for content, one for repository and related resources - importers, distributors.

Migration Plan schema

The Migration Plan (MP) will consist of the plugin type and repositories specification for each plugin. If the desire is to migrate content only, repositories specification should not be mentioned at all. The Repositories section will describe Repositories, RepositoryVersions, Publications, and Distributions to create.

Examples

The MP below will:

  • migrate all content for ISO and Docker plugins
  • create a repository with name 'foo' in Pulp 3.
  • create a Remote in Pulp 3 based on details of an importer for the repo "my-repo-id-for-importer" in Pulp 2
  • create one repository version that will have the same content set as the ISO 'my-repo-id' repo in Pulp 2
  • implicitly create a Publication from this repository version
  • create 2 distributions based on details of distributors in Pulp 2
  • both of these distributions will serve the same publication.
 {
    "plugins": [
     { 
         "type": "iso", 
         "repositories" :  [
           {
             "name": "foo",
             "pulp2_importer_repository_id":  "my-repo-id-for-importer" 
             "repository_versions": [
                 {
                   "pulp2_repository_id": "my-repo-id",
                   "distributor_ids":  ["distributor_id1", "distributor_id2"]
                  },          
               ]
            }
        ]
     },
     {
       "type": "docker",
     }
  ]
}

The MP below will:

  • migrate all ISO content
  • implicitly create a repository with a single repository version in Pulp 3 for every repository in Pulp 2
  • the content set of the repository version will match the content of the repository in Pulp 2
  • implicitly create a Publication for all repositories that need publications
  • implicitly create a Remote for every importer associated with the repository
  • implicitly create a Distribution for every non-export distributor associated with the repository and associate either the RepositoryVersion or Publication with that Distribution
{
    "plugins": [
       { 
         "type": "iso",
         "repositories": [],
       }
     ]
}

The MP below will:

  • migrate all downloaded ISO content
  • not migrate any on_demand content
  • not migrate/create any repositories/remotes/distributions
{
    "plugins": [
       { 
         "type": "iso",
       }
     ]
}

Subtasks

Story #2962: As an User I would like to migrate in-place from Pulp2 to Pulp3 NEW

Actions
Story #3822: As a user, I can migrate all content units of a specific type from Pulp 2 to Pulp 3MODIFIED

Actions
Story #3810: As a user, I can migrate ISO content units from Pulp 2 into Pulp 3 as File ContentMODIFIEDttereshc

Actions
Task #4205: bootstrap pulp-2to3-migrate django appMODIFIEDttereshc

Actions
Story #5337: As a user, I can fully migrate on_demand contentMODIFIEDttereshc

Actions
Story #5363: As a user, I can migrate docker content units from Pulp 2 into Pulp 3MODIFIEDipanova@redhat.com

Actions
File Support - Task #3823: Analyze repository version creation performance for pulp_fileNEW

Actions
RPM Support - Task #3824: Analyze repository version creation performance for pulp_rpmNEW

Actions
Story #5003: As a user, I can use the REST API to dispatch a Pulp 2 to 3 migration task.MODIFIEDttereshc

Actions
Story #5080: As a user, I can track which tasks were using a specific Migration PlanCLOSED - DUPLICATEttereshc

Actions
Task #5081: Add ability to the migration tool to connect to mongodb from Pulp 3 environment MODIFIEDttereshc

Actions
Story #5082: As a developer, pulplift can create a dev environment for migration toolMODIFIEDdkliban@redhat.com

Actions
Task #5083: Add model for Pulp 2 data and dump Pulp 2 generic data using this model MODIFIEDttereshc

Actions
Story #5178: As a user, a Migration Plan is validatedMODIFIEDipanova@redhat.com

Actions
Story #5229: As a user, I can see progress of Migration Plan executionMODIFIEDttereshc

Actions
Story #5230: As a user, I have a default configuration for MongoDB connection NEW

Actions
Story #5231: As a user, I can query Pulp 2->Pulp 3 mapping for migrated contentMODIFIEDdkliban@redhat.com

Actions
Story #5307: As a user, I can migrate repositories from Pulp2 to Pulp3NEW

Actions
Task #5306: Add models to pre-migrate pulp 2 data about repositories/importers/distributorsMODIFIEDttereshc

Actions
Story #5340: As a user, I can migrate Pulp 2 repository and importer to Pulp 3MODIFIEDttereshc

Actions
Story #5342: As a user, I can migrate repository into a set of repository versionsMODIFIEDttereshc

Actions
Story #5343: As a user, I can migrate Pulp 2 distributor into publication/distribution in Pulp 3MODIFIEDipanova@redhat.com

Actions
Story #5446: As a user, pulp2 global settings are migratedNEW

Actions
Story #5319: As a user I have validated resources Migration plan refers toMODIFIEDdalley

Actions
Story #5407: As a user I have a pulp2 api that generates a MP templateNEW

Actions
Story #5450: As a user, a Migration Plan is respected MODIFIEDdalley

Actions
Task #5476: Let plugin migrate all its content models at once and not one by oneMODIFIEDttereshc

Actions
Issue #5485: Migration plugin should filter only those repo types that belong to the plugin that is being migratedMODIFIEDipanova@redhat.comActions
Refactor #5491: Stop using mappings for plugin specific items, move everyting to a plugin migrator classMODIFIEDttereshc

Actions
Story #5549: As a user, I can migrate content onlyASSIGNEDdalley

Actions
Task #5632: Mark changed/removed Pulp 2 repositories/importers/distributorsMODIFIEDttereshc

Actions
Story #5634: As a user, I can query Pulp 2->Pulp 3 mapping for migrated resourcesMODIFIEDfabricio.aguiar

Actions
Task #5837: Specify pulp2_distributor_repository_ids instead of distributor_idsASSIGNEDdalley

Actions
Task #5845: Update get_pulp3_repository_setup so repos are grouped by plugin typeNEW

Actions
Task #5849: Add an example of a migration planNEW

Actions
Task #5852: Add ability to pre-migrate importers and distributors independently from the repo they belong toASSIGNEDdalley

Actions

Related issues

Related to Pulp - Test #5194: [Epic] Test migration from Pulp 2 to Pulp 3 NEW Actions

History

#1 Updated by dkliban@redhat.com over 1 year ago

  • Tracker changed from Issue to Story
  • % Done set to 0

#2 Updated by jsherril@redhat.com over 1 year ago

  • Tags Katello-P1 added

#3 Updated by bmbouter 8 months ago

  • Tags deleted (Pulp 3)

#4 Updated by dkliban@redhat.com 7 months ago

  • Description updated (diff)

#5 Updated by dkliban@redhat.com 6 months ago

  • Description updated (diff)

#6 Updated by dkliban@redhat.com 6 months ago

  • Description updated (diff)

#7 Updated by jsherril@redhat.com 6 months ago

  • Tags Katello-P3 added
  • Tags deleted (Katello-P1)

#8 Updated by kersom 5 months ago

  • Related to Test #5194: [Epic] Test migration from Pulp 2 to Pulp 3 added

#9 Updated by ttereshc about 2 months ago

  • Description updated (diff)

#10 Updated by ttereshc about 2 months ago

  • Description updated (diff)

#11 Updated by ttereshc about 2 months ago

  • Description updated (diff)

#12 Updated by ttereshc about 2 months ago

  • Description updated (diff)

#13 Updated by ttereshc about 2 months ago

  • Tags Pulp 2 to 3 Migrate added
  • Tags deleted (Katello-P3)

Please register to edit this issue

Also available in: Atom PDF