Project

Profile

Help

Story #3821

As a user, I can migrate from Pulp 2 to Pulp 3

Added by dkliban@redhat.com over 1 year ago. Updated 3 days ago.

Status:
NEW
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
07/05/2019
Due date:
07/12/2019
% Done:

58%

Estimated time:
(Total: 4:00 h)
Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Katello-P3
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:

Description

Description

Pulp 3 will provide optional plugins that will enable users to migrate their Pulp 2 instances to Pulp 3. The main package will be called pulp-2to3-migrate. Each plugin will also provide a package that will enable migration of the content types managed by that plugin.

After installing pulp-2to3-migrate on a Pulp 3 system, users will be able to use the REST API on Pulp 3 to start the migration of the Pulp 2 instance to Pulp 3. Users will be able to describe the final state of their Pulp 3 instance using a Migration Plan (MP). The MP is a JSON document that describes which artifacts, content, repositories, repository versions, publication, remotes, and distributions get created in Pulp 3 from their equivalent counterparts in Pulp 2. Users will be able to use Pulp 2's REST API to generate a default MP for migrating everything. Users will then submit the MP via REST API to Pulp 3.

After the migration task start running in Pulp 3, the user will be able to use the REST API to query the mappings between Pulp 2 and Pulp 3 resources.

Features

Pulp 2 will provide a REST API that returns a default Migration Plan.

This Migration Plan will migrate all content, all repositories, all importers, and all distributors.

Pulp 3 will provide a REST API that will accept three parameters: Migration Plan, concurrency, and dry_run.

This API will dispatch one more tasks to perform the migration. The default concurrency will be (Pulp 3 worker count) - 1. The concurrency will determine the number of tasks that will be dispatched in parallel. The tasking system will limit one migration "operation" at a time, although many tasks may run concurrently for that operation.

This API will dispatch one more tasks to perform the migration. The default concurrency will be (Pulp 3 worker count) - 1. The concurrency will determine the number of tasks that will be dispatched in parallel. The tasking system will limit one migration "operation" at a time, although many tasks may run concurrently for that operation.

When dry_run is set to True, the Migration Plan validated and some concreate information about the plan is returned to the user. Some validation that will be performed will include:
  • JSON syntax
  • do the listed Pulp 2 entities exist
    If all validation passes, the user receives a list of Content types and Repositories from Pulp 2 that will be used to create content and repositories in Pulp 3.

Pulp 3 will provide multiple REST APIs for querying mappings between content in Pulp 2 and Pulp 3.

TODO: figure out what exact APIs are needed

Migration Plan schema

The Migration Plan (MP) will consist of 2 sections: Content and Repositories. The Content section of the MP will describe the state of Content units and their Artifacts in Pulp 3. The Repositories section will describe Repositories, RepositoryVersions, Publications, and Distributions.

Examples

The MP below will:

  • migrate all content.
  • create a repository with name 'foo' in Pulp 3.
  • create one repository version that will have the same content set as the 'my-repo-id' repo in Pulp 2
  • implicitly create a Publication from this repository version\
  • create 2 distributions based on details of distributors in Pulp 2
  • both of these distributions will serve the same publication.
{
    "content" : {},
    "repositories" : [
       {
           "name" : "foo",
           "repository_versions" : [
                {
                    "pulp2_repository" : "my-repo-id",
                    "distributions":  [
                        {"pulp2_distributor" :  "pulp2-distributor-id"},
                        {"pulp2_distributor" :  "another-pulp2-distributor-id"}
                    ]
               }
           ]
        }
  ]
}

The MP below will:

  • implicitly create artifacts and content units in Pulp 3 for all the content units in Pulp 2 for which a Pulp 3 plugin is installed
  • implicitly create a repository with a single repository version in Pulp 3 for every repository in Pulp 2
  • the content set of the repository version will match the content of the repository in Pulp 2
  • implicitly create a Publication for all repositories that need publications
  • implicitly create a Remote for every importer associated with the repository
  • implicitly create a Distribution for every non-export distributor associated with the repository and associate either the RepositoryVersion or Publication with that Distribution
{
    "content" : {},
    "repositories" : []
}

Subtasks

Story #2962: As an User I would like to migrate in-place from Pulp2 to Pulp3 NEW

Actions
Story #3822: As a user, I can migrate all content units of a specific type from Pulp 2 to Pulp 3NEW

Actions
Story #3810: As a user, I can migrate ISO content units from Pulp 2 into Pulp 3 as File ContentMODIFIEDttereshc

Actions
Task #4205: bootstrap pulp-2to3-migrate django appMODIFIEDttereshc

Actions
Story #5337: As a user, I can fully migrate on_demand contentMODIFIEDttereshc

Actions
Story #5363: As a user, I can migrate docker content units from Pulp 2 into Pulp 3MODIFIEDipanova@redhat.com

Actions
File Support - Task #3823: Analyze repository version creation performance for pulp_fileNEW

Actions
RPM Support - Task #3824: Analyze repository version creation performance for pulp_rpmNEW

Actions
Story #5003: As a user, I can use the REST API to dispatch a Pulp 2 to 3 migration task.MODIFIEDttereshc

Actions
Story #5080: As a user, I can track which tasks were using a specific Migration PlanCLOSED - DUPLICATEttereshc

Actions
Task #5081: Add ability to the migration tool to connect to mongodb from Pulp 3 environment MODIFIEDttereshc

Actions
Story #5082: As a developer, pulplift can create a dev environment for migration toolMODIFIEDdkliban@redhat.com

Actions
Task #5083: Add model for Pulp 2 data and dump Pulp 2 generic data using this model MODIFIEDttereshc

Actions
Story #5178: As a user, a Migration Plan is validatedMODIFIEDipanova@redhat.com

Actions
Story #5229: As a user, I can see progress of Migration Plan executionMODIFIEDttereshc

Actions
Story #5230: As a user, I have a default configuration for MongoDB connection NEW

Actions
Story #5231: As a user, I can query Pulp 2->Pulp 3 mapping for migrated contentMODIFIEDdkliban@redhat.com

Actions
Story #5307: As a user, I can migrate repositories from Pulp2 to Pulp3NEW

Actions
Task #5306: Add models to pre-migrate pulp 2 data about repositories/importers/distributorsMODIFIEDttereshc

Actions
Story #5340: As a user, I can migrate Pulp 2 repository and importer to Pulp 3MODIFIEDttereshc

Actions
Story #5342: As a user, I can migrate repository into a set of repository versionsASSIGNEDttereshc

Actions
Story #5343: As a user, I can migrate Pulp 2 distributor into publication/distribution in Pulp 3NEW

Actions
Story #5446: As a user, pulp2 global settings are migratedNEW

Actions
Story #5319: As a user I have validated resources Migration plan refers toMODIFIEDdalley

Actions
Story #5407: As a user I have a pulp2 api that generates a MP templateNEW

Actions
Story #5450: As a user, a Migration Plan is respected MODIFIEDdalley

Actions
Task #5476: Let plugin migrate all its content models at once and not one by oneMODIFIEDttereshc

Actions
Issue #5485: Migration plugin should filter only those repo types that belong to the plugin that is being migratedASSIGNEDipanova@redhat.comActions
Refactor #5491: Stop using mappings for plugin specific items, move everyting to a plugin migrator classMODIFIEDttereshc

Actions
Story #5549: As a user, I can migrate content onlyASSIGNEDdalley

Actions

Related issues

Related to Pulp - Test #5194: [Epic] Test migration from Pulp 2 to Pulp 3 NEW Actions

History

#1 Updated by dkliban@redhat.com over 1 year ago

  • Tracker changed from Issue to Story
  • % Done set to 0

#2 Updated by jsherril@redhat.com about 1 year ago

  • Tags Katello-P1 added

#3 Updated by bmbouter 6 months ago

  • Tags deleted (Pulp 3)

#4 Updated by dkliban@redhat.com 5 months ago

  • Description updated (diff)

#5 Updated by dkliban@redhat.com 5 months ago

  • Description updated (diff)

#6 Updated by dkliban@redhat.com 5 months ago

  • Description updated (diff)

#7 Updated by jsherril@redhat.com 4 months ago

  • Tags Katello-P3 added
  • Tags deleted (Katello-P1)

#8 Updated by kersom 3 months ago

  • Related to Test #5194: [Epic] Test migration from Pulp 2 to Pulp 3 added

Please register to edit this issue

Also available in: Atom PDF