Migration Plugin

Motivation¶

Currently, Remotes are recreated on every run. When Remotes are removed, RemoteArtifacts are removed as well. We sent through the content migration pipeline all the content on every run only to recreate RemoteArtifacts. We need a way to minimise the amount of content which goes through the pipeline during migration re-run. If there are no changes in pulp2, almost no content should go through the migration pipeline.

Proposal¶

Track Remotes, if they were changed or not. Track any lazy catalog changes, were there any updates or not. Recreate Remotes only if needed. Recreate RemoteArtifacts only for the affected content.

Track Remotes, recreate Remotes only if needed.¶

This commit removed some initial attempts to track the Remotes, see the deletion in mark_removed_resources and pre_migrate_*.

• Pulp2Importer model for pre-migration needs a new field not_in_plan
• Update not_in_plan accordingly in the mark_removed_resources step
• When pre-migrating an importer, determine if there were any changes and mark it by setting is_migrated to False
• When pre-migrating an importer, If there are changes to the feed or any configuration which is available on RemoteArtifact, remove such Remote from pulp3
• At the migration step, update a Remote for each Pulp2Importer which has pulp3_remote but is_migrated=False
• At the migration step create a Remote for each Pulp2Importer which has no pulp3_remote

Track any lazy catalog changes¶

• Move pre_migrate_all_content step to happen before any importers are migrated to Pulp 3
• Pulp2LazyCatalog needs a new field is_migrated
• At pre-migration time is_migrated field should be set
• At pre-migration time, check if the lazy catalog entry is for the Pulp2Importer which has pulp3_remote or not.
  • If a Remote exists, it means it hasn't changed or the changes were not critical enough to recreate it, thus lazy catalog for this importer is already migrated, set is_migrated=True.
• At content migration time is_migrated will help to identify whether we need to create RemoteArtifacts for content or not

Recreate RemoteArtifacts only for the affected content¶

• At content migration stage, determine which migrated Pulp2Content (aka it has pulp3content) needs RemoteArtifact creation by querying pre-migrated lazy catalog entires which are is_migrated=False
• Create RemoteArtifatcs for those either by sending the content through the content migration pipeline or by explicitly creating RemoteArtifatcs for them
• Pulp2Content which has no pulp3content needs to be migrated fully
• I think mutable content will be handled properly because if it changed new Pulp2Content is created and old records a re removed