Project

Profile

Help

Story #6375

As a user, I can track Remotes and not remigrate them on every run

Added by ttereshc over 1 year ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 72
Quarter:

Description

Motivation

Currently, Remotes are recreated on every run. When Remotes are removed, RemoteArtifacts are removed as well. We sent through the content migration pipeline all the content on every run only to recreate RemoteArtifacts. We need a way to minimise the amount of content which goes through the pipeline during migration re-run. If there are no changes in pulp2, almost no content should go through the migration pipeline.

Proposal

Track Remotes, if they were changed or not. Track any lazy catalog changes, were there any updates or not. Recreate Remotes only if needed. Recreate RemoteArtifacts only for the affected content.

Track Remotes, recreate Remotes only if needed.

This commit removed some initial attempts to track the Remotes, see the deletion in mark_removed_resources and pre_migrate_*.

  • Pulp2Importer model for pre-migration needs a new field not_in_plan
  • Update not_in_plan accordingly in the mark_removed_resources step
  • When pre-migrating an importer, determine if there were any changes and mark it by setting is_migrated to False
  • When pre-migrating an importer, If there are changes to the feed or any configuration which is available on RemoteArtifact, remove such Remote from pulp3
  • At the migration step, update a Remote for each Pulp2Importer which has pulp3_remote but is_migrated=False
  • At the migration step create a Remote for each Pulp2Importer which has no pulp3_remote
Track any lazy catalog changes
  • Move pre_migrate_all_content step to happen before any importers are migrated to Pulp 3
  • Pulp2LazyCatalog needs a new field is_migrated
  • At pre-migration time is_migrated field should be set
  • At pre-migration time, check if the lazy catalog entry is for the Pulp2Importer which has pulp3_remote or not.
    • If a Remote exists, it means it hasn't changed or the changes were not critical enough to recreate it, thus lazy catalog for this importer is already migrated, set is_migrated=True.
  • At content migration time is_migrated will help to identify whether we need to create RemoteArtifacts for content or not
Recreate RemoteArtifacts only for the affected content
  • At content migration stage, determine which migrated Pulp2Content (aka it has pulp3content) needs RemoteArtifact creation by querying pre-migrated lazy catalog entires which are is_migrated=False
  • Create RemoteArtifatcs for those either by sending the content through the content migration pipeline or by explicitly creating RemoteArtifatcs for them
  • Pulp2Content which has no pulp3content needs to be migrated fully
  • I think mutable content will be handled properly because if it changed new Pulp2Content is created and old records a re removed

Associated revisions

Revision 01ef5a3a View on GitHub
Added by ipanova@redhat.com over 1 year ago

As a user, I can track Remotes and not remigrate them on every run

closes #6375 https://pulp.plan.io/issues/6375

Revision 01ef5a3a View on GitHub
Added by ipanova@redhat.com over 1 year ago

As a user, I can track Remotes and not remigrate them on every run

closes #6375 https://pulp.plan.io/issues/6375

Revision 01ef5a3a View on GitHub
Added by ipanova@redhat.com over 1 year ago

As a user, I can track Remotes and not remigrate them on every run

closes #6375 https://pulp.plan.io/issues/6375

History

#1 Updated by ipanova@redhat.com over 1 year ago

I think mutable content will be handled properly because if it changed new Pulp2Content is created and old records a re removed

That's correct. We remove Pulp2Content and re-create a new one https://github.com/pulp/pulp-2to3-migration/blob/master/pulp_2to3_migration/app/pre_migration.py#L133

#2 Updated by dalley over 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

#3 Updated by dalley over 1 year ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (dalley)

#4 Updated by ipanova@redhat.com over 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com
  • Sprint set to Sprint 71

#5 Updated by ttereshc over 1 year ago

  • Description updated (diff)

#6 Updated by rchan over 1 year ago

  • Sprint changed from Sprint 71 to Sprint 72

#7 Updated by ipanova@redhat.com over 1 year ago

  • Status changed from ASSIGNED to MODIFIED
  • % Done changed from 0 to 100

#8 Updated by ttereshc over 1 year ago

  • Sprint/Milestone set to 0.2.0

#9 Updated by ttereshc over 1 year ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF