Project

Profile

Help

Issue #7889

Handle migration when same distributor is being re-used for different repositories in between the plans

Added by ipanova@redhat.com 5 months ago. Updated about 1 month ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Quarter:

Description

Create plan and run:


{
    "plan": {
        "plugins": [
            {
                "repositories": [
                    {
                        "name": "file",
                        "pulp2_importer_repository_id": "file",
                        "repository_versions": [
                            {
                                "pulp2_distributor_repository_ids": [
                                    "file2"
                                ],
                                "pulp2_repository_id": "file"
                            }
                        ]
                    },
                    {
                        "name": "file2",
                        "pulp2_importer_repository_id": "file2",
                        "repository_versions": [
                            {
                                "pulp2_distributor_repository_ids": [
                                    "file-many"
                                ],
                                "pulp2_repository_id": "file2"
                            }
                        ]
                    }
                ],
                "type": "iso"
            }
        ]
    },

Create another plan and run:

{
    "plan": {
        "plugins": [
            {
                "repositories": [
                    {
                        "name": "file",
                        "pulp2_importer_repository_id": "file",
                        "repository_versions": [
                            {
                                "pulp2_distributor_repository_ids": [
                                    "file-many"
                                ],
                                "pulp2_repository_id": "file"
                            }
                        ]
                    },
                    {
                        "name": "file2",
                        "pulp2_importer_repository_id": "file2",
                        "repository_versions": [
                            {
                                "pulp2_distributor_repository_ids": [
                                    "file-large"
                                ],
                                "pulp2_repository_id": "file2"
                            }
                        ]
                    }
                ],
                "type": "iso"
            }
        ]
    },

File2 repo will have 2 distributions displayed file-large and file-many. In addition there is a concern of hitting base_path collision because same distributor file-many has been used to distribute 2 different repos.

NOTE Importers will suffer as well from correctness problem in case same importer will be used for 2 different repos in between the migration plans

Associated revisions

Revision 6173a64d View on GitHub
Added by ttereshc about 2 months ago

Fix re-migration when distributors/importers changed in migration plan.

closes #7889 https://pulp.plan.io/issues/7889

Revision 6173a64d View on GitHub
Added by ttereshc about 2 months ago

Fix re-migration when distributors/importers changed in migration plan.

closes #7889 https://pulp.plan.io/issues/7889

Revision 6173a64d View on GitHub
Added by ttereshc about 2 months ago

Fix re-migration when distributors/importers changed in migration plan.

closes #7889 https://pulp.plan.io/issues/7889

History

#1 Updated by ipanova@redhat.com 5 months ago

  • Description updated (diff)

#2 Updated by ttereshc 5 months ago

Arguably simpler situation to grasp the problem, is to imagine when two distributors are being swapped.

First migration plan/run:
repo A with dist A
repo B with dist B

Second migration plan/run:
repo A with dist B
repo B with dist A

#3 Updated by ttereshc 5 months ago

One idea would be to put relations from migration plan in a table and use it to identify old/new data.

class RepoSetup:
    repo_id = TextField()
    importer_id = TextField()
    distributor_id = TextField()
    status = IntegerField(choices=models.IntegerChoices('Status', 'old up_to_date new').choices)
    
    class Meta:
        unique_together = (
            (repo_id, distributor_id),
    )

On each migration run:

  • before pre-migration, set status for all rows to old.
  • then create records with status new or update with status up_to_date. (FWIW< this would likely be a per triplet operation and not a bulk one)
  • use this data to clean up old relations between premigrated repos and premigrated importers/distributors and set is_migrated=false for the distributors which changed the repository they are distributing.
  • at pre-migration time, use this table to ensure all relations are correct

Extend the reset functionality to clean up this table.

Outside of the scope of this ticket: this model can also be used as one of conditions to identify no-to-few changes.

#4 Updated by ipanova@redhat.com 4 months ago

Before I forget again, posting here the rough idea how to deal with the problem ( one of the discussed approaches).

Use repo_version_setup() instead of get_importers_repos/get_distributors_repo(). This will require to get rid of simple migration plan first. https://github.com/pulp/pulp-2to3-migration/blob/master/pulp_2to3_migration/app/pre_migration.py#L361

#5 Updated by ttereshc 3 months ago

  • Sprint/Milestone set to 0.9.0

#6 Updated by jsherril@redhat.com 3 months ago

  • Tags Katello added

#7 Updated by ttereshc 2 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ttereshc

#8 Updated by ipanova@redhat.com 2 months ago

wrote:

Before I forget again, posting here the rough idea how to deal with the problem ( one of the discussed approaches).

Use repo_version_setup() instead of get_importers_repos/get_distributors_repo(). This will require to get rid of simple migration plan first. https://github.com/pulp/pulp-2to3-migration/blob/master/pulp_2to3_migration/app/pre_migration.py#L361

This option is no longer valid after the recent optimizations that were made on the re-runs https://github.com/pulp/pulp-2to3-migration/pull/311

#10 Updated by ttereshc about 2 months ago

  • Status changed from ASSIGNED to MODIFIED

#11 Updated by pulpbot about 1 month ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF