Issue #7889: Handle migration when same distributor is being re-used for different repositories in between the plans - Migration Plugin - Pulp

Actions

Send by e-mail Copy link

Issue #7889

closed

Handle migration when same distributor is being re-used for different repositories in between the plans

Added by ipanova@redhat.com almost 4 years ago. Updated over 3 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

Normal

Assignee:

ttereshc

Sprint/Milestone:

0.9.0

Start date:

Due date:

Estimated time:

Severity:

2. Medium

Platform Release:

OS:

Triaged:

Groomed:

Sprint Candidate:

Tags:

Katello

Sprint:

Quarter:

Description

Create plan and run:


{
    "plan": {
        "plugins": [
            {
                "repositories": [
                    {
                        "name": "file",
                        "pulp2_importer_repository_id": "file",
                        "repository_versions": [
                            {
                                "pulp2_distributor_repository_ids": [
                                    "file2"
                                ],
                                "pulp2_repository_id": "file"
                            }
                        ]
                    },
                    {
                        "name": "file2",
                        "pulp2_importer_repository_id": "file2",
                        "repository_versions": [
                            {
                                "pulp2_distributor_repository_ids": [
                                    "file-many"
                                ],
                                "pulp2_repository_id": "file2"
                            }
                        ]
                    }
                ],
                "type": "iso"
            }
        ]
    },

Create another plan and run:

{
    "plan": {
        "plugins": [
            {
                "repositories": [
                    {
                        "name": "file",
                        "pulp2_importer_repository_id": "file",
                        "repository_versions": [
                            {
                                "pulp2_distributor_repository_ids": [
                                    "file-many"
                                ],
                                "pulp2_repository_id": "file"
                            }
                        ]
                    },
                    {
                        "name": "file2",
                        "pulp2_importer_repository_id": "file2",
                        "repository_versions": [
                            {
                                "pulp2_distributor_repository_ids": [
                                    "file-large"
                                ],
                                "pulp2_repository_id": "file2"
                            }
                        ]
                    }
                ],
                "type": "iso"
            }
        ]
    },

File2 repo will have 2 distributions displayed file-large and file-many. In addition there is a concern of hitting base_path collision because same distributor file-many has been used to distribute 2 different repos.

NOTE Importers will suffer as well from correctness problem in case same importer will be used for 2 different repos in between the migration plans

Actions

Copy link

Updated by ipanova@redhat.com almost 4 years ago

Description updated (diff)

Actions

Copy link

Updated by ttereshc almost 4 years ago

Arguably simpler situation to grasp the problem, is to imagine when two distributors are being swapped.

First migration plan/run:
repo A with dist A
repo B with dist B

Second migration plan/run:
repo A with dist B
repo B with dist A

Actions

Copy link

Updated by ttereshc almost 4 years ago

One idea would be to put relations from migration plan in a table and use it to identify old/new data.

class RepoSetup:
    repo_id = TextField()
    importer_id = TextField()
    distributor_id = TextField()
    status = IntegerField(choices=models.IntegerChoices('Status', 'old up_to_date new').choices)
    
    class Meta:
        unique_together = (
            (repo_id, distributor_id),
    )

On each migration run:

before pre-migration, set status for all rows to old.
then create records with status new or update with status up_to_date. (FWIW< this would likely be a per triplet operation and not a bulk one)
use this data to clean up old relations between premigrated repos and premigrated importers/distributors and set is_migrated=false for the distributors which changed the repository they are distributing.
at pre-migration time, use this table to ensure all relations are correct

Extend the reset functionality to clean up this table.

Outside of the scope of this ticket: this model can also be used as one of conditions to identify no-to-few changes.

Actions

Copy link

Updated by ipanova@redhat.com almost 4 years ago

Before I forget again, posting here the rough idea how to deal with the problem ( one of the discussed approaches).

Use repo_version_setup() instead of get_importers_repos/get_distributors_repo(). This will require to get rid of simple migration plan first. https://github.com/pulp/pulp-2to3-migration/blob/master/pulp_2to3_migration/app/pre_migration.py#L361

Actions

Copy link

Updated by ttereshc almost 4 years ago

Sprint/Milestone set to 0.9.0

Actions

Copy link

Updated by jsherril@redhat.com almost 4 years ago

Tags Katello added

Actions

Copy link

Updated by ttereshc almost 4 years ago

Status changed from NEW to ASSIGNED
Assignee set to ttereshc

Actions

Copy link

Updated by ipanova@redhat.com almost 4 years ago

ipanova@redhat.com wrote:

Before I forget again, posting here the rough idea how to deal with the problem ( one of the discussed approaches).

Use repo_version_setup() instead of get_importers_repos/get_distributors_repo(). This will require to get rid of simple migration plan first. https://github.com/pulp/pulp-2to3-migration/blob/master/pulp_2to3_migration/app/pre_migration.py#L361

This option is no longer valid after the recent optimizations that were made on the re-runs https://github.com/pulp/pulp-2to3-migration/pull/311