Issue #7889
closed
Handle migration when same distributor is being re-used for different repositories in between the plans
Description
Create plan and run:
{
"plan": {
"plugins": [
{
"repositories": [
{
"name": "file",
"pulp2_importer_repository_id": "file",
"repository_versions": [
{
"pulp2_distributor_repository_ids": [
"file2"
],
"pulp2_repository_id": "file"
}
]
},
{
"name": "file2",
"pulp2_importer_repository_id": "file2",
"repository_versions": [
{
"pulp2_distributor_repository_ids": [
"file-many"
],
"pulp2_repository_id": "file2"
}
]
}
],
"type": "iso"
}
]
},
Create another plan and run:
{
"plan": {
"plugins": [
{
"repositories": [
{
"name": "file",
"pulp2_importer_repository_id": "file",
"repository_versions": [
{
"pulp2_distributor_repository_ids": [
"file-many"
],
"pulp2_repository_id": "file"
}
]
},
{
"name": "file2",
"pulp2_importer_repository_id": "file2",
"repository_versions": [
{
"pulp2_distributor_repository_ids": [
"file-large"
],
"pulp2_repository_id": "file2"
}
]
}
],
"type": "iso"
}
]
},
File2 repo will have 2 distributions displayed file-large and file-many. In addition there is a concern of hitting base_path collision because same distributor file-many has been used to distribute 2 different repos.
NOTE Importers will suffer as well from correctness problem in case same importer will be used for 2 different repos in between the migration plans
Updated by ttereshc over 3 years ago
Arguably simpler situation to grasp the problem, is to imagine when two distributors are being swapped.
First migration plan/run:
repo A with dist A
repo B with dist B
Second migration plan/run:
repo A with dist B
repo B with dist A
Updated by ttereshc over 3 years ago
One idea would be to put relations from migration plan in a table and use it to identify old/new data.
class RepoSetup:
repo_id = TextField()
importer_id = TextField()
distributor_id = TextField()
status = IntegerField(choices=models.IntegerChoices('Status', 'old up_to_date new').choices)
class Meta:
unique_together = (
(repo_id, distributor_id),
)
On each migration run:
- before pre-migration, set status for all rows to
old. - then create records with status new or update with status up_to_date. (FWIW< this would likely be a per triplet operation and not a bulk one)
- use this data to clean up old relations between premigrated repos and premigrated importers/distributors and set is_migrated=false for the distributors which changed the repository they are distributing.
- at pre-migration time, use this table to ensure all relations are correct
Extend the reset functionality to clean up this table.
Outside of the scope of this ticket: this model can also be used as one of conditions to identify no-to-few changes.
Updated by ipanova@redhat.com over 3 years ago
Before I forget again, posting here the rough idea how to deal with the problem ( one of the discussed approaches).
Use repo_version_setup() instead of get_importers_repos/get_distributors_repo(). This will require to get rid of simple migration plan first. https://github.com/pulp/pulp-2to3-migration/blob/master/pulp_2to3_migration/app/pre_migration.py#L361
Updated by ttereshc about 3 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to ttereshc
Updated by ipanova@redhat.com about 3 years ago
ipanova@redhat.com wrote:
Before I forget again, posting here the rough idea how to deal with the problem ( one of the discussed approaches).
Use repo_version_setup() instead of get_importers_repos/get_distributors_repo(). This will require to get rid of simple migration plan first. https://github.com/pulp/pulp-2to3-migration/blob/master/pulp_2to3_migration/app/pre_migration.py#L361
This option is no longer valid after the recent optimizations that were made on the re-runs https://github.com/pulp/pulp-2to3-migration/pull/311
Added by ttereshc about 3 years ago
Added by ttereshc about 3 years ago
Revision 6173a64d | View on GitHub
Fix re-migration when distributors/importers changed in migration plan.
Added by ttereshc about 3 years ago
Revision 6173a64d | View on GitHub
Fix re-migration when distributors/importers changed in migration plan.
Updated by ttereshc about 3 years ago
- Status changed from ASSIGNED to MODIFIED
Applied in changeset pulp:pulp-2to3-migration|6173a64d62bad465c6789d08613c07c7f205a846.
Updated by pulpbot about 3 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
.
Fix re-migration when distributors/importers changed in migration plan.
closes #7889 https://pulp.plan.io/issues/7889