Project

Profile

Help

Issue #5374

re-run of migration tooling has incorrect counts

Added by jsherril@redhat.com over 2 years ago. Updated 12 months ago.

Status:
CLOSED - WONTFIX
Priority:
Low
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Quarter:

Description

I synced a repo with ~14K files in pulp2 and then migrated all content to pulp3:

{
    "_created": "2019-08-29T18:59:50.347635Z", 
    "_href": "/pulp/api/v3/tasks/1a1a93c2-dfd6-4ba6-b54e-c74735a3ce5d/", 
    "created_resources": [], 
    "error": null, 
    "finished_at": "2019-08-29T19:01:37.544664Z", 
    "name": "pulp_2to3_migrate.app.tasks.migrate.migrate_from_pulp2", 
    "non_fatal_errors": [], 
    "parent": null, 
    "progress_reports": [
        {
            "done": 14444, 
            "message": "Pre-migrating Pulp 2 ISO content (general info)", 
            "state": "completed", 
            "suffix": null, 
            "total": 14447
        }, 
        {
            "done": 14444, 
            "message": "Pre-migrating Pulp 2 ISO content (detail info)", 
            "state": "completed", 
            "suffix": null, 
            "total": 14447
        }, 
        {
            "done": 14051, 
            "message": "Migrating ISO content to Pulp 3", 
            "state": "completed", 
            "suffix": null, 
            "total": 14453
        }, 
        {
            "done": 14453, 
            "message": "Migrating content to Pulp 3", 
            "state": "completed", 
            "suffix": null, 
            "total": 14453
        }
    ], 
    "reserved_resources_record": [
        "pulp_2to3_migration"
    ], 
    "spawned_tasks": [], 
    "started_at": "2019-08-29T18:59:50.474079Z", 
    "state": "completed", 
    "worker": "/pulp/api/v3/workers/75176664-9633-46f5-8989-456923ba333a/"
}

the task counts here looked correct. I then re-ran the migration without any changes, and it looked like this:

{
    "_created": "2019-08-29T19:24:08.615486Z", 
    "_href": "/pulp/api/v3/tasks/eee0440a-0d46-48f8-8d3b-2bf711b7b0c4/", 
    "created_resources": [], 
    "error": null, 
    "finished_at": "2019-08-29T19:24:11.410935Z", 
    "name": "pulp_2to3_migrate.app.tasks.migrate.migrate_from_pulp2", 
    "non_fatal_errors": [], 
    "parent": null, 
    "progress_reports": [
        {
            "done": 390, 
            "message": "Migrating ISO content to Pulp 3", 
            "state": "completed", 
            "suffix": null, 
            "total": 402
        }, 
        {
            "done": 402, 
            "message": "Migrating content to Pulp 3", 
            "state": "completed", 
            "suffix": null, 
            "total": 402
        }, 
        {
            "done": 0, 
            "message": "Pre-migrating Pulp 2 ISO content (general info)", 
            "state": "completed", 
            "suffix": null, 
            "total": 0
        }, 
        {
            "done": 0, 
            "message": "Pre-migrating Pulp 2 ISO content (detail info)", 
            "state": "completed", 
            "suffix": null, 
            "total": 0
        }
    ], 
    "reserved_resources_record": [
        "pulp_2to3_migration"
    ], 
    "spawned_tasks": [], 
    "started_at": "2019-08-29T19:24:08.732119Z", 
    "state": "completed", 
    "worker": "/pulp/api/v3/workers/75176664-9633-46f5-8989-456923ba333a/"
}

the counts for that first progress_report appears incorrect

History

#1 Updated by amacdona@redhat.com over 2 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 58

#2 Updated by ttereshc over 2 years ago

Just for the context:
Migration plugin looks at the last_updated time and chooses what to migrate based on that. last_updated precision is 1 sec.
Since it can be re-run N times on a working system, it's possible that some content from that second has been migrated and some has not.
So only in the process of migration we figure out if something was migrated before or not. It's likely some counters need to be adjusted accordingly. It's done for the pre-migration part, maybe the later parts need the same.

Maybe it's a wrong guess ^ but the context might be useful.

However, +1 the first report also shows incorrect counters.

#3 Updated by rchan over 2 years ago

  • Sprint deleted (Sprint 58)

#4 Updated by ttereshc about 2 years ago

  • Project changed from Pulp to Migration Plugin

#5 Updated by dalley about 2 years ago

At least for the first report, it no longer seems to be an issue, at least using the 4 standard test ISO repos. I can't vouch for the more comprehensive repos Justin is using.

The second time around, the numbers are still different.

        {
            "code": "premigrating.content.detail",
            "done": 0,
            "message": "Pre-migrating Pulp 2 ISO content (detail info)",
            "state": "completed",
            "suffix": null,
            "total": 0
        },
        {
            "code": "migrating.content",
            "done": 253,
            "message": "Migrating content to Pulp 3",
            "state": "completed",
            "suffix": null,
            "total": 253
        },
        {
            "code": "migrating.iso.content",
            "done": 266,
            "message": "Migrating iso content to Pulp 3 iso",
            "state": "completed",
            "suffix": null,
            "total": 266
        },
        {
            "code": "migrating.distributors",
            "done": 0,
            "message": "Migrating distributors to Pulp 3",
            "state": "completed",
            "suffix": null,
            "total": 4
        }

#6 Updated by ggainey over 1 year ago

  • Priority changed from Normal to Low

#7 Updated by ggainey over 1 year ago

  • Tags Katello added
  • Tags deleted (Katello-P3)

#8 Updated by ttereshc over 1 year ago

  • Sprint/Milestone set to 0.2.0

#9 Updated by ttereshc over 1 year ago

Needs to be retested

#10 Updated by ttereshc over 1 year ago

  • Sprint/Milestone deleted (0.2.0)

#11 Updated by jsherril@redhat.com 12 months ago

  • Status changed from NEW to CLOSED - WONTFIX

going to close this due to priorities and will reopen if it is still a major problem

Also available in: Atom PDF