Story #6374
closed[Epic] As a user, performance can be improved by increasing number of workers
100%
Description
Motivation¶
There are parts of the migration process that can be run in parallel in multiple tasks. Even by having different parts run in parallel in coroutines, database requests will still be sequential. The parts which can run in parallel - content migration and repo version/publication/distribution creation - currently consumes 70% of re-migration time.
Proposal¶
Dispatch multiple tasks for the parts that can be run in parallel.
Workflow¶
- User creates a MigrationPlan and triggers a migration task.
- A migration task:
- performs any migration validations if needed
- creates a migration report
- dispatches a poll task
- finishes and is marked as complete
- User tracks the migration progress by looking at the migration report
- The poll task:
- checks the progress of migration by looking at the migration report and the status of the tasks from the migration report and acts accordingly
- if a migration step is complete, dispatch one or more tasks for the next step
- if the migration step is not complete, re-dispatch itself
- If the overall status of the migration report is set to 'cancelled', the poll task should finish and not dispatch itself
- if the last step of migration is finished, the poll task should finish and not dispatch itself
- checks the progress of migration by looking at the migration report and the status of the tasks from the migration report and acts accordingly
- To cancel the migration user needs to update the state in the migration report to 'cancel'.
MigrationReport¶
There needs to be a MigrationReport object which will track the migration progress, including the tasks being dispatched. It should be accessible at the dedicated endpoint /pulp/api/v3/migrationreport/. It is created by the task which initiates the migration process.
To prevent triggering multiple migrations by user, no MigrationReport can be created if there is a MigrationReport in the unfinished state.
At a minimum, a migration report has:
- migration plan href
- status of the migration
- start/finish timestamps
- for each progress step:
- task href
- status
- code (string, id of the step)
- total/done counters
GET /pulp/api/v3/migrationreport//
- to query a specific MigrationReport
PATCH /pulp/api/v3/migrationreport//
- allows to update only the status of the migration to cancel any unfinished tasks related to this migration
- before updating the status of migration report to 'cancel', cancel all the tasks for the migration
Poll task¶
There needs to be a poll task which serves as a synchronisation mechanism for multiple tasks between different steps of the migration process. It checks the progress of migration, re-dispatches itself when done, dispatches a new batch of tasks for the next step of the migration when needed.
It needs to be aware of :
- the order of the migration steps
- which tasks to trigger for each of them
- when the migration step is done
Tasks to run in parallel¶
- Pre-migration is run as a single task.
- Content migration can be run in multiple tasks, the order of content migration has to be preserved
- Creation of repo versions, publications, distributions can be run in multiple tasks.
Updated by bmbouter over 4 years ago
This captures what we talked about well. It's also well written. One idea/detail I wanted to suggest is the idea that each "phase" or step have its own poll task. This might make the writing of each of the tasks easier than one which needs to handle the problem more generally. That's the only additional thought I had after reading this.
Thank you!
Updated by dalley over 4 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to dalley
Added by dalley over 4 years ago
Updated by dalley over 4 years ago
- Status changed from ASSIGNED to MODIFIED
- % Done changed from 0 to 100
Applied in changeset pulp:pulp-2to3-migration|b102ff9da0e304e4f00c9af59a54d9090ec2ad3b.
Updated by ttereshc almost 4 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Add concurrency
closes: #6374 https://pulp.plan.io/issues/6374