Migration Plugin

I have a problem by migrating from Pulp2 to Pulp3.

Pulp Version               2.1.0
Pulp server version    2.21.5
foreman-2.3.3-1.el7.noarch
katello-3.18.2-1.el7.noarch

pulp-2to3-migration (0.10.0)
pulp-certguard (1.0.3)
pulp-container (2.1.0)
pulp-deb (2.7.0)
pulp-file (1.3.0)
pulp-rpm (3.9.0)
pulpcore (3.7.3)


1. 

**foreman-maintain content prepare**

Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3:
Starting task.
2021-03-26 09:33:09 +0100: Content migration starting. Migration failed, You will want to investigate: https://foreman.com/foreman_tasks/tasks/fae253a9-0f64-48da-97f0-1326b8e6a467
rake aborted!
ForemanTasks::TaskError: Task fae253a9-0f64-48da-97f0-1326b8e6a467: Pulp2to3MigrationClient::ApiError: Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Fri, 26 Mar 2021 08:33:10 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"57", "via"=>"1.1 foreman.com", "connection"=>"close"}
Response body: ["Only one migration plan can run or be reset at a time"]
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

2.

**foreman-rake katello:pulp3_migration --trace**

** Invoke katello:pulp3_migration (first_time)
** Invoke dynflow:client (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute dynflow:client
** Execute katello:pulp3_migration
Starting task.
2021-03-26 09:34:30 +0100: Content migration starting. Migration failed, You will want to investigate: https://foreman.com/foreman_tasks/tasks/17132279-07da-457a-b488-d26f57fcef92
rake aborted!
ForemanTasks::TaskError: Task 17132279-07da-457a-b488-d26f57fcef92: Pulp2to3MigrationClient::ApiError: Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Fri, 26 Mar 2021 08:34:31 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"57", "via"=>"1.1 foreman.com", "connection"=>"close"}
Response body: ["Only one migration plan can run or be reset at a time"]
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in `invoke_task'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in `run_with_threads'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in `top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in `block in run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in `run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `load'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `<main>'
Tasks: TOP => katello:pulp3_migration

Pulp3 only allows one migration-task to run at a time. A migration is considered to be 'running' if any task associated with it is in 'running' or 'waiting' state.

katello considers a migration to have 'failed' if any task in the migration fails, the instant that happens. The problem with this is that there are a number of edge-cases that can cause individual migration-tasks to fail without affecting the bulk of the migration, and once it decides a migration has failed, katello won't let you cancel it (ie cancel any remaining tasks). And, since katello believes the migration has stopped, it will let you start/restart a new one, which pulp won't allow while it's still chugging away at the previous one.

I believe there is a katello issue open to fix this, but I don't have it handy.

This seems to be an issue on the katello side not handling http 409 from pulp. https://projects.theforeman.org/issues/32226

I'm closing this one as not a Pulp bug. Feel free to comment or open a new issue if you see it again after the katello issue is fixed.