Migration Plugin

(found a typo, I meant to say 502 instead of 504)

I tried on a katello box but I used the python bindings. Pulp2RepositoriesApi(migration_client).list(offset=0, limit=2000). It took few seconds but I didn't notice any issues, though it does create a short spike on the cpu load. I have 659 repos to list.

I tried it also when I kicked off a migration task. Nothing is failing so far but I'll keep trying at different stages.

I've discovered that I am only able to reproduce this on Katello nightly production boxes after migrating >= 400 repositories. The issue is not reproducible on a development box, however. I'm not sure yet what difference would cause this issue, it's not Pulp plugin versions at least.

I've found it's only reproducible if you do the following:

foreman-rake console

def api_client
  Pulp2to3MigrationClient::ApiClient.new(SmartProxy.pulp_primary!.pulp3_configuration(Pulp2to3MigrationClient::Configuration))
end

def pulp2_repositories_api
  Pulp2to3MigrationClient::Pulp2RepositoriesApi.new(api_client)
end

imported = Katello::Pulp3::Api::Core.fetch_from_list { |opts| pulp2_repositories_api.list(opts) }

Still trying to figure out why fetching with the above code causes the pulpcore worker to die.

Thanks, jsherrill. Agreed.

Just adding a bit more detail here. The likely problem is a serialization. It's very heavy in certain situations because of the following incorrect behaviour:
If pulp 2 has N copies of the same repository, it will be one repo version and one publication in pulp 3 which is good. It will create N distributions, which is correct. However, we serialize pulp2repository in a way that it shows all the distributions for each publication, so it will be N distributions for each pulp 2 repo, which is wrong.

Migration plugin needs to show only distributions relevant to that pulp 2 repo.

We might need to add pulp3_distributions relation to the pulp2repository model to resolve that.