Issue #6099
Updated by dalley about 4 years ago
Running the a docker migrator more than once migration plan twice blows up on relating manifest blobs. It doesn't matter what the migration plan is specifically as long as the repos (/ repo content) overlaps. Tested with these plans: ~~~ http POST :24817/pulp/api/v3/migration-plans/ plan='{"plugins": [{"type": "docker"}]}' ~~~ ~~~ http POST :24817/pulp/api/v3/migration-plans/ plan='{"plugins": [{"type": "docker", "repositories": [{"name": "test-fixture-1", "repository_versions": [{"pulp2_repository_id": "test-fixture-1", "pulp2_distributor_repository_ids": ["test-fixture-1"]}]}]}]}' ~~~ Results: ~~~ Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: future: <Task finished coro=<DockerMigrator.migrate_content_to_pulp3() done, defined at /home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/docker/migrator.py:73> exception=IntegrityError('duplicate key value violates unique constraint "container_blobmanifest_manifest_id_manifest_blo_05da3e8d_uniq"\nDETAIL: Key (manifest_id, manifest_blob_id)=(c20c1b65-83a0-4044-9f28-1e06ee2084fc, f03fbe62-6b09-48af-b53e-59d439a01313) already exists.\n',)> Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: Traceback (most recent call last): Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: return self.cursor.execute(sql, params) Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "container_blobmanifest_manifest_id_manifest_blo_05da3e8d_uniq" Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: DETAIL: Key (manifest_id, manifest_blob_id)=(c20c1b65-83a0-4044-9f28-1e06ee2084fc, f03fbe62-6b09-48af-b53e-59d439a01313) already exists. Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: The above exception was the direct cause of the following exception: Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: Traceback (most recent call last): Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/docker/migrator.py", line 80, in migrate_content_to_pulp3 Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: await dm.create() Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/content.py", line 83, in create Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: await pipeline Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 209, in create_pipeline Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: await asyncio.gather(*futures) Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__ Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: await self.run() Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/docker/migrator.py", line 178, in run Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: self.relate_blob(dc) Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/docker/migrator.py", line 214, in relate_blob Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: thru.save() Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py", line 741, in save Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: force_update=force_update, update_fields=update_fields) Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py", line 779, in save_base Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: force_update, using, update_fields, Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py", line 870, in _save_table Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: result = self._do_insert(cls._base_manager, using, fields, update_pk, raw) Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py", line 908, in _do_insert Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: using=using, raw=raw) Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: return getattr(self.get_queryset(), name)(*args, **kwargs) Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: return query.get_compiler(using=using).execute_sql(return_id) Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 1368, in execute_sql Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: cursor.execute(sql, params) Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: return executor(sql, params, many, context) Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: return self.cursor.execute(sql, params) Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/utils.py", line 89, in __exit__ Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: raise dj_exc_value.with_traceback(traceback) from exc_value Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: return self.cursor.execute(sql, params) Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: django.db.utils.IntegrityError: duplicate key value violates unique constraint "container_blobmanifest_manifest_id_manifest_blo_05da3e8d_uniq" Feb 04 20:18:08 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[11164]: DETAIL: Key (manifest_id, manifest_blob_id)=(c20c1b65-83a0-4044-9f28-1e06ee2084fc, f03fbe62-6b09-48af-b53e-59d439a01313) already exists. ~~~ The same thing happens with a more specific migration plan such as ~~~ http POST :24817/pulp/api/v3/migration-plans/ plan='{"plugins": [{"type": "docker", "repositories": [{"name": "test-fixture-1", "repository_versions": [{"pulp2_repository_id": "test-fixture-1", "pulp2_distributor_repository_ids": ["test-fixture-1"]}]}]}]}' ~~~
.