Container Support

Ticket moved to GitHub: "pulp/pulp_container/495":https://github.com/pulp/pulp_container/issues/495

The relations between content units should be handled in the same transaction where the relating content is created. The current approach can lead to not fully related content units in case the sync pipeline did not finish successfully.

The tools to handle this should be Content.resolution and the _post_save hook of the content stage.

By using Content.resolution we will also control the order of content that is being saved/processed in the sync pipeline, this way such errors like this will be mitigated:

Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: pulp [a2b82b5f-0427-4420-919e-b149d00c759e]: pulpcore.tasking.pulpcore_worker:INFO: Task fb0f3c7c-e537-4d15-9363-ea357f73d910 failed (null value in column "manifest_id" violates not-null constraint
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: DETAIL:  Failing row contains (34796245, null, 30fd903d-0869-4787-9743-67aa57e068ca).
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: )
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: pulp [a2b82b5f-0427-4420-919e-b149d00c759e]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: result = func(*args, **kwargs)
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: File "/usr/lib/python3.6/site-packages/pulp_container/app/tasks/synchronize.py", line 44, in synchronize
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: return dv.create()
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: loop.run_until_complete(pipeline)
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: return future.result()
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: await asyncio.gather(*futures)
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: await self.run()
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: File "/usr/lib/python3.6/site-packages/pulp_container/app/tasks/sync_stages.py", line 461, in run
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: BlobManifest.objects.bulk_create(objs=blob_list, ignore_conflicts=True, batch_size=1000)
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: return getattr(self.get_queryset(), name)(*args, **kwargs)
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 474, in bulk_create
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: ids = self._batched_insert(objs_without_pk, fields, batch_size, ignore_conflicts=ignore_conflicts)
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1211, in _batched_insert
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: self._insert(item, fields=fields, using=self.db, ignore_conflicts=ignore_conflicts)
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: return query.get_compiler(using=using).execute_sql(return_id)
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: cursor.execute(sql, params)
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: return executor(sql, params, many, context)
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: return self.cursor.execute(sql, params)
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: raise dj_exc_value.with_traceback(traceback) from exc_value
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
Nov 30 03:44:59 satellite.local pulpcore-worker-3[1231]: return self.cursor.execute(sql, params)

Nov 23 09:42:23 satellite.local pulpcore-worker-5: pulp [65351aa8-eb7b-4167-ab35-57e104b5a339]: pulpcore.tasking.pulpcore_worker:INFO: Task 7d38dc27-fb2e-489a-8895-bc250e59fe4b failed (null value in column "manifest_id" violates not-null constraint
Nov 23 09:42:23 satellite.local pulpcore-worker-5: DETAIL:  Failing row contains (41938, null, 3cdba870-bc13-4637-9e30-ae4a8d93e135).
Nov 23 09:42:23 satellite.local pulpcore-worker-5: )
Nov 23 09:42:23 satellite.local pulpcore-worker-5: pulp [65351aa8-eb7b-4167-ab35-57e104b5a339]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
Nov 23 09:42:23 satellite.local pulpcore-worker-5: result = func(*args, **kwargs)
Nov 23 09:42:23 satellite.local pulpcore-worker-5: File "/usr/lib/python3.6/site-packages/pulp_container/app/tasks/synchronize.py", line 44, in synchronize
Nov 23 09:42:23 satellite.local pulpcore-worker-5: return dv.create()
Nov 23 09:42:23 satellite.local pulpcore-worker-5: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
Nov 23 09:42:23 satellite.local pulpcore-worker-5: loop.run_until_complete(pipeline)
Nov 23 09:42:23 satellite.local pulpcore-worker-5: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Nov 23 09:42:23 satellite.local pulpcore-worker-5: return future.result()
Nov 23 09:42:23 satellite.local pulpcore-worker-5: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Nov 23 09:42:23 satellite.local pulpcore-worker-5: await asyncio.gather(*futures)
Nov 23 09:42:23 satellite.local pulpcore-worker-5: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Nov 23 09:42:23 satellite.local pulpcore-worker-5: await self.run()
Nov 23 09:42:23 satellite.local pulpcore-worker-5: File "/usr/lib/python3.6/site-packages/pulp_container/app/tasks/sync_stages.py", line 461, in run
Nov 23 09:42:23 satellite.local pulpcore-worker-5: BlobManifest.objects.bulk_create(objs=blob_list, ignore_conflicts=True, batch_size=1000)
Nov 23 09:42:23 satellite.local pulpcore-worker-5: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
Nov 23 09:42:23 satellite.local pulpcore-worker-5: return getattr(self.get_queryset(), name)(*args, **kwargs)
Nov 23 09:42:23 satellite.local pulpcore-worker-5: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 474, in bulk_create
Nov 23 09:42:23 satellite.local pulpcore-worker-5: ids = self._batched_insert(objs_without_pk, fields, batch_size, ignore_conflicts=ignore_conflicts)
Nov 23 09:42:23 satellite.local pulpcore-worker-5: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1211, in _batched_insert
Nov 23 09:42:23 satellite.local pulpcore-worker-5: self._insert(item, fields=fields, using=self.db, ignore_conflicts=ignore_conflicts)
Nov 23 09:42:23 satellite.local pulpcore-worker-5: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
Nov 23 09:42:23 satellite.local pulpcore-worker-5: return query.get_compiler(using=using).execute_sql(return_id)
Nov 23 09:42:23 satellite.local pulpcore-worker-5: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
Nov 23 09:42:23 satellite.local pulpcore-worker-5: cursor.execute(sql, params)
Nov 23 09:42:23 satellite.local pulpcore-worker-5: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
Nov 23 09:42:23 satellite.local pulpcore-worker-5: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
Nov 23 09:42:23 satellite.local pulpcore-worker-5: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
Nov 23 09:42:23 satellite.local pulpcore-worker-5: return executor(sql, params, many, context)
Nov 23 09:42:23 satellite.local pulpcore-worker-5: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
Nov 23 09:42:23 satellite.local pulpcore-worker-5: return self.cursor.execute(sql, params)
Nov 23 09:42:23 satellite.local pulpcore-worker-5: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
Nov 23 09:42:23 satellite.local pulpcore-worker-5: raise dj_exc_value.with_traceback(traceback) from exc_value
Nov 23 09:42:23 satellite.local pulpcore-worker-5: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
Nov 23 09:42:23 satellite.local pulpcore-worker-5: return self.cursor.execute(sql, params)

Current planning doc https://hackmd.io/zE3Nctw_QgmwIsiFKXxVgQ