Actions
Issue #8985
closed
deadlock detected during pulp3 to pulp3 sync
Status:
CLOSED - DUPLICATE
Priority:
High
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 100
Quarter:
Description
It happens during Pulp 3 to Pulp 3 sync. Not clear how reproducible it is. Check the related BZ for repo list
Variations seen
deadlock detected
DETAIL: Process 21456 waits for ShareLock on transaction 25847; blocked by process 21471.
Process 21471 waits for ShareLock on transaction 25727; blocked by process 21456.
HINT: See server log for query details.
CONTEXT: while inserting index tuple (0,2) in relation "rpm_repometadatafile_data_type_checksum_relat_c9d7364a_uniq"
deadlock detected
DETAIL: Process 35582 waits for ShareLock on transaction 218181; blocked by process 35563.
Process 35563 waits for ShareLock on transaction 218140; blocked by process 35582.
HINT: See server log for query details.
CONTEXT: while inserting index tuple (32,1) in relation "rpm_package_pkgId_key"
Full traceback for one
pulpcore-worker-2[5987]: pulp [75194b26-9465-42fe-97fc-23b4d0d33c7b]: rq.worker:ERROR: Traceback (most recent call last):
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-worker-2[5987]: return self.cursor.execute(sql, params)
pulpcore-worker-2[5987]: psycopg2.errors.DeadlockDetected: deadlock detected
pulpcore-worker-2[5987]: DETAIL: Process 8294 waits for ShareLock on transaction 4698; blocked by process 8300.
pulpcore-worker-2[5987]: Process 8300 waits for ShareLock on transaction 4629; blocked by process 8294.
pulpcore-worker-2[5987]: HINT: See server log for query details.
pulpcore-worker-2[5987]: CONTEXT: while inserting index tuple (0,2) in relation "rpm_repometadatafile_data_type_checksum_relat_c9d7364a_uniq"
pulpcore-worker-2[5987]: The above exception was the direct cause of the following exception:
pulpcore-worker-2[5987]: Traceback (most recent call last):
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 975, in perform_job
pulpcore-worker-2[5987]: rv = job.perform()
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 696, in perform
pulpcore-worker-2[5987]: self._result = self._execute()
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 719, in _execute
pulpcore-worker-2[5987]: return self.func(*self.args, **self.kwargs)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 269, in synchronize
pulpcore-worker-2[5987]: dv.create()
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 147, in create
pulpcore-worker-2[5987]: loop.run_until_complete(pipeline)
pulpcore-worker-2[5987]: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
pulpcore-worker-2[5987]: return future.result()
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
pulpcore-worker-2[5987]: await asyncio.gather(*futures)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
pulpcore-worker-2[5987]: await self.run()
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py", line 95, in run
pulpcore-worker-2[5987]: d_content.content.save()
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/base.py", line 149, in save
pulpcore-worker-2[5987]: return super().save(*args, **kwargs)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 134, in save
pulpcore-worker-2[5987]: save(*args, **kwargs)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 744, in save
pulpcore-worker-2[5987]: force_update=force_update, update_fields=update_fields)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 782, in save_base
pulpcore-worker-2[5987]: force_update, using, update_fields,
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 873, in _save_table
pulpcore-worker-2[5987]: result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 911, in _do_insert
pulpcore-worker-2[5987]: using=using, raw=raw)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
pulpcore-worker-2[5987]: return getattr(self.get_queryset(), name)(*args, **kwargs)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
pulpcore-worker-2[5987]: return query.get_compiler(using=using).execute_sql(return_id)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
pulpcore-worker-2[5987]: cursor.execute(sql, params)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
pulpcore-worker-2[5987]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
pulpcore-worker-2[5987]: return executor(sql, params, many, context)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-worker-2[5987]: return self.cursor.execute(sql, params)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
pulpcore-worker-2[5987]: raise dj_exc_value.with_traceback(traceback) from exc_value
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-worker-2[5987]: return self.cursor.execute(sql, params)
pulpcore-worker-2[5987]: django.db.utils.OperationalError: deadlock detected
pulpcore-worker-2[5987]: DETAIL: Process 8294 waits for ShareLock on transaction 4698; blocked by process 8300.
pulpcore-worker-2[5987]: Process 8300 waits for ShareLock on transaction 4629; blocked by process 8294.
pulpcore-worker-2[5987]: HINT: See server log for query details.
pulpcore-worker-2[5987]: CONTEXT: while inserting index tuple (0,2) in relation "rpm_repometadatafile_data_type_checksum_relat_c9d7364a_uniq"
pulpcore-worker-2[5987]: Traceback (most recent call last):
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-worker-2[5987]: return self.cursor.execute(sql, params)
pulpcore-worker-2[5987]: psycopg2.errors.DeadlockDetected: deadlock detected
pulpcore-worker-2[5987]: DETAIL: Process 8294 waits for ShareLock on transaction 4698; blocked by process 8300.
pulpcore-worker-2[5987]: Process 8300 waits for ShareLock on transaction 4629; blocked by process 8294.
pulpcore-worker-2[5987]: HINT: See server log for query details.
pulpcore-worker-2[5987]: CONTEXT: while inserting index tuple (0,2) in relation "rpm_repometadatafile_data_type_checksum_relat_c9d7364a_uniq"
pulpcore-worker-2[5987]: The above exception was the direct cause of the following exception:
pulpcore-worker-2[5987]: Traceback (most recent call last):
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 975, in perform_job
pulpcore-worker-2[5987]: rv = job.perform()
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 696, in perform
pulpcore-worker-2[5987]: self._result = self._execute()
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 719, in _execute
pulpcore-worker-2[5987]: return self.func(*self.args, **self.kwargs)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 269, in synchronize
pulpcore-worker-2[5987]: dv.create()
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 147, in create
pulpcore-worker-2[5987]: loop.run_until_complete(pipeline)
pulpcore-worker-2[5987]: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
pulpcore-worker-2[5987]: return future.result()
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
pulpcore-worker-2[5987]: await asyncio.gather(*futures)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
pulpcore-worker-2[5987]: await self.run()
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py", line 95, in run
pulpcore-worker-2[5987]: d_content.content.save()
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/base.py", line 149, in save
pulpcore-worker-2[5987]: return super().save(*args, **kwargs)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 134, in save
pulpcore-worker-2[5987]: save(*args, **kwargs)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 744, in save
pulpcore-worker-2[5987]: force_update=force_update, update_fields=update_fields)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 782, in save_base
pulpcore-worker-2[5987]: force_update, using, update_fields,
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 873, in _save_table
pulpcore-worker-2[5987]: result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 911, in _do_insert
pulpcore-worker-2[5987]: using=using, raw=raw)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
pulpcore-worker-2[5987]: return getattr(self.get_queryset(), name)(*args, **kwargs)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
pulpcore-worker-2[5987]: return query.get_compiler(using=using).execute_sql(return_id)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
pulpcore-worker-2[5987]: cursor.execute(sql, params)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
pulpcore-worker-2[5987]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
pulpcore-worker-2[5987]: return executor(sql, params, many, context)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-worker-2[5987]: return self.cursor.execute(sql, params)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
pulpcore-worker-2[5987]: raise dj_exc_value.with_traceback(traceback) from exc_value
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
pulpcore-worker-2[5987]: return self.cursor.execute(sql, params)
pulpcore-worker-2[5987]: django.db.utils.OperationalError: deadlock detected
pulpcore-worker-2[5987]: DETAIL: Process 8294 waits for ShareLock on transaction 4698; blocked by process 8300.
pulpcore-worker-2[5987]: Process 8300 waits for ShareLock on transaction 4629; blocked by process 8294.
pulpcore-worker-2[5987]: HINT: See server log for query details.
pulpcore-worker-2[5987]: CONTEXT: while inserting index tuple (0,2) in relation "rpm_repometadatafile_data_type_checksum_relat_c9d7364a_uniq"
Related issues
Actions
.