Issue #8985: deadlock detected during pulp3 to pulp3 sync - RPM Support - Pulp

Actions

Send by e-mail Copy link

Issue #8985

closed

deadlock detected during pulp3 to pulp3 sync

Added by ttereshc over 3 years ago. Updated over 3 years ago.

Status:

CLOSED - DUPLICATE

Priority:

High

Assignee:

Sprint/Milestone:

Start date:

Due date:

Estimated time:

Severity:

3. High

Version:

Platform Release:

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Katello

Sprint:

Sprint 100

Quarter:

Description

It happens during Pulp 3 to Pulp 3 sync. Not clear how reproducible it is. Check the related BZ for repo list

Variations seen

deadlock detected
DETAIL:  Process 21456 waits for ShareLock on transaction 25847; blocked by process 21471.
Process 21471 waits for ShareLock on transaction 25727; blocked by process 21456.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (0,2) in relation "rpm_repometadatafile_data_type_checksum_relat_c9d7364a_uniq"

deadlock detected
DETAIL:  Process 35582 waits for ShareLock on transaction 218181; blocked by process 35563.
Process 35563 waits for ShareLock on transaction 218140; blocked by process 35582.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (32,1) in relation "rpm_package_pkgId_key"

Full traceback for one

pulpcore-worker-2[5987]: pulp [75194b26-9465-42fe-97fc-23b4d0d33c7b]: rq.worker:ERROR: Traceback (most recent call last):
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
 pulpcore-worker-2[5987]: return self.cursor.execute(sql, params)
 pulpcore-worker-2[5987]: psycopg2.errors.DeadlockDetected: deadlock detected
 pulpcore-worker-2[5987]: DETAIL:  Process 8294 waits for ShareLock on transaction 4698; blocked by process 8300.
 pulpcore-worker-2[5987]: Process 8300 waits for ShareLock on transaction 4629; blocked by process 8294.
 pulpcore-worker-2[5987]: HINT:  See server log for query details.
 pulpcore-worker-2[5987]: CONTEXT:  while inserting index tuple (0,2) in relation "rpm_repometadatafile_data_type_checksum_relat_c9d7364a_uniq"
 pulpcore-worker-2[5987]: The above exception was the direct cause of the following exception:
 pulpcore-worker-2[5987]: Traceback (most recent call last):
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 975, in perform_job
 pulpcore-worker-2[5987]: rv = job.perform()
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 696, in perform
 pulpcore-worker-2[5987]: self._result = self._execute()
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 719, in _execute
 pulpcore-worker-2[5987]: return self.func(*self.args, **self.kwargs)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 269, in synchronize
 pulpcore-worker-2[5987]: dv.create()
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 147, in create
 pulpcore-worker-2[5987]: loop.run_until_complete(pipeline)
 pulpcore-worker-2[5987]: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
 pulpcore-worker-2[5987]: return future.result()
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
 pulpcore-worker-2[5987]: await asyncio.gather(*futures)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
 pulpcore-worker-2[5987]: await self.run()
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py", line 95, in run
 pulpcore-worker-2[5987]: d_content.content.save()
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/base.py", line 149, in save
 pulpcore-worker-2[5987]: return super().save(*args, **kwargs)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 134, in save
 pulpcore-worker-2[5987]: save(*args, **kwargs)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 744, in save
 pulpcore-worker-2[5987]: force_update=force_update, update_fields=update_fields)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 782, in save_base
 pulpcore-worker-2[5987]: force_update, using, update_fields,
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 873, in _save_table
 pulpcore-worker-2[5987]: result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 911, in _do_insert
 pulpcore-worker-2[5987]: using=using, raw=raw)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
 pulpcore-worker-2[5987]: return getattr(self.get_queryset(), name)(*args, **kwargs)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
 pulpcore-worker-2[5987]: return query.get_compiler(using=using).execute_sql(return_id)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
 pulpcore-worker-2[5987]: cursor.execute(sql, params)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
 pulpcore-worker-2[5987]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
 pulpcore-worker-2[5987]: return executor(sql, params, many, context)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
 pulpcore-worker-2[5987]: return self.cursor.execute(sql, params)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
pulpcore-worker-2[5987]: raise dj_exc_value.with_traceback(traceback) from exc_value
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
 pulpcore-worker-2[5987]: return self.cursor.execute(sql, params)
 pulpcore-worker-2[5987]: django.db.utils.OperationalError: deadlock detected
 pulpcore-worker-2[5987]: DETAIL:  Process 8294 waits for ShareLock on transaction 4698; blocked by process 8300.
 pulpcore-worker-2[5987]: Process 8300 waits for ShareLock on transaction 4629; blocked by process 8294.
 pulpcore-worker-2[5987]: HINT:  See server log for query details.
 pulpcore-worker-2[5987]: CONTEXT:  while inserting index tuple (0,2) in relation "rpm_repometadatafile_data_type_checksum_relat_c9d7364a_uniq"
 pulpcore-worker-2[5987]: Traceback (most recent call last):
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
 pulpcore-worker-2[5987]: return self.cursor.execute(sql, params)
 pulpcore-worker-2[5987]: psycopg2.errors.DeadlockDetected: deadlock detected
 pulpcore-worker-2[5987]: DETAIL:  Process 8294 waits for ShareLock on transaction 4698; blocked by process 8300.
 pulpcore-worker-2[5987]: Process 8300 waits for ShareLock on transaction 4629; blocked by process 8294.
 pulpcore-worker-2[5987]: HINT:  See server log for query details.
 pulpcore-worker-2[5987]: CONTEXT:  while inserting index tuple (0,2) in relation "rpm_repometadatafile_data_type_checksum_relat_c9d7364a_uniq"
 pulpcore-worker-2[5987]: The above exception was the direct cause of the following exception:
 pulpcore-worker-2[5987]: Traceback (most recent call last):
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 975, in perform_job
 pulpcore-worker-2[5987]: rv = job.perform()
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 696, in perform
 pulpcore-worker-2[5987]: self._result = self._execute()
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 719, in _execute
 pulpcore-worker-2[5987]: return self.func(*self.args, **self.kwargs)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 269, in synchronize
 pulpcore-worker-2[5987]: dv.create()
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 147, in create
 pulpcore-worker-2[5987]: loop.run_until_complete(pipeline)
 pulpcore-worker-2[5987]: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
 pulpcore-worker-2[5987]: return future.result()
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
 pulpcore-worker-2[5987]: await asyncio.gather(*futures)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
 pulpcore-worker-2[5987]: await self.run()
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py", line 95, in run
 pulpcore-worker-2[5987]: d_content.content.save()
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/base.py", line 149, in save
 pulpcore-worker-2[5987]: return super().save(*args, **kwargs)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 134, in save
 pulpcore-worker-2[5987]: save(*args, **kwargs)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 744, in save
 pulpcore-worker-2[5987]: force_update=force_update, update_fields=update_fields)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 782, in save_base
 pulpcore-worker-2[5987]: force_update, using, update_fields,
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 873, in _save_table
 pulpcore-worker-2[5987]: result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 911, in _do_insert
 pulpcore-worker-2[5987]: using=using, raw=raw)
pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
 pulpcore-worker-2[5987]: return getattr(self.get_queryset(), name)(*args, **kwargs)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
 pulpcore-worker-2[5987]: return query.get_compiler(using=using).execute_sql(return_id)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
 pulpcore-worker-2[5987]: cursor.execute(sql, params)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
 pulpcore-worker-2[5987]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
 pulpcore-worker-2[5987]: return executor(sql, params, many, context)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
 pulpcore-worker-2[5987]: return self.cursor.execute(sql, params)
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
 pulpcore-worker-2[5987]: raise dj_exc_value.with_traceback(traceback) from exc_value
 pulpcore-worker-2[5987]: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
 pulpcore-worker-2[5987]: return self.cursor.execute(sql, params)
 pulpcore-worker-2[5987]: django.db.utils.OperationalError: deadlock detected
 pulpcore-worker-2[5987]: DETAIL:  Process 8294 waits for ShareLock on transaction 4698; blocked by process 8300.
 pulpcore-worker-2[5987]: Process 8300 waits for ShareLock on transaction 4629; blocked by process 8294.
 pulpcore-worker-2[5987]: HINT:  See server log for query details.
 pulpcore-worker-2[5987]: CONTEXT:  while inserting index tuple (0,2) in relation "rpm_repometadatafile_data_type_checksum_relat_c9d7364a_uniq"

Related issues