Project

Profile

Help

Issue #9441

deadlocks from touch() in several paths

Added by ggainey about 1 month ago. Updated 22 days ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 106
Quarter:

Description

As noted from RHUI4, here: https://pulp.plan.io/issues/8750#note-37

Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]: pulp [62672f3c1f5844b29a8c0869db07cf4c]: pulpcore.tasking.pulpcore_worker:INFO: Task a2d9b01f-3a62-4f35-8ad6-9b793c43a9c1 failed (deadlock detected
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]: DETAIL:  Process 244121 waits for ShareLock on transaction 2141154; blocked by process 244124.
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]: Process 244124 waits for ShareLock on transaction 2141157; blocked by process 244121.
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]: HINT:  See server log for query details.
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]: CONTEXT:  while updating tuple (24547,1) in relation "core_artifact"
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]: )
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]: pulp [62672f3c1f5844b29a8c0869db07cf4c]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 272, in _perform_task
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    result = func(*args, **kwargs)
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 475, in synchronize
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    version = dv.create()
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    loop.run_until_complete(pipeline)
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    return future.result()
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    await asyncio.gather(*futures)
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    await self.run()
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 89, in run
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    existing_artifacts.touch()
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/pulpcore/app/models/content.py", line 103, in touch
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    return self.update(timestamp_of_interest=now())
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 741, in update
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    rows = query.get_compiler(self.db).execute_sql(CURSOR)
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1471, in execute_sql
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    cursor = super().execute_sql(result_type)
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    cursor.execute(sql, params)
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    return executor(sql, params, many, context)
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    return self.cursor.execute(sql, params)
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    raise dj_exc_value.with_traceback(traceback) from exc_value
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    return self.cursor.execute(sql, params)

Related issues

Copied to Pulp - Backport #9445: Backport request to 3.14.z: deadlocks from touch() in several pathsCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision cd9b5785 View on GitHub
Added by ggainey 29 days ago

Teach the touch() path to apply order to avoid deadlocks.

fixes #9441. [nocoverage] Required PR: https://github.com/pulp/pulpcore/pull/1637

History

#1 Updated by ggainey about 1 month ago

  • Copied to Backport #9445: Backport request to 3.14.z: deadlocks from touch() in several paths added

#2 Updated by pulpbot about 1 month ago

  • Status changed from NEW to POST

#3 Updated by dkliban@redhat.com 29 days ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 106

#4 Updated by ggainey 29 days ago

  • Status changed from POST to MODIFIED

#5 Updated by pulpbot 22 days ago

  • Sprint/Milestone set to 3.16.0

#6 Updated by pulpbot 22 days ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF