Project

Profile

Help

Backport #9445

closed

Backport request to 3.14.z: deadlocks from touch() in several paths

Added by ggainey over 2 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

0%

Estimated time:
Triaged:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

As noted from RHUI4, here: https://pulp.plan.io/issues/8750#note-37

Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]: pulp [62672f3c1f5844b29a8c0869db07cf4c]: pulpcore.tasking.pulpcore_worker:INFO: Task a2d9b01f-3a62-4f35-8ad6-9b793c43a9c1 failed (deadlock detected
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]: DETAIL:  Process 244121 waits for ShareLock on transaction 2141154; blocked by process 244124.
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]: Process 244124 waits for ShareLock on transaction 2141157; blocked by process 244121.
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]: HINT:  See server log for query details.
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]: CONTEXT:  while updating tuple (24547,1) in relation "core_artifact"
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]: )
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]: pulp [62672f3c1f5844b29a8c0869db07cf4c]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 272, in _perform_task
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    result = func(*args, **kwargs)
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 475, in synchronize
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    version = dv.create()
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    loop.run_until_complete(pipeline)
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    return future.result()
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    await asyncio.gather(*futures)
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    await self.run()
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 89, in run
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    existing_artifacts.touch()
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/pulpcore/app/models/content.py", line 103, in touch
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    return self.update(timestamp_of_interest=now())
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 741, in update
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    rows = query.get_compiler(self.db).execute_sql(CURSOR)
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1471, in execute_sql
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    cursor = super().execute_sql(result_type)
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    cursor.execute(sql, params)
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    return executor(sql, params, many, context)
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    return self.cursor.execute(sql, params)
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    raise dj_exc_value.with_traceback(traceback) from exc_value
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:  File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
Sep 15 06:42:11 pulp314-machine pulpcore-worker[49536]:    return self.cursor.execute(sql, params)

Related issues

Copied from Pulp - Issue #9441: deadlocks from touch() in several pathsCLOSED - CURRENTRELEASEggaineyActions
Actions #1

Updated by ggainey over 2 years ago

  • Copied from Issue #9441: deadlocks from touch() in several paths added
Actions #3

Updated by dalley over 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee changed from ggainey to dalley
Actions #4

Updated by dalley over 2 years ago

  • Sprint/Milestone set to 3.14.7

Added by dalley over 2 years ago

Revision 69b2e1dc | View on GitHub

Teach the touch() path to apply order to avoid deadlocks.

backports #9441. [nocoverage] Required PR: https://github.com/pulp/pulpcore/pull/1637

fixes #9445

(cherry picked from commit cd9b5785279582297030007f559958829a927084)

Actions #5

Updated by dalley over 2 years ago

  • Status changed from ASSIGNED to MODIFIED
Actions #6

Updated by pulpbot over 2 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions #7

Updated by ttereshc over 2 years ago

  • Tracker changed from Issue to Backport
  • % Done set to 0
  • Severity deleted (3. High)
  • Groomed deleted (No)

Also available in: Atom PDF