Project

Profile

Help

Issue #8603

possible tasking race condition: update or delete on table "core_reservedresource" violates foreign key constraint "core_taskreservedres_resource_id_ee0b7c62_fk_core_rese" on table "core_taskreservedresource"

Added by jsherril@redhat.com 18 days ago. Updated 10 days ago.

Status:
MODIFIED
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
4. Urgent
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 95
Quarter:

Description

With pulpcore 3.11.0

pulp [34656514-03a4-4255-b3b7-8986492b87b1]: rq.worker:ERROR: Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
return self.connection.commit()
psycopg2.errors.ForeignKeyViolation: update or delete on table "core_reservedresource" violates foreign key constraint "core_taskreservedres_resource_id_ee0b7c62_fk_core_rese" on table "core_taskreservedresource"
DETAIL:  Key (pulp_id)=(c0b7db83-401e-4a46-8d9e-07f5ab70de1e) is still referenced from table "core_taskreservedresource".
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/rq/worker.py", line 984, in perform_job
started_job_registry=started_job_registry)
File "/usr/lib/python3.6/site-packages/pulpcore/tasking/worker.py", line 146, in handle_job_success
task.release_resources()
File "/usr/lib/python3.6/site-packages/pulpcore/app/models/task.py", line 430, in release_resources
reservation.delete()
File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
value = super().delete(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 922, in delete
return collector.delete()
File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 318, in delete
sender=model, instance=obj, using=self.using
File "/usr/lib/python3.6/site-packages/django/db/transaction.py", line 240, in __exit__
connection.commit()
File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 262, in commit
self._commit()
File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
return self.connection.commit()
File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
return self.connection.commit()
django.db.utils.IntegrityError: update or delete on table "core_reservedresource" violates foreign key constraint "core_taskreservedres_resource_id_ee0b7c62_fk_core_rese" on table "core_taskreservedresource"
DETAIL:  Key (pulp_id)=(c0b7db83-401e-4a46-8d9e-07f5ab70de1e) is still referenced from table "core_taskreservedresource".
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
return self.connection.commit()
psycopg2.errors.ForeignKeyViolation: update or delete on table "core_reservedresource" violates foreign key constraint "core_taskreservedres_resource_id_ee0b7c62_fk_core_rese" on table "core_taskreservedresource"
DETAIL:  Key (pulp_id)=(c0b7db83-401e-4a46-8d9e-07f5ab70de1e) is still referenced from table "core_taskreservedresource".
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/rq/worker.py", line 984, in perform_job
started_job_registry=started_job_registry)
File "/usr/lib/python3.6/site-packages/pulpcore/tasking/worker.py", line 146, in handle_job_success
task.release_resources()
File "/usr/lib/python3.6/site-packages/pulpcore/app/models/task.py", line 430, in release_resources
reservation.delete()
File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
value = super().delete(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 922, in delete
return collector.delete()
File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 318, in delete
sender=model, instance=obj, using=self.using
File "/usr/lib/python3.6/site-packages/django/db/transaction.py", line 240, in __exit__
connection.commit()
File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 262, in commit
self._commit()
File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
return self.connection.commit()
File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
return self.connection.commit()
django.db.utils.IntegrityError: update or delete on table "core_reservedresource" violates foreign key constraint "core_taskreservedres_resource_id_ee0b7c62_fk_core_rese" on table "core_taskreservedresource"
DETAIL:  Key (pulp_id)=(c0b7db83-401e-4a46-8d9e-07f5ab70de1e) is still referenced from table "core_taskreservedresource".
pulp [None]: rq.worker:INFO: 28341@pipe-up-katello-proxy-nightly-centos7.n60.example.com: a9f5653c-8d52-47ea-b36b-52f86d91e9a3


Related issues

Related to Pulp - Issue #8637: Possible race condition where task's reservations get deletedNEW<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Blocks Pulp - Backport #8632: Backport #8352 to pulp 3.11CLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 5613b5a6 View on GitHub
Added by daviddavis 10 days ago

Handle tasking race condition when cleaning up reservations

fixes #8603

History

#1 Updated by jsherril@redhat.com 18 days ago

  • Description updated (diff)

#2 Updated by daviddavis 18 days ago

I believe I see the race condition. In between when the code checks if the reserved resource has any tasks and when it deletes the resource reservation, it's possible that another task gets associated to the reservation:

https://github.com/pulp/pulpcore/blob/9129c45cd6d7484848abbbcdf4a4314681aa0980/pulpcore/app/models/task.py#L429-L430

#3 Updated by daviddavis 17 days ago

There's another issue here that @x9c4 and I uncovered as well. The TaskReservedResource has a cascade delete on its ReservedResource FK, which could cause a race condition:

  1. Task 1 confirms that the reserved resource has no other tasks
  2. Task 2 reserves the resource
  3. Task 1 calls delete which cascades and deletes Task 2's reservations of the resource

Edit: Filed a separate issue #8637

#4 Updated by fao89 15 days ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 95

#5 Updated by jsherril@redhat.com 11 days ago

  • Severity changed from 2. Medium to 4. Urgent

#6 Updated by dalley 11 days ago

#7 Updated by daviddavis 11 days ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to daviddavis

#8 Updated by daviddavis 11 days ago

  • Related to Issue #8637: Possible race condition where task's reservations get deleted added

#9 Updated by pulpbot 11 days ago

  • Status changed from ASSIGNED to POST

#10 Updated by daviddavis 10 days ago

  • Status changed from POST to MODIFIED

Please register to edit this issue

Also available in: Atom PDF