Project

Profile

Help

Issue #8637

closed

Issue #8912: [EPIC] Issues with the traditional tasking system

Possible race condition where task's reservations get deleted

Added by daviddavis almost 3 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 99
Quarter:

Description

This was something I noticed while working on #8603. I haven't actually reproduced it so it's purely theoretical.

The TaskReservedResource has a cascade delete on its ReservedResource FK, which could cause a race condition:

  1. Task 1 confirms that the reserved resource has no other tasks
  2. Task 2 reserves the resource
  3. Task 1 calls delete which cascades and deletes Task 2's reservations of the resource

Related issues

Related to Pulp - Issue #8603: possible tasking race condition: update or delete on table "core_reservedresource" violates foreign key constraint "core_taskreservedres_resource_id_ee0b7c62_fk_core_rese" on table "core_taskreservedresource" CLOSED - CURRENTRELEASEdaviddavisActions
Related to Pulp - Issue #8568: Overall task status remains in 'running' state while all the reports are in the final stateCLOSED - WONTFIXActions
Actions #1

Updated by daviddavis almost 3 years ago

  • Related to Issue #8603: possible tasking race condition: update or delete on table "core_reservedresource" violates foreign key constraint "core_taskreservedres_resource_id_ee0b7c62_fk_core_rese" on table "core_taskreservedresource" added
Actions #2

Updated by daviddavis almost 3 years ago

  • Description updated (diff)
Actions #3

Updated by dalley almost 3 years ago

  • Related to Issue #8568: Overall task status remains in 'running' state while all the reports are in the final state added
Actions #4

Updated by fao89 almost 3 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 95
Actions #5

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 95 to Sprint 96
Actions #6

Updated by daviddavis almost 3 years ago

  • Category set to 29
Actions #7

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 96 to Sprint 97
Actions #8

Updated by dalley almost 3 years ago

  • Priority changed from Normal to High
Actions #9

Updated by dalley almost 3 years ago

  • Severity changed from 2. Medium to 3. High
Actions #10

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 97 to Sprint 98
Actions #11

Updated by mdellweg almost 3 years ago

Is the cascaded delete needed or even useful here? Isn't the cleanup process, that the association (TaskReservedResource) is deleted first and the then the ReservedResource is cleaned up if no longer needed? If we change the cascade to protect here, it should be fine, right? And if another task is assigned to the Resource in the meantime, we get a ConsistencyException we can simply ignore.

Actions #12

Updated by daviddavis almost 3 years ago

Yup, that's my thinking as well.

Actions #13

Updated by bmbouter almost 3 years ago

I want to confirm the thinking here. When you say "the cleanup process, that the association (TaskReservedResource) is deleted first and the then the ReservedResource is cleaned up " you mean here right?.

If that's the case then I agree the cascade delete isn't needed. So say we got rid of the cascade delete ... couldn't this race condition still happen just between these two lines?.

Actions #14

Updated by daviddavis almost 3 years ago

bmbouter wrote:

I want to confirm the thinking here. When you say "the cleanup process, that the association (TaskReservedResource) is deleted first and the then the ReservedResource is cleaned up " you mean here right?.

If that's the case then I agree the cascade delete isn't needed. So say we got rid of the cascade delete ... couldn't this race condition still happen just between these two lines?.

If you go back to my example, then when task 1 goes to clean up the ReservedResource, then a cascade protect from task 2 will prevent the ReservedResource from being deleted. This would avoid the race condition.

Actions #15

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 98 to Sprint 99
Actions #16

Updated by bmbouter almost 3 years ago

Ah this does make sense @david. +1 to this plan, let's change this line to PROTECT.

Actions #17

Updated by daviddavis almost 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to daviddavis
Actions #18

Updated by pulpbot almost 3 years ago

  • Status changed from ASSIGNED to POST
Actions #19

Updated by bmbouter almost 3 years ago

  • Parent issue set to #8912

Added by daviddavis almost 3 years ago

Revision cf2cde7e | View on GitHub

Fix race condition during reserved resource cleanup

fixes #8637

Actions #20

Updated by daviddavis almost 3 years ago

  • Status changed from POST to MODIFIED
Actions #21

Updated by pulpbot over 2 years ago

  • Sprint/Milestone set to 3.14.0
Actions #22

Updated by pulpbot over 2 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF