Project

Profile

Help

Issue #7907

closed

Failed task did not clean up properly resource reservations

Added by osapryki almost 2 years ago. Updated almost 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 86
Quarter:

Description

In Automation Hub the task galaxy_ng.app.tasks.synclist.curate_synclist_repository failed due to redis failure. However resource reservation for that task remained in the database blocking entire tasking system (if workers number = 1).

https://gist.github.com/cutwater/4ec7960f0eac2793ca17a78723dca75d

Environment:

pulpcore 3.7.1

pulp-ansible 0.4.3

galaxy-ng 1326eb5f1679880b68e05a48d4377def7c72a95b

Workers number: 1

Analysis

After review, the failure scenario goes like this:

  1. The tasking code itself runs to completion
  2. RQ attempts to notify Redis the task is completed (in the RQ registry) in its (handle_job_success)[https://github.com/rq/rq/blob/master/rq/worker.py#L925].
  3. Interacting with Redis raised an exception at this line: https://github.com/rq/rq/blob/master/rq/worker.py#L932
  4. This fatal exception raised and handled by Pulp's handle_job_failure handler implementation which records the exception (how we know this) and also marks the task as failed
  5. Also when Redis became unavailable, it forgot the tasks it was storing in memory which includes the _release_resources that pairs with the now failed task and is intended to release the locks
  6. The worker never died so other lock cleanup processes never occurred.
  7. Tasks backup and eventually a sysadmin restarts the processes
  8. The cleanup code in mark_worker_offline is triggered, but since the task is already at FAILED, this line does not issue it's cancellation which would release the locks
  9. The locks are never released....

Solution

Add in code to mark_worker_offline that will ensure all locks for a worker being cleaned up are released even if the a task failed and its _release_resources was never delivered. This should occur after the cancellation for all tasks in "completed" states.


Related issues

Related to Pulp - Issue #7386: Task that does not exist in worker or resource-manager are never cleaned upCLOSED - WONTFIXActions
Related to Pulp - Backport #7951: Backport 7907CLOSED - WONTFIX

Actions
Copied to Pulp - Backport #9547: Backport "Failed task did not clean up properly resource reservations" to 3.7.zCLOSED - CURRENTRELEASEbmbouter

Actions
Actions #1

Updated by fao89 almost 2 years ago

  • Related to Issue #7386: Task that does not exist in worker or resource-manager are never cleaned up added
Actions #2

Updated by fao89 almost 2 years ago

  • Priority changed from Normal to High
  • Triaged changed from No to Yes
Actions #3

Updated by bmbouter almost 2 years ago

  • Subject changed from Failed curate_synclist_repository task did not clean up properly resource reservations to Failed task did not clean up properly resource reservations
  • Description updated (diff)
Actions #4

Updated by bmbouter almost 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bmbouter
Actions #5

Updated by dkliban@redhat.com almost 2 years ago

  • Sprint set to Sprint 86
Actions #6

Updated by pulpbot almost 2 years ago

  • Status changed from ASSIGNED to POST

Added by bmbouter almost 2 years ago

Revision 516df314

Adds additional lock cleanup to worker cleanup

As another layer of security to guard against lock cleanup not occurring due to Redis not delivering the _release_resource task, ensure all locks are also cleaned up even for tasks that are in their final states.

closes #7907

Actions #7

Updated by bmbouter almost 2 years ago

  • Status changed from POST to MODIFIED
Actions #8

Updated by daviddavis almost 2 years ago

  • Sprint/Milestone set to 3.9.0
Actions #9

Updated by pulpbot almost 2 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions #10

Updated by daviddavis almost 2 years ago

Actions #11

Updated by ttereshc about 1 year ago

  • Copied to Backport #9547: Backport "Failed task did not clean up properly resource reservations" to 3.7.z added

Also available in: Atom PDF