Project

Profile

Help

Issue #7389

closed

Cancelled tasks are processed by worker

Added by osapryki over 3 years ago. Updated over 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 81
Quarter:

Description

If task is in queue and cancelled before being picked up by resource-manager, it may be still scheduled to a worker and processed.

Steps to reproduce.

  1. Start API service. Worker and resource-manager are not running.
  2. Trigger a task by calling enqueue_with_reservation().
  3. Cancel task via API:
curl -u admin:admin -XPATCH -H 'Content-Type: application/json' -d '{"state": "canceled"}' 'http://localhost:5001/pulp/api/v3/tasks/9e9009ea-e95c-40ff-870e-09219db62a05/'
  1. Verify that task is in cancelled state.
  2. Start worker and resource manager.

Actual result:

The task is assigned to worker and is executed with the following warning message logged:

worker_1            | pulp: pulpcore.app.models.task:WARNING: Task __call__() occurred but Task 9e9009ea-e95c-40ff-870e-09219db62a05 is not at WAITING
worker_1            | pulp: rq.worker:INFO: 1@ee04748f4846: Job OK (9e9009ea-e95c-40ff-870e-09219db62a05)
worker_1            | pulp: rq.worker:INFO: 1@ee04748f4846: pulpcore.tasking.tasks._release_resources('9e9009ea-e95c-40ff-870e-09219db62a05') (49d580eb-e483-4df8-b520-296d28fd5185)
worker_1            | pulp: rq.worker:INFO: 1@ee04748f4846: Job OK (49d580eb-e483-4df8-b520-296d28fd5185)

After the task is finished it changes it's state to completed.

Expected result:

Task is not executed.

Actions #1

Updated by fao89 over 3 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 81
Actions #2

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 81 to Sprint 80

fixing Sprint - 80 is current. 81 is reserved for next sprint.

Actions #3

Updated by dalley over 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley
Actions #4

Updated by pulpbot over 3 years ago

  • Status changed from ASSIGNED to POST
Actions #5

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 80 to Sprint 81

Added by dalley over 3 years ago

Revision 7caa50d4 | View on GitHub

Fix canceled tasks being processed by worker

Retain the job id of the scheduling job that runs on the resource manager. Use this ID to cancel the scheduling RQ job also when the task is canceled.

The job that runs on the resource manager has a different RQ job ID than the one which ultimately does the work (which uses the task ID as a job ID) therefore if util.cancel(task_id) is called on a task that only exists on the resource manager queue, canceling the RQ job won't do anything because the job does not yet exist.

Also remove a workaround which is no longer needed.

closes: #7389 https://pulp.plan.io/issues/7389

Actions #6

Updated by dalley over 3 years ago

  • Status changed from POST to MODIFIED
Actions #7

Updated by bmbouter over 3 years ago

  • Sprint/Milestone set to 3.7.0
Actions #8

Updated by pulpbot over 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF