Actions
Issue #7389
closedCancelled tasks are processed by worker
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 81
Quarter:
Description
If task is in queue and cancelled before being picked up by resource-manager, it may be still scheduled to a worker and processed.
Steps to reproduce.
- Start API service. Worker and resource-manager are not running.
- Trigger a task by calling
enqueue_with_reservation()
. - Cancel task via API:
curl -u admin:admin -XPATCH -H 'Content-Type: application/json' -d '{"state": "canceled"}' 'http://localhost:5001/pulp/api/v3/tasks/9e9009ea-e95c-40ff-870e-09219db62a05/'
- Verify that task is in
cancelled
state. - Start worker and resource manager.
Actual result:
The task is assigned to worker and is executed with the following warning message logged:
worker_1 | pulp: pulpcore.app.models.task:WARNING: Task __call__() occurred but Task 9e9009ea-e95c-40ff-870e-09219db62a05 is not at WAITING
worker_1 | pulp: rq.worker:INFO: 1@ee04748f4846: Job OK (9e9009ea-e95c-40ff-870e-09219db62a05)
worker_1 | pulp: rq.worker:INFO: 1@ee04748f4846: pulpcore.tasking.tasks._release_resources('9e9009ea-e95c-40ff-870e-09219db62a05') (49d580eb-e483-4df8-b520-296d28fd5185)
worker_1 | pulp: rq.worker:INFO: 1@ee04748f4846: Job OK (49d580eb-e483-4df8-b520-296d28fd5185)
After the task is finished it changes it's state to completed
.
Expected result:
Task is not executed.
Actions
Fix canceled tasks being processed by worker
Retain the job id of the scheduling job that runs on the resource manager. Use this ID to cancel the scheduling RQ job also when the task is canceled.
The job that runs on the resource manager has a different RQ job ID than the one which ultimately does the work (which uses the task ID as a job ID) therefore if util.cancel(task_id) is called on a task that only exists on the resource manager queue, canceling the RQ job won't do anything because the job does not yet exist.
Also remove a workaround which is no longer needed.
closes: #7389 https://pulp.plan.io/issues/7389