Project

Profile

Help

Issue #7389

Cancelled tasks are processed by worker

Added by osapryki about 2 months ago. Updated 28 days ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 81
Quarter:

Description

If task is in queue and cancelled before being picked up by resource-manager, it may be still scheduled to a worker and processed.

Steps to reproduce.

  1. Start API service. Worker and resource-manager are not running.
  2. Trigger a task by calling enqueue_with_reservation().
  3. Cancel task via API:
curl -u admin:admin -XPATCH -H 'Content-Type: application/json' -d '{"state": "canceled"}' 'http://localhost:5001/pulp/api/v3/tasks/9e9009ea-e95c-40ff-870e-09219db62a05/'
  1. Verify that task is in cancelled state.
  2. Start worker and resource manager.

Actual result:

The task is assigned to worker and is executed with the following warning message logged:

worker_1            | pulp: pulpcore.app.models.task:WARNING: Task __call__() occurred but Task 9e9009ea-e95c-40ff-870e-09219db62a05 is not at WAITING
worker_1            | pulp: rq.worker:INFO: 1@ee04748f4846: Job OK (9e9009ea-e95c-40ff-870e-09219db62a05)
worker_1            | pulp: rq.worker:INFO: 1@ee04748f4846: pulpcore.tasking.tasks._release_resources('9e9009ea-e95c-40ff-870e-09219db62a05') (49d580eb-e483-4df8-b520-296d28fd5185)
worker_1            | pulp: rq.worker:INFO: 1@ee04748f4846: Job OK (49d580eb-e483-4df8-b520-296d28fd5185)

After the task is finished it changes it's state to completed.

Expected result:

Task is not executed.

Associated revisions

Revision 7caa50d4 View on GitHub
Added by dalley about 2 months ago

Fix canceled tasks being processed by worker

Retain the job id of the scheduling job that runs on the resource manager. Use this ID to cancel the scheduling RQ job also when the task is canceled.

The job that runs on the resource manager has a different RQ job ID than the one which ultimately does the work (which uses the task ID as a job ID) therefore if util.cancel(task_id) is called on a task that only exists on the resource manager queue, canceling the RQ job won't do anything because the job does not yet exist.

Also remove a workaround which is no longer needed.

closes: #7389 https://pulp.plan.io/issues/7389

History

#1 Updated by fao89 about 2 months ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 81

#2 Updated by rchan about 2 months ago

  • Sprint changed from Sprint 81 to Sprint 80

fixing Sprint - 80 is current. 81 is reserved for next sprint.

#3 Updated by dalley about 2 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

#4 Updated by pulpbot about 2 months ago

  • Status changed from ASSIGNED to POST

#5 Updated by rchan about 2 months ago

  • Sprint changed from Sprint 80 to Sprint 81

#6 Updated by dalley about 2 months ago

  • Status changed from POST to MODIFIED

#7 Updated by bmbouter 28 days ago

  • Sprint/Milestone set to 3.7.0

#8 Updated by pulpbot 28 days ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF