Project

Profile

Help

Issue #6449

closed

Tasks stuck in Waiting state

Added by binlinf0 almost 4 years ago. Updated over 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 81
Quarter:

Description

We have been seeing many waiting tasks. They seem to be stuck forever. e.g.

./get /pulp/api/v3/tasks/14b76b27-9f34-4297-88ed-5ec13cbe5e50/

HTTP/1.1 200 OK Allow: GET, PATCH, DELETE, HEAD, OPTIONS Connection: keep-alive Content-Length: 323 Content-Type: application/json Date: Fri, 03 Apr 2020 12:56:02 GMT Server: nginx/1.16.1 Vary: Accept, Cookie X-Frame-Options: SAMEORIGIN

{ "created_resources": [], "error": null, "finished_at": null, "name": "pulpcore.app.tasks.base.general_update", "progress_reports": [], "pulp_created": "2020-04-02T13:00:14.881212Z", "pulp_href": "/pulp/api/v3/tasks/14b76b27-9f34-4297-88ed-5ec13cbe5e50/", "reserved_resources_record": [], "started_at": null, "state": "waiting", "worker": null }

Per Brian "So the problematic thing I see in this output is the "resource-manager | 0". This tells me that Pulp's record of the task is in postgresql (and was never run), but RQ has lost the task from the "resource-manager" queue in Redis. So the next question is how did that happen?"


Related issues

Related to Pulp - Issue #7119: Tasks stay in waiting state if worker that had resource reservation goneCLOSED - NOTABUGActions
Related to Pulp - Issue #7387: Tasks not delivered to resource-manager are not cleaned upCLOSED - DUPLICATEActions
Has duplicate Pulp - Issue #7387: Tasks not delivered to resource-manager are not cleaned upCLOSED - DUPLICATEActions

Also available in: Atom PDF