Project

Profile

Help

Issue #2770

closed

Tasks stuck in waiting state if received while qpidd is down

Added by bmbouter over 7 years ago. Updated over 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
2.13.2
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 20
Quarter:

Description

To reproduce:

1. Start a normal, healthy Pulp system.
2. sync the zoo repo to make sure your system is health. Ensure this completes
3. Observe that there are 0 currently running tasks using pulp-admin tasks list
4. Stop qpidd or rabbitmq
5. sync the zoo repo and you'll receive this error (which could be improved, but not what this bug is about)

[vagrant@dev ~]$ pulp-admin rpm repo sync run --repo-id zoo
+----------------------------------------------------------------------+
                     Synchronizing Repository [zoo]
+----------------------------------------------------------------------+

An internal error occurred on the Pulp server:

RequestException: POST request
on /pulp/api/v2/repositories/zoo/actions/sync/ failed with 500 - [Errno 111]
Connection refused

6. List the tasks with pulp-admin tasks list and observe that a task was created that can never be run!

Operations:  sync
Resources:   zoo (repository)
State:       Waiting
Start Time:  Unstarted
Finish Time: Incomplete
Task Id:     8e47a36a-2767-48a8-b6f6-e497d6f5cdec

The task above ^ will incorrectly reserve the worker it is assigned to and will incorrectly show as Waiting when it is never run. It will stay this way until the worker itself is restarted at which point the cleanup logic will make everything right again.

Also available in: Atom PDF