Project

Profile

Help

Issue #489

Updated by bmbouter over 5 years ago

Description of problem:

When a worker restarts (or dies) the tasks assigned to that worker get marked as cancelled. The cancelled state is correct, but users who dispatch work don't expect random cancelling of tasks so this isn't a great behavior. This affects a minimal number of tasks so it's are not a big deal, but fixing it should happen processed. Previously, in Celery not Pulp.

To reproduce:
1. Start
pulp 2.3.x, tasks would be restarted when Pulp system with exactly 1 worker
2. Start a really long sync
3. List
restarted. In 2.4 the tasks and observe that the task you submitted queue is in being cleared when the running state
4. During
worker stops.

If
the sync `kill -9` worker dies unexpectedly or the worker
5. Start that worker
power goes this can end up again
6. List the
with tasks again and observe being listed as "NOT_STARTED" or "IN_PROGRESS" but with a reservation that will prevent other tasks that require that resource from occurring until the task you submitted tasks are manually canceled by the user.

How reproducible:
Every time (this behavior
is by design in the cancelled state 2.4.0)

Expected result: That the work you submitted Steps to Reproduce:
1. Queue a bunch of tasks
2. Restart
Pulp doesn't end up

Actual results:
Tasks are either canceled or hang
in the cancelled waiting/running state when a worker dies. I expect it to be re-picked up by

Expected results:
Tasks are processed or restarted after
the worker and be in the running state. restart.

The root cause is because the CELERY_WORKER_DIRECT feature doesn't use durable queues. An upstream + This bug [0] has been filed, and this bug tracks the inclusion of an upstream fix into Pulp. was cloned from "Bugzilla Bug #1129758":https://bugzilla.redhat.com/show_bug.cgi?id=1129758 +

Back