Issue #489
closed
When an task worker dies or restarts, work assigned to that worker is cancelled
Description
Description of problem:
When an RQ restarts (or dies) the tasks assigned to that worker get marked as cancelled. The cancelled state is correct, but users who dispatch work don't expect random cancelling of tasks so this isn't a great behavior.
To reproduce:
1. Start Pulp system with exactly 1 worker
2. Start a really long sync
3. List the tasks and observe that the task you submitted is in the running state
4. During the sync `kill -9` the worker
5. Start that worker up again
6. List the tasks again and observe that the task you submitted is in the cancelled state
Expected result: That the work you submitted to Pulp doesn't end up in the cancelled state when a worker dies. I expect it to be re-picked up by the worker and be in the running state.
- Severity changed from High to 3. High
- Subject changed from When a worker dies or restarts the tasks assigned to it are not processed to When a worker dies or restarts, work assigned to that worker is cancelled
- Description updated (diff)
- Status changed from ASSIGNED to NEW
- Assignee deleted (
bmbouter)
- Priority changed from High to Low
- Severity changed from 3. High to 1. Low
- Version changed from 2.4.0 to 2.6.0
- Description updated (diff)
- Priority changed from Low to Normal
- Related to Task #2527: Ensure Pulp 2.y works with both python-celery 3.1.z and 4.0.z added
Katello is okay with this change. The result would be any tasks in katello that are assigned to a worker would no longer show as 'cancell'ed' forcing the user to resume them (or moving them to an errored state). I think this is better behavior and am okay with accepting potential differences in the future.
- Status changed from NEW to CLOSED - WONTFIX
Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.
- Subject changed from When a worker dies or restarts, work assigned to that worker is cancelled to When an RQ worker dies or restarts, work assigned to that worker is cancelled
- Description updated (diff)
- Status changed from CLOSED - WONTFIX to NEW
- Severity changed from 1. Low to 2. Medium
- Version deleted (
2.6.0)
- Tags deleted (
Pulp 2)
- Status changed from NEW to ASSIGNED
- Assignee set to dalley
- Status changed from ASSIGNED to CLOSED - CURRENTRELEASE
- Status changed from CLOSED - CURRENTRELEASE to NEW
I'm pretty confident this is still an issue. When a worker is noticed missing, my understanding is its tasks are cancelled. dalley what do you believe occurs in the die/restart case? I'm reopening it because I think it's broken, but dalley let me know if what I'm doing here isn't right.
- Subject changed from When an RQ worker dies or restarts, work assigned to that worker is cancelled to When an task worker dies or restarts, work assigned to that worker is cancelled
- Assignee deleted (
dalley)
- Status changed from NEW to MODIFIED
Also available in: Atom
PDF
Added forgotten changelog entry for 0.11.8
For PR https://github.com/pulp/pulp-2to3-migration/pull/482
closes #489