Project

Profile

Help

Issue #489

When an RQ worker dies or restarts, work assigned to that worker is cancelled

Added by bcourt over 4 years ago. Updated 3 months ago.

Status:
NEW
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Severity:
2. Medium
Version:
Platform Release:
Blocks Release:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:

Description

Description of problem:

When an RQ restarts (or dies) the tasks assigned to that worker get marked as cancelled. The cancelled state is correct, but users who dispatch work don't expect random cancelling of tasks so this isn't a great behavior.

To reproduce:
1. Start Pulp system with exactly 1 worker
2. Start a really long sync
3. List the tasks and observe that the task you submitted is in the running state
4. During the sync `kill -9` the worker
5. Start that worker up again
6. List the tasks again and observe that the task you submitted is in the cancelled state

Expected result: That the work you submitted to Pulp doesn't end up in the cancelled state when a worker dies. I expect it to be re-picked up by the worker and be in the running state.


Related issues

Related to Pulp - Task #2527: Ensure Pulp 2.y works with both python-celery 3.1.z and 4.0.z CLOSED - CURRENTRELEASE Actions

History

#1 Updated by bmbouter over 4 years ago

  • Severity changed from High to 3. High

#2 Updated by bmbouter over 4 years ago

  • Subject changed from When a worker dies or restarts the tasks assigned to it are not processed to When a worker dies or restarts, work assigned to that worker is cancelled
  • Description updated (diff)
  • Status changed from ASSIGNED to NEW
  • Assignee deleted (bmbouter)
  • Priority changed from High to Low
  • Severity changed from 3. High to 1. Low
  • Version changed from 2.4.0 to 2.6.0

#3 Updated by bmbouter over 4 years ago

  • Description updated (diff)

#4 Updated by bmbouter over 3 years ago

  • Priority changed from Low to Normal

Raising priority as another Pulp user ran into a problem with this.

Also note, there is a new idea suggested by @asksol. https://github.com/celery/celery/issues/2492#issuecomment-177050069

We should try that.

#5 Updated by bmbouter over 2 years ago

  • Related to Task #2527: Ensure Pulp 2.y works with both python-celery 3.1.z and 4.0.z added

#6 Updated by jsherril@redhat.com over 2 years ago

Katello is okay with this change. The result would be any tasks in katello that are assigned to a worker would no longer show as 'cancell'ed' forcing the user to resume them (or moving them to an errored state). I think this is better behavior and am okay with accepting potential differences in the future.

#7 Updated by bmbouter 6 months ago

  • Status changed from NEW to CLOSED - WONTFIX

#8 Updated by bmbouter 6 months ago

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

#9 Updated by bmbouter 6 months ago

  • Tags Pulp 2 added

#10 Updated by bmbouter 3 months ago

  • Subject changed from When a worker dies or restarts, work assigned to that worker is cancelled to When an RQ worker dies or restarts, work assigned to that worker is cancelled
  • Description updated (diff)
  • Status changed from CLOSED - WONTFIX to NEW
  • Severity changed from 1. Low to 2. Medium
  • Version deleted (2.6.0)
  • Tags deleted (Pulp 2)

Please register to edit this issue

Also available in: Atom PDF