Project

Profile

Help

Maintenance: Planio will be observing a scheduled maintenance window this Sunday, November 10, 2024 from 20:00 UTC until 21:00 UTC to perform important network maintenance in our primary data center. Your Planio account will be unavailable for a few minutes during this maintenance window.

Issue #489

closed

When an task worker dies or restarts, work assigned to that worker is cancelled

Added by bcourt over 9 years ago. Updated almost 3 years ago.

Status:
MODIFIED
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Description of problem:

When an RQ restarts (or dies) the tasks assigned to that worker get marked as cancelled. The cancelled state is correct, but users who dispatch work don't expect random cancelling of tasks so this isn't a great behavior.

To reproduce:
1. Start Pulp system with exactly 1 worker
2. Start a really long sync
3. List the tasks and observe that the task you submitted is in the running state
4. During the sync `kill -9` the worker
5. Start that worker up again
6. List the tasks again and observe that the task you submitted is in the cancelled state

Expected result: That the work you submitted to Pulp doesn't end up in the cancelled state when a worker dies. I expect it to be re-picked up by the worker and be in the running state.


Related issues

Related to Pulp - Task #2527: Ensure Pulp 2.y works with both python-celery 3.1.z and 4.0.zCLOSED - CURRENTRELEASEdalley

Actions
Actions #1

Updated by bmbouter over 9 years ago

  • Severity changed from High to 3. High
Actions #2

Updated by bmbouter over 9 years ago

  • Subject changed from When a worker dies or restarts the tasks assigned to it are not processed to When a worker dies or restarts, work assigned to that worker is cancelled
  • Description updated (diff)
  • Status changed from ASSIGNED to NEW
  • Assignee deleted (bmbouter)
  • Priority changed from High to Low
  • Severity changed from 3. High to 1. Low
  • Version changed from 2.4.0 to 2.6.0
Actions #3

Updated by bmbouter over 9 years ago

  • Description updated (diff)
Actions #4

Updated by bmbouter over 8 years ago

  • Priority changed from Low to Normal

Raising priority as another Pulp user ran into a problem with this.

Also note, there is a new idea suggested by @asksol. https://github.com/celery/celery/issues/2492#issuecomment-177050069

We should try that.

Actions #5

Updated by bmbouter almost 8 years ago

  • Related to Task #2527: Ensure Pulp 2.y works with both python-celery 3.1.z and 4.0.z added
Actions #6

Updated by jsherril@redhat.com over 7 years ago

Katello is okay with this change. The result would be any tasks in katello that are assigned to a worker would no longer show as 'cancell'ed' forcing the user to resume them (or moving them to an errored state). I think this is better behavior and am okay with accepting potential differences in the future.

Actions #7

Updated by bmbouter over 5 years ago

  • Status changed from NEW to CLOSED - WONTFIX
Actions #8

Updated by bmbouter over 5 years ago

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

Actions #9

Updated by bmbouter over 5 years ago

  • Tags Pulp 2 added
Actions #10

Updated by bmbouter over 5 years ago

  • Subject changed from When a worker dies or restarts, work assigned to that worker is cancelled to When an RQ worker dies or restarts, work assigned to that worker is cancelled
  • Description updated (diff)
  • Status changed from CLOSED - WONTFIX to NEW
  • Severity changed from 1. Low to 2. Medium
  • Version deleted (2.6.0)
  • Tags deleted (Pulp 2)
Actions #11

Updated by dalley about 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley
Actions #12

Updated by dalley about 4 years ago

  • Status changed from ASSIGNED to CLOSED - CURRENTRELEASE
Actions #13

Updated by bmbouter about 4 years ago

  • Status changed from CLOSED - CURRENTRELEASE to NEW

I'm pretty confident this is still an issue. When a worker is noticed missing, my understanding is its tasks are cancelled. dalley what do you believe occurs in the die/restart case? I'm reopening it because I think it's broken, but dalley let me know if what I'm doing here isn't right.

Actions #14

Updated by dalley over 3 years ago

  • Subject changed from When an RQ worker dies or restarts, work assigned to that worker is cancelled to When an task worker dies or restarts, work assigned to that worker is cancelled
  • Assignee deleted (dalley)

Added by ttereshc almost 3 years ago

Revision d00d7d67 | View on GitHub

Added forgotten changelog entry for 0.11.8

For PR https://github.com/pulp/pulp-2to3-migration/pull/482

closes #489

Actions #15

Updated by ttereshc almost 3 years ago

  • Status changed from NEW to MODIFIED

Also available in: Atom PDF