Project

Profile

Help

Task #2430

Keep Worker model records around instead of deleting them

Added by bmbouter over 3 years ago. Updated 3 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:

Description

Currently Pulp worker DB records are deleted when workers go missing or are shut down, and created when they are first observed or updated. It would be beneficial to keep these records. This allows someone to read these records post-mortem. It has a tertiary benefit that Tasks won't be cascade deleted due to the Worker objects being removed.

To resolve this, introduce a boolean named 'online' on the Worker model. The code in pulp_celerybeat that manages these Worker records would be updated to set the 'online' value when workers start and stop. The field's default should correspond with its usage.

Currently on 3.0-dev the pulp_celerybeat code all lives inside of here[0].

[0]: https://github.com/pulp/pulp/blob/3.0-dev/platform/pulp/tasking/services/worker_watcher.py


Checklist

Associated revisions

Revision e6e6e06f View on GitHub
Added by fdobrovo over 2 years ago

Keep workers records when they gone offline

closes #2430 https://pulp.plan.io/issues/2430

Revision e6e6e06f View on GitHub
Added by fdobrovo over 2 years ago

Keep workers records when they gone offline

closes #2430 https://pulp.plan.io/issues/2430

History

#1 Updated by mhrivnak over 3 years ago

This may require us to re-think the uniqueness constraint. I suspect the tasking system will want to continue having an easy way to guarantee that there's only one active worker at a time with a given name.

Perhaps we could add a field that's a timestamp for when the worker was "archived", let that field be null, and make that field plus the "name" be unique together? That would allow just one non-archived worker, and any number of archived ones.

#2 Updated by bmbouter over 3 years ago

  • Checklist item ensure that the /status/ API only shows workers that are online added

I was imagining we would keep the existing uniqueness constraint[0], and when pulp_celerybeat brings a worker online and there is an existing record in the db, it would update(online=True, last_heartbeat=<now>)

I do think we want to "reuse" worker records because we want to aggregate all Task foreign keys onto the same Worker instance by name. Otherwise when we use worker.tasks we will only be getting the tasks for the most recent instance which is generally not what we want when we write that code. To then get the full list you would have to aggregate all Tasks that have a foreign key to a worker whose name=<someworkername>. Given ^ just reusing the same worker instance would allow us to sidestep this aggregation issue completely.

[0]: https://github.com/pulp/pulp/blob/e29b9e796e5f02699558fc7be85d5f94f85000e4/app/pulp/app/models/task.py#L47

#3 Updated by mhrivnak over 3 years ago

Very nice idea. I like it.

#4 Updated by mhrivnak almost 3 years ago

  • Description updated (diff)
  • Groomed changed from No to Yes

#5 Updated by bmbouter almost 3 years ago

  • Description updated (diff)

#6 Updated by fdobrovo almost 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to fdobrovo

#7 Updated by fdobrovo almost 3 years ago

  • Status changed from ASSIGNED to POST

#8 Updated by fdobrovo over 2 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#9 Updated by rchan about 2 years ago

  • Sprint Candidate deleted (Yes)

#10 Updated by daviddavis 10 months ago

  • Sprint/Milestone set to 3.0.0

#11 Updated by bmbouter 10 months ago

  • Tags deleted (Pulp 3)

#12 Updated by bmbouter 3 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF