Project

Profile

Help

Task #2430

Keep Worker model records around instead of deleting them

Added by bmbouter almost 3 years ago. Updated 6 months ago.

Status:
MODIFIED
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:

Description

Currently Pulp worker DB records are deleted when workers go missing or are shut down, and created when they are first observed or updated. It would be beneficial to keep these records. This allows someone to read these records post-mortem. It has a tertiary benefit that Tasks won't be cascade deleted due to the Worker objects being removed.

To resolve this, introduce a boolean named 'online' on the Worker model. The code in pulp_celerybeat that manages these Worker records would be updated to set the 'online' value when workers start and stop. The field's default should correspond with its usage.

Currently on 3.0-dev the pulp_celerybeat code all lives inside of here0.

[0]: https://github.com/pulp/pulp/blob/3.0-dev/platform/pulp/tasking/services/worker_watcher.py


Checklist

Associated revisions

Revision e6e6e06f View on GitHub
Added by fdobrovo over 2 years ago

Keep workers records when they gone offline

closes #2430
https://pulp.plan.io/issues/2430

Revision e6e6e06f View on GitHub
Added by fdobrovo over 2 years ago

Keep workers records when they gone offline

closes #2430
https://pulp.plan.io/issues/2430

Revision e6e6e06f View on GitHub
Added by fdobrovo over 2 years ago

Keep workers records when they gone offline

closes #2430
https://pulp.plan.io/issues/2430

History

#1 Updated by mhrivnak almost 3 years ago

This may require us to re-think the uniqueness constraint. I suspect the tasking system will want to continue having an easy way to guarantee that there's only one active worker at a time with a given name.

Perhaps we could add a field that's a timestamp for when the worker was "archived", let that field be null, and make that field plus the "name" be unique together? That would allow just one non-archived worker, and any number of archived ones.

#2 Updated by bmbouter almost 3 years ago

  • Checklist item ensure that the /status/ API only shows workers that are online added

I was imagining we would keep the existing uniqueness constraint0, and when pulp_celerybeat brings a worker online and there is an existing record in the db, it would update(online=True, last_heartbeat=<now>)

I do think we want to "reuse" worker records because we want to aggregate all Task foreign keys onto the same Worker instance by name. Otherwise when we use worker.tasks we will only be getting the tasks for the most recent instance which is generally not what we want when we write that code. To then get the full list you would have to aggregate all Tasks that have a foreign key to a worker whose name=<someworkername>. Given ^ just reusing the same worker instance would allow us to sidestep this aggregation issue completely.

[0]: https://github.com/pulp/pulp/blob/e29b9e796e5f02699558fc7be85d5f94f85000e4/app/pulp/app/models/task.py#L47

#3 Updated by mhrivnak almost 3 years ago

Very nice idea. I like it.

#4 Updated by mhrivnak over 2 years ago

  • Description updated (diff)
  • Groomed changed from No to Yes

#5 Updated by bmbouter over 2 years ago

  • Description updated (diff)

#6 Updated by fdobrovo over 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to fdobrovo

#7 Updated by fdobrovo over 2 years ago

  • Status changed from ASSIGNED to POST

#8 Updated by fdobrovo over 2 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#9 Updated by rchan over 1 year ago

  • Sprint Candidate deleted (Yes)

#10 Updated by daviddavis 6 months ago

  • Sprint/Milestone set to 3.0

#11 Updated by bmbouter 6 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF