Project

Profile

Help

Issue #2423

As a user, I want pulp_celerybeat to survive a brief database outage without having to use the unsafe_autoretry feature.

Added by daviddavis almost 4 years ago. Updated over 1 year ago.

Status:
CLOSED - WORKSFORME
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.8.7
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Related issues

Related to Pulp - Issue #2331: The "unsafe_autoretry" feature does not retry with mongoengine.QuerySet codepathsCLOSED - WONTFIX<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

History

#1 Updated by daviddavis almost 4 years ago

  • Related to Issue #2331: The "unsafe_autoretry" feature does not retry with mongoengine.QuerySet codepaths added

#3 Updated by bmbouter almost 4 years ago

Here is a recap of some testing I just did against master with unsafe_autoretry=False.

All processes survive a mongodb downtime of 30 seconds already. This is the pymongo default and pymongo doesn't raise the exception until 30 seconds of time has passed. So normal restarts and brief (<30 second) database outages are already handled correctly. After 30 seconds, pulp_celerybeat is the only process that exits and does not recover when mongodb recovers.

Note that if mongodb is down when pulp processes are starting, pulp_celerybeat, pulp_workers, and pulp_resource_manager all will fail to start and will not recover when mongodb recovers. This is an initial connection availability problem and affects almost all of Pulp (except httpd).

Note we identify in the docs[0] that if users want to catch mongodb errors including connection errors, they should turn on unsafe_autoretry.

[0]: http://docs.pulpproject.org/user-guide/scaling.html#mongodb-automatic-retry

#4 Updated by bmbouter almost 4 years ago

I think since it survives a brief outage already, what about closing this as WONTFIX?

The long-term fix for these problems is already being tracked as Issue #2417

#5 Updated by daviddavis almost 4 years ago

bmbouter, thanks for doing research. I agree with your assessment although I am not sure how #2147 is related. Did you mean #2331?

#6 Updated by bmbouter almost 4 years ago

daviddavis, oh right that issue number was a typo. I updated it to point to the correct issue which is #2417

#7 Updated by daviddavis almost 4 years ago

  • Status changed from NEW to CLOSED - WONTFIX

#8 Updated by daviddavis almost 4 years ago

  • Status changed from CLOSED - WONTFIX to CLOSED - WORKSFORME

#9 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF