Pulp

Here is a recap of some testing I just did against master with unsafe_autoretry=False.

All processes survive a mongodb downtime of 30 seconds already. This is the pymongo default and pymongo doesn't raise the exception until 30 seconds of time has passed. So normal restarts and brief (<30 second) database outages are already handled correctly. After 30 seconds, pulp_celerybeat is the only process that exits and does not recover when mongodb recovers.

Note that if mongodb is down when pulp processes are starting, pulp_celerybeat, pulp_workers, and pulp_resource_manager all will fail to start and will not recover when mongodb recovers. This is an initial connection availability problem and affects almost all of Pulp (except httpd).

Note we identify in the docs[0] that if users want to catch mongodb errors including connection errors, they should turn on unsafe_autoretry.

[0]: http://docs.pulpproject.org/user-guide/scaling.html#mongodb-automatic-retry

I think since it survives a brief outage already, what about closing this as WONTFIX?

The long-term fix for these problems is already being tracked as Issue #2417

bmbouter, thanks for doing research. I agree with your assessment although I am not sure how #2147 is related. Did you mean #2331?

@daviddavis, oh right that issue number was a typo. I updated it to point to the correct issue which is #2417