Story #1
Updated by bmbouter over 9 years ago
Along with the conversion to mongoengine, we should ensure that it works correctly with an HA setup. The mongoengine connection should be tested with an HA mongodb setup.
1. The tester should remove the auto-retry behavior of Pulp from PulpCollection which should disable auto-retry everywhere.
2. Test with a single mongoDB broker and observe Pulp's logging behavior if MongoDB is killed or stopped.
3. Setup a two-host HA pair of MongoDB servers. Have Pulp connected to one or both, and then kill the one that Pulp is connected to. What is Pulp's behavior for failover? At what point are the Connection erorrs raised to the codebase?
4. Does the multi-host failover work with any number of MongoDB hosts, or just two?
5. Test the celery connection also. Celery has its own database connection and it needs to be tested that it also works with an HA setup.
Based on this information PulpCollection [0] and mongoengine models (which do not use PulpCollection) need to have the correct retry behavior in place so that high availability failover happens. After failover can no longer continue the connection error should be raised so that the component using the database can decide how to handle it.
[0]: https://github.com/pulp/pulp/blob/master/server/pulp/server/db/connection.py#L170
.