Story #1
Updated by rbarlow over 9 years ago
Along with the conversion to mongoengine, we should ensure that it works correctly with an HA setup. The mongoengine connection should be tested with an HA mongodb setup.
1. The tester should remove the auto-retry behavior of Pulp from PulpCollection which should disable auto-retry everywhere.
2. Test with a single mongoDB broker and observe Pulp's logging behavior if MongoDB is killed or stopped.
3. Setup a three-node replica set of MongoDB servers. Have Pulp connected to them, and then kill the one that Pulp is connected to. What is Pulp's behavior for failover? At what point are the Connection erorrs raised to the codebase, i.e., how many nodes have to fail before Pulp stops? Test behavior when more than half the nodes are missing. When this happens MongoDB goes into read only-mode. Test Pulp's behavior in this scenario.
4. Does the multi-host failover work with any number of MongoDB hosts, or just three?
5. Consider using Docker for testing.
6. Test the celery connection also. Celery has its own database connection and it needs to be tested that it also works with an HA setup.
Based on this information PulpCollection [0] and mongoengine models (which do not use PulpCollection) need to have the correct retry behavior in place so that high availability failover happens. After failover can no longer continue the connection error should be raised so that the component using the database can decide how to handle it.
[0]: https://github.com/pulp/pulp/blob/master/server/pulp/server/db/connection.py#L170
.