Story #1
Updated by bmbouter over 9 years ago
Pulp used Along with the conversion to use have mongoengine, we should ensure that it works correctly with an "auto_retry decorator":https://github.com/pulp/pulp/blob/d17e9e5c3fe2a3a2b81ef252304984c40003b7b7/server/pulp/server/db/connection.py#L163-L189 on HA setup. The mongoengine connection should be tested with an HA mongodb setup. 1. The tester should remove the auto-retry behavior of Pulp from PulpCollection which should disable auto-retry everywhere. 2. Test with a single mongoDB queries that was used everywhere broker and could result in duplicate records. Collections observe Pulp's logging behavior if MongoDB is killed or stopped. 3. Setup a three-node replica set of MongoDB servers. Have Pulp connected to them, and then kill the one that Pulp is connected to. What is Pulp's behavior for failover? At what point are the Connection erorrs raised to the codebase, i.e., how many nodes have been converted to mongoengine in fail before Pulp 2.7.0+ do not have stops? Test behavior when more than half the auto_retry decorator nodes are missing. When this happens MongoDB goes into read only-mode. Test Pulp's behavior at all currently. The auto_retry behavior should be able to be enabled/disabled by the user. This story introduces a boolean 'unsafe_autoretry' setting in this scenario. 4. Does the the [database] section multi-host failover work with any number of server.conf which enabled/disables MongoDB hosts, or just three? 5. Consider using Docker for testing. 6. Test the use of auto_retry on both mongoengine Documents celery connection also. Celery has its own database connection and non-mongoengine collections. The parameter will default it needs to False. If set to True, all mongoDB queries will use be tested that it also works with an auto_retry decorator. Both HA setup. Based on this information PulpCollection [0] and mongoengine Documents and non-mongoengine Collections models (which do not use PulpCollection) need to use have the same correct retry decorator behavior in place so that its behavior high availability failover happens. After failover can no longer continue the connection error should be kept consistent. To effectively decorate mongoengine it needs to be done during the __init__ of each mongoengine model. This would cause a lot of duplication raised so it's recommended to have all mongoengine objects that inherit directly from Document inherit from a new class you will create called AutoRetryDocument. AutoRetryDocument should contain the __init__ that does component using the decorating and takes *args and **args similar to: database can decide how to handle it. <pre> This link [1] may be helpful in quick setting up mongodb HA stuff for testing. [0]: https://github.com/pulp/pulp/blob/master/server/pulp/server/db/connection.py#L170 class AutoRetryDocument(Document): _decorated_methods = (<figure out which methods to decorate>) def __init__(self, *args, **kwargs): super(AutoRetryDocument, self).__init__(*args, **kwargs) # do the decorating here similar to https://github.com/pulp/pulp/blob/d17e9e5c3fe2a3a2b81ef252304984c40003b7b7/server/pulp/server/db/connection.py#L200-L205 </pre> [1]: http://api.mongodb.org/python/current/examples/high_availability.html
.