Project

Profile

Help

Issue #1139

Task #1014: Short Term Improvements for Pulp's use of MongoDB

Fix Pulp's use of replica sets replica sets

Added by bmbouter over 6 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
2.7.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Currently Pulp does not use the MongoDB replica set functionality correctly when the user correctly configures the replica_set setting in the [database] section of server.conf. By default replica_set is not set. If replica_set is set, then Pulp should have pyMongo use the MongoReplicaSetClient type. This will allow Pulp to rely on PyMongo to automatically negotiate who the primary is and failover in the event that the Primary changes. Currently we attempt to do this, but there is a bug caused by parameter name mismatch which causes PyMongo to ignore our setting of this option. That is the first thing to fix. Once fixed it is expected that PyMongo will know about and use the replica_set as it interacts with MongoDB.

One thing to know is that when a failover of the primary happens an AutoReconnect exception is raised. To effectively test this story you'll have to have pulp "auto retry" the operation so that the next operation will go to the newly connected to Primary. On 2.6.x this should be the default behavior, on 2.7.0 you'll need to wait for story #1 to be implemented and set unsafe_autoretry = True in server.conf

In the case where multiple seeds are specified as fixed in issue (#1012), all seeds should be iterated through until a valid MongoReplicaSetClient connection can be found. Either Pulp needs to find the primary or PyMongo needs to reconnect to the primary. Specifically test that in a multi-node mongoDB configuration with one primary, one arbiter, and one secondary with all participating in a replica_set that you can specify only the secondary as the value for seed and that the MongoReplicaSetClient will automatically discover the primary and connect to it even though its not contained in the seed listing.

If Pulp does not have replica_set configured (the default) then PyMongo needs to use the single-mongoDB style client named MongoClient.

If this is story is introduced in a version which does not use mongoengine and then merged forward you'll have to ensure that the newer version with mongoengine continues to work as expected.

This story also introduces a constant named WRITE_CONCERN which should be hard coded to 'majority'. This will be the write concern that the entire Pulp codebase will use as its sets write concern throughout bugfix #1065. You'll need to verify that this will work correctly even if replica_set is not in use and PyMongo is using the MongoClient connection style. You will also need to verify that "majority" is a valid write concern to be specified on mongoDB server 2.4.0+. Verifying that is part of this story.

If Pulp does not have replica_set configured (the default) AND there are more than one seed specified in the [database] section of server.conf then an error should be logged and Pulp should refuse to start. This is because having multiple seeds without using replica set functionality of MongoDB is likely not a correct configuration and should be fixed up by the user. They will either need to have a single seed listed or configure a replica_set.


Checklist


Related issues

Blocked by Pulp - Issue #1012: If the first database seed listed in server.conf is not the current Primary replica, Pulp will not be able to write to the databaseCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Blocks Pulp - Issue #1065: DeprecationWarning on pulp-manage-dbCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 1ad36448 View on GitHub
Added by dkliban@redhat.com about 6 years ago

Adds support for write concern

Pulp now accepts a write concern for the MongoDB connection. All writes to the database are now acknowledged. For replica sets, 'majority' or 'all' members can be asked to acknowledge the write.

closes: #974 closes: #1139

Revision 1ad36448 View on GitHub
Added by dkliban@redhat.com about 6 years ago

Adds support for write concern

Pulp now accepts a write concern for the MongoDB connection. All writes to the database are now acknowledged. For replica sets, 'majority' or 'all' members can be asked to acknowledge the write.

closes: #974 closes: #1139

History

#1 Updated by bmbouter over 6 years ago

  • Parent task set to #1014

#2 Updated by bmbouter over 6 years ago

  • Blocked by Issue #1012: If the first database seed listed in server.conf is not the current Primary replica, Pulp will not be able to write to the database added

#3 Updated by bmbouter over 6 years ago

  • Blocks Issue #1065: DeprecationWarning on pulp-manage-db added

#4 Updated by bmbouter over 6 years ago

  • Platform Release deleted (2.7.0)

#5 Updated by bmbouter over 6 years ago

  • Checklist item changed from [ ] If replica_set is not configured, the MongoClient connection is used., [ ] If replica_set is not configured AND there are multiple seeds listed an error should be logged and Pulp should refuse to start, [ ] WRITE_CONCERN = 'majority' is created as a constant in db/connection.py (module where initialization happens), [ ] Verify that WRITE_CONCERN = 'majority' is compatible with mongoDB server 2.4.0 and newer, [ ] Verify that WRITE_CONCERN = 'majority' will work with both MongoClient and MongoReplicaSetClient, [ ] If replica_set is configured, the MongoReplicaSetClient connection is used, properly configured, and aware of the replica_set, [ ] Test that Pulp will discover the primary if replica_set is configured and the only seed it is given is a a secondary, [ ] Ensure that the functionality of bugfix #1012 is still in place whereby all seeds are checked until one is found to be working, [ ] Write good instructions for QE to verify this issue by to [ ] If replica_set is not configured, the MongoClient connection is used., [ ] If replica_set is not configured AND there are multiple seeds listed an error should be logged and Pulp should refuse to start, [ ] WRITE_CONCERN = 'majority' is created as a constant in db/connection.py (module where initialization happens), [ ] Verify that WRITE_CONCERN = 'majority' is compatible with mongoDB server 2.4.0 and newer, [ ] Verify that WRITE_CONCERN = 'majority' will work with both MongoClient and MongoReplicaSetClient, [ ] If replica_set is configured, the MongoReplicaSetClient connection is used, properly configured, and aware of the replica_set, [ ] Test that Pulp will discover the primary if replica_set is configured and the only seed it is given is a a secondary, [ ] Ensure that the functionality of bugfix #1012 is still in place whereby all seeds are checked until one is found to be working, [ ] Write good instructions for QE to verify this issue by, [ ] Test that failover works with everything properly configured and MongoReplicaSetClient is in use. You'll need to ensure to have auto_retry configured so that the write operation occurs when the AutoReconnect is raised
  • Description updated (diff)

#6 Updated by jortel@redhat.com over 6 years ago

  • Triaged changed from No to Yes

#7 Updated by dkliban@redhat.com about 6 years ago

  • Status changed from NEW to MODIFIED
  • % Done changed from 0 to 100

#8 Updated by dkliban@redhat.com almost 6 years ago

  • Platform Release set to 2.7.0

#9 Updated by amacdona@redhat.com almost 6 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

#10 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF