Project

Profile

Help

Issue #1012

Task #1014: Short Term Improvements for Pulp's use of MongoDB

If the first database seed listed in server.conf is not the current Primary replica, Pulp will not be able to write to the database

Added by rbarlow over 5 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.7.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Pulp only tells pymongo about the first of the database seeds, even if there are multiple of the specified in the seeds field of the [database] section in server.conf. This means that if the first seed listed is not the current primary, Pulp will not be able to issue any writes.

The fix for this is to try the first seed, and if it is not the primary or cannot connect for some network reason to iterate through the remaining seeds until a valid connection is formed. It should log at info level for each of these connections so Pulp reports as it progresses. If a valid connection cannot be formed it should log at error level as it does today.

QE instructions

1. create a 3 node mongoDB setup that uses a basic replica_set configuration
2. determine which mongoDB hostname is the secondary
3. configure server.conf [database] section with seeds="" to the comma separated list of these 3 mongoDB hosts. It's very important that the "primary" mongoDB server be the last one in the seeds list
4. start Pulp and verify that it logs that a connection is created to the first seed, the second seed, and finally the primary.
5. ensure pulp can be fully used while connected to the third seed (the primary)
6. stop all pulp services.
7. edit server.conf and remove the primary from the seeds list so that only the two secondary servers are present
8) start Pulp and verify that it cannot form a valid connection to a mongoDB server


Checklist


Related issues

Blocks Pulp - Issue #1139: Fix Pulp's use of replica sets replica setsCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 75397ecb View on GitHub
Added by dkliban@redhat.com about 5 years ago

Properly handles ['database']['seeds'] config

This patch enables pulp to iterate over a list of seeds until a connection is made. When more than one seed is present, an exception is raised if no 'replica_set' config is specified. An exception is also raised if an empty string is passed in as value for 'seeds'.

re #1012 https://pulp.plan.io/issues/1012

Revision 75397ecb View on GitHub
Added by dkliban@redhat.com about 5 years ago

Properly handles ['database']['seeds'] config

This patch enables pulp to iterate over a list of seeds until a connection is made. When more than one seed is present, an exception is raised if no 'replica_set' config is specified. An exception is also raised if an empty string is passed in as value for 'seeds'.

re #1012 https://pulp.plan.io/issues/1012

History

#1 Updated by rbarlow over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to rbarlow

#2 Updated by dkliban@redhat.com over 5 years ago

  • Parent task set to #1014

#3 Updated by dkliban@redhat.com over 5 years ago

  • Blocks Issue #974: GET of a task sometimes 404s on first request added

#4 Updated by rbarlow over 5 years ago

  • Blocks Story #1: As a user, I can have Pulp attempt use auto_retry application wide using the 'unsafe_autoretry' parameter added

#5 Updated by rbarlow over 5 years ago

  • Platform Release set to 2.6.3

#6 Updated by jortel@redhat.com over 5 years ago

  • Triaged changed from No to Yes

#7 Updated by rbarlow over 5 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (rbarlow)

#8 Updated by bmbouter about 5 years ago

  • Blocks deleted (Issue #974: GET of a task sometimes 404s on first request)

#9 Updated by bmbouter about 5 years ago

  • Blocks deleted (Story #1: As a user, I can have Pulp attempt use auto_retry application wide using the 'unsafe_autoretry' parameter)

#10 Updated by bmbouter about 5 years ago

  • Checklist item changed from to [ ] Ensure that Pulp can tell if it is connected to a mongo node that is not the primary, [ ] Pulp iterates through all seeds until it connects to the primary, [ ] Pulp logs at info level for each connection it attempts, [ ] Pulp logs at info level for each connection attempt that failed, [ ] Preserve the existing behavior where Pulp logs at error level if all a valid connection could not be formed
  • Description updated (diff)
  • Platform Release changed from 2.6.3 to 2.6.4

#11 Updated by bmbouter about 5 years ago

  • Subject changed from If the first database seed listed in server.conf is not the current Primary replica, Pulp will not be able to read or write to the database to If the first database seed listed in server.conf is not the current Primary replica, Pulp will not be able to write to the database

#12 Updated by bmbouter about 5 years ago

  • Blocks Issue #1139: Fix Pulp's use of replica sets replica sets added

#13 Updated by bmbouter about 5 years ago

  • Platform Release deleted (2.6.4)

#14 Updated by dkliban@redhat.com about 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com
  • Platform Release set to 2.7.0

#15 Updated by dkliban@redhat.com about 5 years ago

  • Status changed from ASSIGNED to MODIFIED

#16 Updated by dkliban@redhat.com almost 5 years ago

  • Status changed from MODIFIED to 5

#17 Updated by pthomas@redhat.com almost 5 years ago

  • Status changed from 5 to 6

verified

[root@sparks ~]# rpm -qa pulp-server
pulp-server-2.7.0-0.8.rc.el7.noarch
[root@sparks ~]# 

#18 Updated by amacdona@redhat.com almost 5 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#20 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF