Project

Profile

Help

Story #1

Task #1014: Short Term Improvements for Pulp's use of MongoDB

As a user, I can have Pulp attempt use auto_retry application wide using the 'unsafe_autoretry' parameter

Added by Anonymous over 5 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Platform Release:
2.7.0
Groomed:
No
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:

Description

Pulp used to use have an auto_retry decorator on mongoDB queries that was used everywhere and could result in duplicate records. Collections that have been converted to mongoengine in Pulp 2.7.0+ do not have the auto_retry decorator behavior at all currently.

The auto_retry behavior should be able to be enabled/disabled by the user. This story introduces a boolean 'unsafe_autoretry' setting in the the [database] section of server.conf which enabled/disables the use of auto_retry on both mongoengine Documents and non-mongoengine collections. The parameter will default to False. If set to True, all mongoDB queries will use an auto_retry decorator. Both mongoengine Documents and non-mongoengine Collections need to use the same retry decorator so that its behavior can be kept consistent.

To effectively decorate mongoengine it needs to be done during the init of each mongoengine model. This would cause a lot of duplication so it's recommended to have all mongoengine objects that inherit directly from Document inherit from a new class you will create called AutoRetryDocument. AutoRetryDocument should contain the init that does the decorating and takes *args and **args similar to:

class AutoRetryDocument(Document):

    _decorated_methods = (<figure out which methods to decorate>)

    def __init__(self, *args, **kwargs):
        super(AutoRetryDocument, self).__init__(*args, **kwargs)
        # do the decorating here similar to https://github.com/pulp/pulp/blob/d17e9e5c3fe2a3a2b81ef252304984c40003b7b7/server/pulp/server/db/connection.py#L210

Checklist

Associated revisions

Revision 805211fb View on GitHub
Added by Austin Macdonald almost 5 years ago

Introduce unsafe_autoretry setting

re #1

Revision 805211fb View on GitHub
Added by Austin Macdonald almost 5 years ago

Introduce unsafe_autoretry setting

re #1

History

#1 Updated by rbarlow over 5 years ago

  • Tracker changed from Story to Task

#3 Updated by bmbouter over 5 years ago

  • Subject changed from [Task] Test mongoengine connection with no auto-retry and an HA mongoengine configuration to Test mongoengine connection with no auto-retry and an HA mongoengine configuration

#4 Updated by bmbouter over 5 years ago

  • Description updated (diff)

#5 Updated by bmbouter over 5 years ago

  • Description updated (diff)

#6 Updated by bcourt over 5 years ago

  • Priority changed from Normal to High

#7 Updated by bmbouter over 5 years ago

  • Tags Sprint Candidate added

#8 Updated by mhrivnak over 5 years ago

  • Priority changed from High to Normal

#9 Updated by bmbouter over 5 years ago

  • Description updated (diff)

#10 Updated by rbarlow over 5 years ago

  • Description updated (diff)

#11 Updated by rbarlow over 5 years ago

  • Description updated (diff)

#12 Updated by bmbouter over 5 years ago

  • Tags Groomed added

#13 Updated by bmbouter over 5 years ago

  • Description updated (diff)

#14 Updated by bmbouter over 5 years ago

  • Groomed set to Yes
  • Tags deleted (Groomed)

#15 Updated by bmbouter over 5 years ago

  • Sprint Candidate set to Yes
  • Tags deleted (Sprint Candidate)

#16 Updated by rbarlow about 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to rbarlow

#17 Updated by rbarlow about 5 years ago

  • Tags Easy Fix added

Brian told me this would be super easy…

#18 Updated by dkliban@redhat.com about 5 years ago

  • Parent task set to #1014

#19 Updated by rbarlow about 5 years ago

  • Blocked by Issue #1012: If the first database seed listed in server.conf is not the current Primary replica, Pulp will not be able to write to the database added

#20 Updated by rbarlow about 5 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (rbarlow)

#21 Updated by bmbouter about 5 years ago

  • Checklist item changed from to [ ] non-mongoengine collections do/don't decorate pymongo methods based on this setting, [ ] mongoengine methods are/aren't decorated based on this setting, [ ] release note on the new setting introduced, [ ] commented setting introduced in server.conf (false is default), [ ] setting key and value defaults work if setting in server.conf is commented, [ ] setting description added to server.conf, [ ] create AutoRetryDocument, [ ] have all Pulp mongoengine documents inherit from AutoRetryDocument, [ ] update the mongoengine conversion guide [0] to identify this inheritance as a requirement [0]: https://fedorahosted.org/pulp/wiki/ConvertingModelsToMongoengine
  • Tracker changed from Task to Story
  • Subject changed from Test mongoengine connection with no auto-retry and an HA mongoengine configuration to As a user, I can have Pulp attempt use auto_retry application wide using the 'unsafe_autoretry' parameter
  • Description updated (diff)
  • Groomed changed from Yes to No
  • Tags deleted (Easy Fix)

#22 Updated by bmbouter about 5 years ago

  • Checklist item changed from [ ] non-mongoengine collections do/don't decorate pymongo methods based on this setting, [ ] mongoengine methods are/aren't decorated based on this setting, [ ] release note on the new setting introduced, [ ] commented setting introduced in server.conf (false is default), [ ] setting key and value defaults work if setting in server.conf is commented, [ ] setting description added to server.conf, [ ] create AutoRetryDocument, [ ] have all Pulp mongoengine documents inherit from AutoRetryDocument, [ ] update the mongoengine conversion guide [0] to identify this inheritance as a requirement [0]: https://fedorahosted.org/pulp/wiki/ConvertingModelsToMongoengine to [ ] non-mongoengine collections do/don't decorate pymongo methods based on this setting, [ ] mongoengine methods are/aren't decorated based on this setting, [ ] release note on the new setting introduced, [ ] commented setting introduced in server.conf (false is default), [ ] setting key and value defaults work if setting in server.conf is commented, [ ] setting description added to server.conf, [ ] create AutoRetryDocument, [ ] have all Pulp mongoengine documents inherit from AutoRetryDocument, [ ] update the mongoengine conversion guide [0] to identify this inheritance as a requirement [0]: https://fedorahosted.org/pulp/wiki/ConvertingModelsToMongoengine, [ ] both mongoengine and non-mongoengine use the same retry_decorator (not two retry decorators)
  • Description updated (diff)

#23 Updated by bmbouter about 5 years ago

  • Blocked by deleted (Issue #1012: If the first database seed listed in server.conf is not the current Primary replica, Pulp will not be able to write to the database)

#24 Updated by bmbouter about 5 years ago

  • Platform Release set to 2.7.0

#25 Updated by amacdona@redhat.com almost 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to amacdona@redhat.com

#26 Updated by amacdona@redhat.com almost 5 years ago

  • Checklist item changed from [ ] setting description added to server.conf to [x] setting description added to server.conf

#27 Updated by amacdona@redhat.com almost 5 years ago

  • Checklist item changed from [x] setting description added to server.conf to [ ] setting description added to server.conf

#28 Updated by amacdona@redhat.com almost 5 years ago

  • Checklist item changed from [ ] non-mongoengine collections do/don't decorate pymongo methods based on this setting, [ ] mongoengine methods are/aren't decorated based on this setting, [ ] release note on the new setting introduced, [ ] commented setting introduced in server.conf (false is default), [ ] setting key and value defaults work if setting in server.conf is commented, [ ] setting description added to server.conf, [ ] create AutoRetryDocument, [ ] have all Pulp mongoengine documents inherit from AutoRetryDocument, [ ] update the mongoengine conversion guide [0] to identify this inheritance as a requirement [0]: https://fedorahosted.org/pulp/wiki/ConvertingModelsToMongoengine, [ ] both mongoengine and non-mongoengine use the same retry_decorator (not two retry decorators) to [x] non-mongoengine collections do/don't decorate pymongo methods based on this setting, [x] mongoengine methods are/aren't decorated based on this setting, [x] release note on the new setting introduced, [x] commented setting introduced in server.conf (false is default), [x] setting key and value defaults work if setting in server.conf is commented, [x] setting description added to server.conf, [x] create AutoRetryDocument, [x] have all Pulp mongoengine documents inherit from AutoRetryDocument, [ ] update the mongoengine conversion guide [0] to identify this inheritance as a requirement [0]: https://fedorahosted.org/pulp/wiki/ConvertingModelsToMongoengine, [x] both mongoengine and non-mongoengine use the same retry_decorator (not two retry decorators), [ ] modify plugins that have been converted to mongoengine to inherit from AutoRetryDocument
  • Status changed from ASSIGNED to POST

#29 Updated by amacdona@redhat.com almost 5 years ago

  • Status changed from POST to MODIFIED

#30 Updated by amacdona@redhat.com almost 5 years ago

  • Checklist item changed from [ ] update the mongoengine conversion guide [0] to identify this inheritance as a requirement [0]: https://fedorahosted.org/pulp/wiki/ConvertingModelsToMongoengine to [x] update the mongoengine conversion guide [0] to identify this inheritance as a requirement [0]: https://fedorahosted.org/pulp/wiki/ConvertingModelsToMongoengine

#31 Updated by amacdona@redhat.com almost 5 years ago

  • Checklist item changed from [ ] modify plugins that have been converted to mongoengine to inherit from AutoRetryDocument to [x] modify plugins that have been converted to mongoengine to inherit from AutoRetryDocument

#32 Updated by dkliban@redhat.com almost 5 years ago

  • Status changed from MODIFIED to 5

#33 Updated by rbarlow over 4 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#34 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF