Project

Profile

Help

Story #1

Task #1014: Short Term Improvements for Pulp's use of MongoDB

As a user, I can have Pulp attempt use auto_retry application wide using the 'unsafe_autoretry' parameter

Added by Anonymous almost 5 years ago. Updated 7 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

0%

Platform Release:
2.7.0
Blocks Release:
Backwards Incompatible:
No
Groomed:
No
Sprint Candidate:
Yes
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:

Description

Pulp used to use have an auto_retry decorator on mongoDB queries that was used everywhere and could result in duplicate records. Collections that have been converted to mongoengine in Pulp 2.7.0+ do not have the auto_retry decorator behavior at all currently.

The auto_retry behavior should be able to be enabled/disabled by the user. This story introduces a boolean 'unsafe_autoretry' setting in the the [database] section of server.conf which enabled/disables the use of auto_retry on both mongoengine Documents and non-mongoengine collections. The parameter will default to False. If set to True, all mongoDB queries will use an auto_retry decorator. Both mongoengine Documents and non-mongoengine Collections need to use the same retry decorator so that its behavior can be kept consistent.

To effectively decorate mongoengine it needs to be done during the init of each mongoengine model. This would cause a lot of duplication so it's recommended to have all mongoengine objects that inherit directly from Document inherit from a new class you will create called AutoRetryDocument. AutoRetryDocument should contain the init that does the decorating and takes *args and **args similar to:

class AutoRetryDocument(Document):

    _decorated_methods = (<figure out which methods to decorate>)

    def __init__(self, *args, **kwargs):
        super(AutoRetryDocument, self).__init__(*args, **kwargs)
        # do the decorating here similar to https://github.com/pulp/pulp/blob/d17e9e5c3fe2a3a2b81ef252304984c40003b7b7/server/pulp/server/db/connection.py#L210

Checklist

Associated revisions

Revision 805211fb View on GitHub
Added by Austin Macdonald about 4 years ago

Introduce unsafe_autoretry setting

re #1

Revision 805211fb View on GitHub
Added by Austin Macdonald about 4 years ago

Introduce unsafe_autoretry setting

re #1

Revision 805211fb View on GitHub
Added by Austin Macdonald about 4 years ago

Introduce unsafe_autoretry setting

re #1

History

#1 Updated by rbarlow almost 5 years ago

  • Tracker changed from Story to Task

#2 Updated by rbarlow almost 5 years ago

  • Project changed from Import to Pulp

#3 Updated by bmbouter almost 5 years ago

  • Subject changed from [Task] Test mongoengine connection with no auto-retry and an HA mongoengine configuration to Test mongoengine connection with no auto-retry and an HA mongoengine configuration

#4 Updated by bmbouter almost 5 years ago

  • Description updated (diff)

#5 Updated by bmbouter almost 5 years ago

  • Description updated (diff)

#6 Updated by bcourt almost 5 years ago

  • Priority changed from Normal to High

#7 Updated by bmbouter almost 5 years ago

  • Backwards Incompatible set to No
  • Tags Sprint Candidate added

#8 Updated by mhrivnak over 4 years ago

  • Priority changed from High to Normal

#9 Updated by bmbouter over 4 years ago

  • Description updated (diff)

#10 Updated by rbarlow over 4 years ago

  • Description updated (diff)

#11 Updated by rbarlow over 4 years ago

  • Description updated (diff)

#12 Updated by bmbouter over 4 years ago

  • Tags Groomed added

#13 Updated by bmbouter over 4 years ago

  • Description updated (diff)

#14 Updated by bmbouter over 4 years ago

  • Groomed set to Yes
  • Tags deleted (Groomed)

#15 Updated by bmbouter over 4 years ago

  • Sprint Candidate set to Yes
  • Tags deleted (Sprint Candidate)

#16 Updated by rbarlow over 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to rbarlow

#17 Updated by rbarlow over 4 years ago

  • Tags Easy Fix added

Brian told me this would be super easy…

#18 Updated by dkliban@redhat.com over 4 years ago

  • Parent task set to #1014

#19 Updated by rbarlow over 4 years ago

  • Blocked by Issue #1012: If the first database seed listed in server.conf is not the current Primary replica, Pulp will not be able to write to the database added

#20 Updated by rbarlow over 4 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (rbarlow)

#21 Updated by bmbouter over 4 years ago

  • Checklist item changed from to [ ] non-mongoengine collections do/don't decorate pymongo methods based on this setting, [ ] mongoengine methods are/aren't decorated based on this setting, [ ] release note on the new setting introduced, [ ] commented setting introduced in server.conf (false is default), [ ] setting key and value defaults work if setting in server.conf is commented, [ ] setting description added to server.conf, [ ] create AutoRetryDocument, [ ] have all Pulp mongoengine documents inherit from AutoRetryDocument, [ ] update the mongoengine conversion guide [0] to identify this inheritance as a requirement [0]: https://fedorahosted.org/pulp/wiki/ConvertingModelsToMongoengine
  • Tracker changed from Task to Story
  • Subject changed from Test mongoengine connection with no auto-retry and an HA mongoengine configuration to As a user, I can have Pulp attempt use auto_retry application wide using the 'unsafe_autoretry' parameter
  • Description updated (diff)
  • Groomed changed from Yes to No
  • Tags deleted (Easy Fix)

#22 Updated by bmbouter over 4 years ago

  • Checklist item changed from [ ] non-mongoengine collections do/don't decorate pymongo methods based on this setting, [ ] mongoengine methods are/aren't decorated based on this setting, [ ] release note on the new setting introduced, [ ] commented setting introduced in server.conf (false is default), [ ] setting key and value defaults work if setting in server.conf is commented, [ ] setting description added to server.conf, [ ] create AutoRetryDocument, [ ] have all Pulp mongoengine documents inherit from AutoRetryDocument, [ ] update the mongoengine conversion guide [0] to identify this inheritance as a requirement [0]: https://fedorahosted.org/pulp/wiki/ConvertingModelsToMongoengine to [ ] non-mongoengine collections do/don't decorate pymongo methods based on this setting, [ ] mongoengine methods are/aren't decorated based on this setting, [ ] release note on the new setting introduced, [ ] commented setting introduced in server.conf (false is default), [ ] setting key and value defaults work if setting in server.conf is commented, [ ] setting description added to server.conf, [ ] create AutoRetryDocument, [ ] have all Pulp mongoengine documents inherit from AutoRetryDocument, [ ] update the mongoengine conversion guide [0] to identify this inheritance as a requirement [0]: https://fedorahosted.org/pulp/wiki/ConvertingModelsToMongoengine, [ ] both mongoengine and non-mongoengine use the same retry_decorator (not two retry decorators)
  • Description updated (diff)

#23 Updated by bmbouter over 4 years ago

  • Blocked by deleted (Issue #1012: If the first database seed listed in server.conf is not the current Primary replica, Pulp will not be able to write to the database)

#24 Updated by bmbouter over 4 years ago

  • Platform Release set to 2.7.0

#25 Updated by amacdona@redhat.com about 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to amacdona@redhat.com

#26 Updated by amacdona@redhat.com about 4 years ago

  • Checklist item changed from [ ] setting description added to server.conf to [x] setting description added to server.conf

#27 Updated by amacdona@redhat.com about 4 years ago

  • Checklist item changed from [x] setting description added to server.conf to [ ] setting description added to server.conf

#28 Updated by amacdona@redhat.com about 4 years ago

  • Checklist item changed from [ ] non-mongoengine collections do/don't decorate pymongo methods based on this setting, [ ] mongoengine methods are/aren't decorated based on this setting, [ ] release note on the new setting introduced, [ ] commented setting introduced in server.conf (false is default), [ ] setting key and value defaults work if setting in server.conf is commented, [ ] setting description added to server.conf, [ ] create AutoRetryDocument, [ ] have all Pulp mongoengine documents inherit from AutoRetryDocument, [ ] update the mongoengine conversion guide [0] to identify this inheritance as a requirement [0]: https://fedorahosted.org/pulp/wiki/ConvertingModelsToMongoengine, [ ] both mongoengine and non-mongoengine use the same retry_decorator (not two retry decorators) to [x] non-mongoengine collections do/don't decorate pymongo methods based on this setting, [x] mongoengine methods are/aren't decorated based on this setting, [x] release note on the new setting introduced, [x] commented setting introduced in server.conf (false is default), [x] setting key and value defaults work if setting in server.conf is commented, [x] setting description added to server.conf, [x] create AutoRetryDocument, [x] have all Pulp mongoengine documents inherit from AutoRetryDocument, [ ] update the mongoengine conversion guide [0] to identify this inheritance as a requirement [0]: https://fedorahosted.org/pulp/wiki/ConvertingModelsToMongoengine, [x] both mongoengine and non-mongoengine use the same retry_decorator (not two retry decorators), [ ] modify plugins that have been converted to mongoengine to inherit from AutoRetryDocument
  • Status changed from ASSIGNED to POST

#29 Updated by amacdona@redhat.com about 4 years ago

  • Status changed from POST to MODIFIED

#30 Updated by amacdona@redhat.com about 4 years ago

  • Checklist item changed from [ ] update the mongoengine conversion guide [0] to identify this inheritance as a requirement [0]: https://fedorahosted.org/pulp/wiki/ConvertingModelsToMongoengine to [x] update the mongoengine conversion guide [0] to identify this inheritance as a requirement [0]: https://fedorahosted.org/pulp/wiki/ConvertingModelsToMongoengine

#31 Updated by amacdona@redhat.com about 4 years ago

  • Checklist item changed from [ ] modify plugins that have been converted to mongoengine to inherit from AutoRetryDocument to [x] modify plugins that have been converted to mongoengine to inherit from AutoRetryDocument

#32 Updated by dkliban@redhat.com about 4 years ago

  • Status changed from MODIFIED to ON_QA

#33 Updated by rbarlow almost 4 years ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#34 Updated by bmbouter 7 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF