Project

Profile

Help

Refactor #1084

Task #1014: Short Term Improvements for Pulp's use of MongoDB

Stop Pulp from using the Celery results backend

Added by bmbouter over 6 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.7.0
Groomed:
No
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Quarter:

Description

This is the fix for issue #956. If you assign this to yourself, you also need to assign #956 because you'll be fixing that issue with these changes.

Historically, Pulp has used the celery Results backend, but I don't think its needed anymore. By using it we are having mongoDB issues (see #956). The results backend was used in Pulp 2.4 so the resource_manager could tell httpd which queue to send a task into but that is being done differently now.

Deliverables

  • Remove the create_mongo_config() function and it's associated tests
  • Have Pulp stop having celery connect to MongoDB at all by removing the mongodb and results config and fixup associated tests.
  • Rework the FailureWatcher feature as described below
  • Ensure that the TaskResult object works as expected without the results backend present.
  • Write a migration to delete the celery_taskmeta collection
  • Write a release note for this change
  • Run the integration suite before and after and ensure that this change does not break anything.

FailureWatcher

The results backend is used here in the FailureWatcher thread of celerybeat. The call to AsyncResult().result will fail when the results backend is removed so this needs to be done differently. I suggest the following:

  • Delete the FailureWatcher entirely.
  • Remove any vestige of the FailureWatcher from the rest of scheduler.py
  • Reimplement the auto-disabling of scheduled actions which fail N times by creating the appropriate logic in the on_failure handler which is executed by Celery synchronously upon completion.
  • update/write tests accordingly.

Related issues

Related to Pulp - Issue #956: Pulp's Celery result backend connection cannot use Mongo replica sets with automatic failoverCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision aa4d57df View on GitHub
Added by dkliban@redhat.com about 6 years ago

Removes MongoDB as celery result backend

This patch also removes the FailureHandler which relied on checking the results backend to determine if a scheduled task needs to have it's schedule disabled after reaching a failure threshold. The logic is moved to the on_success and on_failure methods for Task defined in Pulp.

https://pulp.plan.io/issues/956 fixes #956 https://pulp.plan.io/issues/1084 fixes #1084

Revision aa4d57df View on GitHub
Added by dkliban@redhat.com about 6 years ago

Removes MongoDB as celery result backend

This patch also removes the FailureHandler which relied on checking the results backend to determine if a scheduled task needs to have it's schedule disabled after reaching a failure threshold. The logic is moved to the on_success and on_failure methods for Task defined in Pulp.

https://pulp.plan.io/issues/956 fixes #956 https://pulp.plan.io/issues/1084 fixes #1084

History

#1 Updated by bmbouter over 6 years ago

  • Description updated (diff)

#2 Updated by bmbouter over 6 years ago

  • Related to Issue #956: Pulp's Celery result backend connection cannot use Mongo replica sets with automatic failover added

#3 Updated by bmbouter over 6 years ago

  • Description updated (diff)

#4 Updated by mhrivnak over 6 years ago

  • Platform Release set to 2.6.4
  • Groomed changed from No to Yes

#5 Updated by mhrivnak over 6 years ago

  • Tracker changed from Story to Refactor

#6 Updated by bmbouter over 6 years ago

  • Description updated (diff)

#7 Updated by mhrivnak over 6 years ago

  • Priority changed from Normal to High

#8 Updated by bmbouter over 6 years ago

  • Platform Release deleted (2.6.4)

#9 Updated by dkliban@redhat.com about 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com

#10 Updated by dkliban@redhat.com about 6 years ago

  • Status changed from ASSIGNED to POST

#11 Updated by dkliban@redhat.com about 6 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#12 Updated by dkliban@redhat.com about 6 years ago

  • Platform Release set to 2.7.0
  • Groomed changed from Yes to No

#13 Updated by dkliban@redhat.com almost 6 years ago

  • Status changed from MODIFIED to 5

#14 Updated by rbarlow over 5 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#15 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF