Project

Profile

Help

Refactor #1084

closed

Task #1014: Short Term Improvements for Pulp's use of MongoDB

Stop Pulp from using the Celery results backend

Added by bmbouter almost 9 years ago. Updated about 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.7.0
Groomed:
No
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Quarter:

Description

This is the fix for issue #956. If you assign this to yourself, you also need to assign #956 because you'll be fixing that issue with these changes.

Historically, Pulp has used the celery Results backend, but I don't think its needed anymore. By using it we are having mongoDB issues (see #956). The results backend was used in Pulp 2.4 so the resource_manager could tell httpd which queue to send a task into but that is being done differently now.

Deliverables

  • Remove the create_mongo_config() function and it's associated tests
  • Have Pulp stop having celery connect to MongoDB at all by removing the mongodb and results config and fixup associated tests.
  • Rework the FailureWatcher feature as described below
  • Ensure that the TaskResult object works as expected without the results backend present.
  • Write a migration to delete the celery_taskmeta collection
  • Write a release note for this change
  • Run the integration suite before and after and ensure that this change does not break anything.

FailureWatcher

The results backend is used here in the FailureWatcher thread of celerybeat. The call to AsyncResult().result will fail when the results backend is removed so this needs to be done differently. I suggest the following:

  • Delete the FailureWatcher entirely.
  • Remove any vestige of the FailureWatcher from the rest of scheduler.py
  • Reimplement the auto-disabling of scheduled actions which fail N times by creating the appropriate logic in the on_failure handler which is executed by Celery synchronously upon completion.
  • update/write tests accordingly.

Related issues

Related to Pulp - Issue #956: Pulp's Celery result backend connection cannot use Mongo replica sets with automatic failoverCLOSED - CURRENTRELEASEdkliban@redhat.comActions

Also available in: Atom PDF