Refactor #1084
closedTask #1014: Short Term Improvements for Pulp's use of MongoDB
Stop Pulp from using the Celery results backend
100%
Description
This is the fix for issue #956. If you assign this to yourself, you also need to assign #956 because you'll be fixing that issue with these changes.
Historically, Pulp has used the celery Results backend, but I don't think its needed anymore. By using it we are having mongoDB issues (see #956). The results backend was used in Pulp 2.4 so the resource_manager could tell httpd which queue to send a task into but that is being done differently now.
Deliverables¶
- Remove the create_mongo_config() function and it's associated tests
- Have Pulp stop having celery connect to MongoDB at all by removing the mongodb and results config and fixup associated tests.
- Rework the FailureWatcher feature as described below
- Ensure that the TaskResult object works as expected without the results backend present.
- Write a migration to delete the celery_taskmeta collection
- Write a release note for this change
- Run the integration suite before and after and ensure that this change does not break anything.
FailureWatcher¶
The results backend is used here in the FailureWatcher thread of celerybeat. The call to AsyncResult().result will fail when the results backend is removed so this needs to be done differently. I suggest the following:
- Delete the FailureWatcher entirely.
- Remove any vestige of the FailureWatcher from the rest of scheduler.py
- Reimplement the auto-disabling of scheduled actions which fail N times by creating the appropriate logic in the on_failure handler which is executed by Celery synchronously upon completion.
- update/write tests accordingly.
Related issues
Removes MongoDB as celery result backend
This patch also removes the FailureHandler which relied on checking the results backend to determine if a scheduled task needs to have it's schedule disabled after reaching a failure threshold. The logic is moved to the on_success and on_failure methods for Task defined in Pulp.
https://pulp.plan.io/issues/956 fixes #956 https://pulp.plan.io/issues/1084 fixes #1084