Refactor #1084
closedTask #1014: Short Term Improvements for Pulp's use of MongoDB
Stop Pulp from using the Celery results backend
100%
Description
This is the fix for issue #956. If you assign this to yourself, you also need to assign #956 because you'll be fixing that issue with these changes.
Historically, Pulp has used the celery Results backend, but I don't think its needed anymore. By using it we are having mongoDB issues (see #956). The results backend was used in Pulp 2.4 so the resource_manager could tell httpd which queue to send a task into but that is being done differently now.
Deliverables¶
- Remove the create_mongo_config() function and it's associated tests
- Have Pulp stop having celery connect to MongoDB at all by removing the mongodb and results config and fixup associated tests.
- Rework the FailureWatcher feature as described below
- Ensure that the TaskResult object works as expected without the results backend present.
- Write a migration to delete the celery_taskmeta collection
- Write a release note for this change
- Run the integration suite before and after and ensure that this change does not break anything.
FailureWatcher¶
The results backend is used here in the FailureWatcher thread of celerybeat. The call to AsyncResult().result will fail when the results backend is removed so this needs to be done differently. I suggest the following:
- Delete the FailureWatcher entirely.
- Remove any vestige of the FailureWatcher from the rest of scheduler.py
- Reimplement the auto-disabling of scheduled actions which fail N times by creating the appropriate logic in the on_failure handler which is executed by Celery synchronously upon completion.
- update/write tests accordingly.
Related issues
Updated by bmbouter over 9 years ago
- Related to Issue #956: Pulp's Celery result backend connection cannot use Mongo replica sets with automatic failover added
Updated by mhrivnak over 9 years ago
- Platform Release set to 2.6.4
- Groomed changed from No to Yes
Updated by dkliban@redhat.com over 9 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to dkliban@redhat.com
Updated by dkliban@redhat.com about 9 years ago
- Status changed from ASSIGNED to POST
Added by dkliban@redhat.com about 9 years ago
Added by dkliban@redhat.com about 9 years ago
Revision aa4d57df | View on GitHub
Removes MongoDB as celery result backend
This patch also removes the FailureHandler which relied on checking the results backend to determine if a scheduled task needs to have it's schedule disabled after reaching a failure threshold. The logic is moved to the on_success and on_failure methods for Task defined in Pulp.
https://pulp.plan.io/issues/956 fixes #956 https://pulp.plan.io/issues/1084 fixes #1084
Updated by dkliban@redhat.com about 9 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset pulp|aa4d57dfd60cb9ac34ec0ce79d95f5c03e1dbf55.
Added by dkliban@redhat.com about 9 years ago
Revision 4be24193 | View on GitHub
Drops celery_taskmeta collection
Added by dkliban@redhat.com about 9 years ago
Revision 4be24193 | View on GitHub
Drops celery_taskmeta collection
Updated by dkliban@redhat.com about 9 years ago
- Platform Release set to 2.7.0
- Groomed changed from Yes to No
Updated by dkliban@redhat.com about 9 years ago
- Status changed from MODIFIED to 5
Updated by rbarlow almost 9 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
Removes MongoDB as celery result backend
This patch also removes the FailureHandler which relied on checking the results backend to determine if a scheduled task needs to have it's schedule disabled after reaching a failure threshold. The logic is moved to the on_success and on_failure methods for Task defined in Pulp.
https://pulp.plan.io/issues/956 fixes #956 https://pulp.plan.io/issues/1084 fixes #1084