Actions
Issue #2584
closedMongo cursor times out during task pulp.server.managers.content.orphan.delete_all_orphans
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.8.7
Platform Release:
2.12.2
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 15
Quarter:
Description
On a very busy system, a user saw the cursor timeout:
Feb 2 04:13:52 satellite1 pulp: py.warnings:WARNING: (1955-62752) /usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py:228: DeprecationWarning: remove is deprecated. Use delete_one or delete_many instead.
Feb 2 04:13:52 satellite1 pulp: py.warnings:WARNING: (1955-62752) content_units_collection.remove(content_unit['_id'])
Feb 2 04:13:52 satellite1 pulp: py.warnings:WARNING: (1955-62752)
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) Task pulp.server.managers.content.orphan.delete_all_orphans[ece93a94-907b-4c1a-b4a4-474a99893be2] raised unexpected: CursorNotFound('Cursor not found, cursor id: 163452381930',)
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) Traceback (most recent call last):
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) R = retval = fun(*args, **kwargs)
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) return super(Task, self).__call__(*args, **kwargs)
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) return super(PulpTask, self).__call__(*args, **kwargs)
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) return self.run(*args, **kwargs)
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py", line 170, in delete_all_orphans
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) OrphanManager.delete_orphans_by_type(content_type_id)
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py", line 219, in delete_orphans_by_type
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) fields=['_id', '_storage_path']):
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/orphan.py", line 106, in generate_orphans_by_type
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) for content_unit in content_units_collection.find({}, projection=fields):
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) if len(self.__data) or self._refresh():
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1039, in _refresh
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) self.__max_await_time_ms))
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 903, in __send_message
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) codec_options=self.__codec_options)
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 114, in _unpack_response
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) raise CursorNotFound(msg, 43, errobj)
Feb 2 04:28:43 satellite1 pulp: celery.worker.job:ERROR: (1358-62752) CursorNotFound: Cursor not found, cursor id: 163452381930
Related issues
Actions
Set batch_size for the cursor in orphan removal tasks
On a busy system a cursor for iterating over all units in a collection could be timed out during orphan removal. The batch size is set to 100 to make sure cursor is active, it also should not introduce any noticable performance impact.
closes #2584 https://pulp.plan.io/issues/2584