Issue #2495
closed
When generating applicability data for multiple repositories, tasks throws error CursorNotFound
Description
I attempted to generate applicability data for multiple repositories (6 to be exact).
The task ended up throwing the following error message at this point in code:
Dec 15 15:25:43 pit-pulp-01 pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Dec 15 15:25:43 pit-pulp-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[541e1117-cc2f-4a26-affe-de3316c0cc2f]
Dec 15 15:39:32 pit-pulp-01 pulp: pulp.server.async.tasks:INFO: Task failed : [64996c8a-c749-488f-8951-60d1fe24976d]
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) Task pulp.server.managers.consumer.applicability.regenerate_applicability_for_repos[64996c8a-c749-488f-8951-60d1fe24976d] raised unexpected: CursorNotFound('Cursor not found, cursor id: 138653478868',)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) Traceback (most recent call last):
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) R = retval = fun(*args, **kwargs)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 488, in __call__
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) return super(Task, self).__call__(*args, **kwargs)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) return super(PulpTask, self).__call__(*args, **kwargs)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) return self.run(*args, **kwargs)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/applicability.py", line 129, in regenerate_applicability_for_repos
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) for existing_applicability in existing_applicabilities:
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) if len(self.__data) or self._refresh():
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1039, in _refresh
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) self.__max_await_time_ms))
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 903, in __send_message
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) codec_options=self.__codec_options)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 114, in _unpack_response
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) raise CursorNotFound(msg, 43, errobj)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) CursorNotFound: Cursor not found, cursor id: 138653478868
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[112aba53-6f05-4738-9ccb-ee2bc5442b40] succeeded in 0.0691354039591s: None
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[ec3071fc-64a9-4d59-8c3c-e58f7adf082e]
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.queue_download_deferred[541e1117-cc2f-4a26-affe-de3316c0cc2f] succeeded in 0.0432541829068s: None
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.download_deferred[ec3071fc-64a9-4d60-8c3c-e58f7adf082e] succeeded in 0.123732904904s: None
In total I have around 220 consumers bind to those repositories.
This error is similar in nature to
https://pulp.plan.io/issues/998
Which was fixed in 2.6.3. I am currently running 2.10.2.
Updated by mhrivnak almost 7 years ago
Upgrading to 2.10.3 or 2.11.0 is likely to help, by getting this fix:
https://pulp.plan.io/issues/2362
That doesn't fix the cursor issue necessarily, but a ~12x speedup in calculation should make it less likely that the cursor will timeout due to inactivity.
Updated by cyrus_mc almost 7 years ago
I will try the upgrade.
To note if I generate the applicability individually it succeeds for each repository.
Updated by mhrivnak almost 7 years ago
To clarify: Pulp is making a query and not using results quickly enough. We should make whatever changes are necessary to use the results before the cursor times out.
The change mentioned in comment 1 does not fix this bug. It only makes it less likely that this bug will be hit by a user.
Updated by bizhang almost 7 years ago
- Severity changed from 2. Medium to 3. High
Updated by bmbouter over 4 years ago
- Status changed from NEW to CLOSED - WONTFIX
Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.
.