Issue #2495
Updated by mhrivnak over 7 years ago
I attempted to generate applicability data for multiple repositories (6 to be exact).
The task ended up throwing the following error message at this point in code:
https://github.com/pulp/pulp/blob/pulp-2.10.2-1/server/pulp/server/managers/consumer/applicability.py#L129
<pre><code class="text">
Dec 15 15:25:43 pit-pulp-01 pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Dec 15 15:25:43 pit-pulp-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[541e1117-cc2f-4a26-affe-de3316c0cc2f]
Dec 15 15:39:32 pit-pulp-01 pulp: pulp.server.async.tasks:INFO: Task failed : [64996c8a-c749-488f-8951-60d1fe24976d]
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) Task pulp.server.managers.consumer.applicability.regenerate_applicability_for_repos[64996c8a-c749-488f-8951-60d1fe24976d] raised unexpected: CursorNotFound('Cursor not found, cursor id: 138653478868',)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) Traceback (most recent call last):
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) R = retval = fun(*args, **kwargs)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 488, in __call__
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) return super(Task, self).__call__(*args, **kwargs)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) return super(PulpTask, self).__call__(*args, **kwargs)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) return self.run(*args, **kwargs)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/applicability.py", line 129, in regenerate_applicability_for_repos
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) for existing_applicability in existing_applicabilities:
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) if len(self.__data) or self._refresh():
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1039, in _refresh
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) self.__max_await_time_ms))
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 903, in __send_message
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) codec_options=self.__codec_options)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 114, in _unpack_response
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) raise CursorNotFound(msg, 43, errobj)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) CursorNotFound: Cursor not found, cursor id: 138653478868
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[112aba53-6f05-4738-9ccb-ee2bc5442b40] succeeded in 0.0691354039591s: None
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[ec3071fc-64a9-4d59-8c3c-e58f7adf082e]
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.queue_download_deferred[541e1117-cc2f-4a26-affe-de3316c0cc2f] succeeded in 0.0432541829068s: None
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.download_deferred[ec3071fc-64a9-4d60-8c3c-e58f7adf082e] succeeded in 0.123732904904s: None
</code></pre>
In total I have around 220 consumers bind to those repositories.
This error is similar in nature to
https://pulp.plan.io/issues/998
Which was fixed in 2.6.3. I am currently running 2.10.2.
.