Project

Profile

Help

Issue #2495

When generating applicability data for multiple repositories, tasks throws error CursorNotFound

Added by cyrus_mc almost 5 years ago. Updated over 2 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.10.2
Platform Release:
OS:
CentOS 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

I attempted to generate applicability data for multiple repositories (6 to be exact).

The task ended up throwing the following error message at this point in code:

https://github.com/pulp/pulp/blob/pulp-2.10.2-1/server/pulp/server/managers/consumer/applicability.py#L129

Dec 15 15:25:43 pit-pulp-01 pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Dec 15 15:25:43 pit-pulp-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[541e1117-cc2f-4a26-affe-de3316c0cc2f]
Dec 15 15:39:32 pit-pulp-01 pulp: pulp.server.async.tasks:INFO: Task failed : [64996c8a-c749-488f-8951-60d1fe24976d]
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) Task pulp.server.managers.consumer.applicability.regenerate_applicability_for_repos[64996c8a-c749-488f-8951-60d1fe24976d] raised unexpected: CursorNotFound('Cursor not found, cursor id: 138653478868',)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) Traceback (most recent call last):
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688)     R = retval = fun(*args, **kwargs)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 488, in __call__
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688)     return super(Task, self).__call__(*args, **kwargs)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688)     return super(PulpTask, self).__call__(*args, **kwargs)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688)     return self.run(*args, **kwargs)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/consumer/applicability.py", line 129, in regenerate_applicability_for_repos
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688)     for existing_applicability in existing_applicabilities:
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688)     if len(self.__data) or self._refresh():
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1039, in _refresh
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688)     self.__max_await_time_ms))
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 903, in __send_message
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688)     codec_options=self.__codec_options)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688)   File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 114, in _unpack_response
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688)     raise CursorNotFound(msg, 43, errobj)
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:ERROR: (908-78688) CursorNotFound: Cursor not found, cursor id: 138653478868
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[112aba53-6f05-4738-9ccb-ee2bc5442b40] succeeded in 0.0691354039591s: None
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[ec3071fc-64a9-4d59-8c3c-e58f7adf082e]
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.queue_download_deferred[541e1117-cc2f-4a26-affe-de3316c0cc2f] succeeded in 0.0432541829068s: None
Dec 15 15:39:32 pit-pulp-01 pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.download_deferred[ec3071fc-64a9-4d60-8c3c-e58f7adf082e] succeeded in 0.123732904904s: None

In total I have around 220 consumers bind to those repositories.

This error is similar in nature to

https://pulp.plan.io/issues/998

Which was fixed in 2.6.3. I am currently running 2.10.2.

History

#1 Updated by mhrivnak almost 5 years ago

Upgrading to 2.10.3 or 2.11.0 is likely to help, by getting this fix:

https://pulp.plan.io/issues/2362

That doesn't fix the cursor issue necessarily, but a ~12x speedup in calculation should make it less likely that the cursor will timeout due to inactivity.

#2 Updated by cyrus_mc almost 5 years ago

I will try the upgrade.

To note if I generate the applicability individually it succeeds for each repository.

#3 Updated by mhrivnak almost 5 years ago

  • Description updated (diff)

#4 Updated by mhrivnak almost 5 years ago

To clarify: Pulp is making a query and not using results quickly enough. We should make whatever changes are necessary to use the results before the cursor times out.

The change mentioned in comment 1 does not fix this bug. It only makes it less likely that this bug will be hit by a user.

#5 Updated by bizhang almost 5 years ago

  • Severity changed from 2. Medium to 3. High

#6 Updated by bizhang almost 5 years ago

  • Triaged changed from No to Yes

#7 Updated by bmbouter over 2 years ago

  • Status changed from NEW to CLOSED - WONTFIX

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

#8 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF