Actions
Issue #2502
closedcursor timeout in _duplicate_key_id_generator_aggregation() during sync
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.8.7
Platform Release:
2.12.1
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 14
Quarter:
Description
From bugzilla:
Description of problem:
Actions::Katello::ContentView::CapsuleGenerateAndSync failed on step Actions::Pulp::Consumer::SyncCapsule i.e. on pulp task pulp.server.managers.repo.sync.sync trying to synchronize a repo from Satellite to Capsule. Error string:
PulpExecutionException: Importer indicated a failed response
/var/log/messages has the reasoning behind:
Dec 15 19:23:38 capsule pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[d12f21ca-23b0-42e4-8270-ae405ec848f4]
..
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) Cursor not found, cursor id: 159997743246
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) Traceback (most recent call last):
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 288, in run
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) purge.remove_repo_duplicate_nevra(self.conduit.repo_id)
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 304, in remove_repo_duplicate_nevra
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) for unit_ids in _duplicate_key_id_generator(unit_type):
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 403, in _duplicate_key_id_generator_aggregation
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) for pkg in aggregation:
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) File "/usr/lib64/python2.7/site-packages/pymongo/command_cursor.py", line 232, in next
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) if len(self.__data) or self._refresh():
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) File "/usr/lib64/python2.7/site-packages/pymongo/command_cursor.py", line 190, in _refresh
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) self.__collection.codec_options))
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) File "/usr/lib64/python2.7/site-packages/pymongo/command_cursor.py", line 116, in __send_message
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) self.__collection.codec_options)
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 114, in _unpack_response
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) raise CursorNotFound(msg, 43, errobj)
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) CursorNotFound: Cursor not found, cursor id: 159997743246
Dec 15 21:19:54 ux0501 pulp: pulp.server.async.tasks:INFO: Task failed : [d12f21ca-23b0-42e4-8270-ae405ec848f4]
..
Dec 15 21:20:02 ux0501 pulp: celery.worker.job:ERROR: (25306-78624) Task pulp.server.managers.repo.sync.sync[d12f21ca-23b0-42e4-8270-ae405ec848f4] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
I.e. in source code:
/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py :
def _duplicate_key_id_generator_aggregation(unit, fields):
..
# When aggregating over hundreds of thousands of packages, mongo can overflow
# To prevent this, mongo needs to be allowed to temporarily use the disk for this transaction
aggregation = unit.objects.aggregate(sort, project, allowDiskUse=True)
# loop state tracking vars
previous_nevra = None
previous_pkg_id = None
yielding_ids = None
for pkg in aggregation:
..
the aggregation timeouts on its cursor.
Version-Release number of selected component (if applicable):
pulp-server-2.8.7.3-1.el7sat.noarch
How reproducible:
???
Steps to Reproduce:
1. Have several large repos being synced in parallel (doesnt matter if from CDN to Sat or from Sat to Caps, I guess)
Actual results:
See above failure of the pulp sync task
Expected results:
No failed task
Additional info:
I expect:
aggregate(sort, project, allowDiskUse=True).batch_size(10)
is a workaround (possibly with different batch size value).
Updated by bizhang over 7 years ago
- Priority changed from Normal to High
- Sprint/Milestone set to 31
- Severity changed from 2. Medium to 3. High
- Triaged changed from No to Yes
Updated by daviddavis over 7 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to daviddavis
Added by daviddavis over 7 years ago
Updated by daviddavis over 7 years ago
- Status changed from ASSIGNED to POST
Updated by daviddavis about 7 years ago
- Status changed from POST to MODIFIED
Applied in changeset b4d57ec8ff31c0a95df191579afbbe586036f177.
Updated by bizhang about 7 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
Updated by bmbouter about 6 years ago
- Sprint changed from Sprint 16 to Sprint 14
Actions
Prevent a cursor timeout when syncing rpm repos
I couldn't actually reproduce the cursor timeout using a variety of configurations for my VM but using a batch size here didn't seem to slow down syncs significantly.
fixes #2502 https://pulp.plan.io/issues/2502