RPM Support

From bugzilla:

Description of problem:
Actions::Katello::ContentView::CapsuleGenerateAndSync failed on step Actions::Pulp::Consumer::SyncCapsule i.e. on pulp task pulp.server.managers.repo.sync.sync trying to synchronize a repo from Satellite to Capsule. Error string:

PulpExecutionException: Importer indicated a failed response

/var/log/messages has the reasoning behind:

Dec 15 19:23:38 capsule pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[d12f21ca-23b0-42e4-8270-ae405ec848f4]
..
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) Cursor not found, cursor id: 159997743246
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) Traceback (most recent call last):
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 288, in run
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)     purge.remove_repo_duplicate_nevra(self.conduit.repo_id)
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 304, in remove_repo_duplicate_nevra
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)     for unit_ids in _duplicate_key_id_generator(unit_type):
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 403, in _duplicate_key_id_generator_aggregation
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)     for pkg in aggregation:
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)   File "/usr/lib64/python2.7/site-packages/pymongo/command_cursor.py", line 232, in next
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)     if len(self.__data) or self._refresh():
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)   File "/usr/lib64/python2.7/site-packages/pymongo/command_cursor.py", line 190, in _refresh
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)     self.__collection.codec_options))
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)   File "/usr/lib64/python2.7/site-packages/pymongo/command_cursor.py", line 116, in __send_message
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)     self.__collection.codec_options)
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)   File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 114, in _unpack_response
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624)     raise CursorNotFound(msg, 43, errobj)
Dec 15 21:18:40 ux0501 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (25461-78624) CursorNotFound: Cursor not found, cursor id: 159997743246
Dec 15 21:19:54 ux0501 pulp: pulp.server.async.tasks:INFO: Task failed : [d12f21ca-23b0-42e4-8270-ae405ec848f4]
..
Dec 15 21:20:02 ux0501 pulp: celery.worker.job:ERROR: (25306-78624) Task pulp.server.managers.repo.sync.sync[d12f21ca-23b0-42e4-8270-ae405ec848f4] raised unexpected: PulpExecutionException('Importer indicated a failed response',)

I.e. in source code:
/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py :

def _duplicate_key_id_generator_aggregation(unit, fields):
..
    # When aggregating over hundreds of thousands of packages, mongo can overflow
    # To prevent this, mongo needs to be allowed to temporarily use the disk for this transaction
    aggregation = unit.objects.aggregate(sort, project, allowDiskUse=True)

    # loop state tracking vars
    previous_nevra = None
    previous_pkg_id = None
    yielding_ids = None

    for pkg in aggregation:
..

the aggregation timeouts on its cursor.

Version-Release number of selected component (if applicable):
pulp-server-2.8.7.3-1.el7sat.noarch

How reproducible:
???

Steps to Reproduce:
1. Have several large repos being synced in parallel (doesnt matter if from CDN to Sat or from Sat to Caps, I guess)

Actual results:
See above failure of the pulp sync task

Expected results:
No failed task

Additional info:
I expect:

aggregate(sort, project, allowDiskUse=True).batch_size(10)

is a workaround (possibly with different batch size value).