Issue #3344
closedCopy content from one repo to another fails
Description
Hi,
I am hitting an issue while trying to copy content between RPM repositories.
I call Pulp API and send multiple tasks to copy content from multiple (3+) RPM repositories to 1 destination.
API calls are done immediatelly. I don't wait for the previous copy tasks to finish.
I get the below error for random source repos each try. I can reproduce it everytime.
Sometimes only 1 task fail, sometimes more tasks fail.
Environment:
RHEL 7.4
Pulp 2.15
API call:
url: /pulp/api/v2/repositories/destination_repo/actions/associate/
json body: {'source_repo_id': source, 'override_config': {'recursive': True}, 'criteria': { 'type_ids' : ['rpm','drpm','package_group','package_category','package_environment']}}
Logs:
Feb 5 11:28:51 pulp-server pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.unit_association.associate_from_repo[6fb6d4ac-8472-4a41-9cb0-92eccb08444d]
Feb 5 11:31:05 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing>
Feb 5 11:31:05 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776) "MongoClient opened before fork. Create MongoClient "
Feb 5 11:31:05 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776)
Feb 5 11:31:06 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776) /usr/lib/python2.7/site-packages/mongoengine/document.py:324: DeprecationWarning: save is deprecated. Use insert_one or replace_one instead
Feb 5 11:31:06 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776) object_id = collection.save(doc, **write_concern)
Feb 5 11:31:06 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776)
Feb 5 11:31:13 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776) /usr/lib/python2.7/site-packages/mongoengine/document.py:367: DeprecationWarning: update is deprecated. Use replace_one, update_one or update_many instead.
Feb 5 11:31:13 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776) upsert=upsert, **write_concern)
Feb 5 11:31:13 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776)
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) Exception from importer [yum_importer] while importing units into repository [destination_repo]
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) Traceback (most recent call last):
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 273, in associate_from_repo
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) units=transfer_units)
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py", line 58, in import_units
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) return associate.associate(source_repo, dest_repo, import_conduit, config, units)
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py", line 88, in associate
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) associate(source_repo, dest_repo, import_conduit, config, group_units))
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) TypeError: unhashable type: 'list'
Feb 5 11:31:16 pulp-server pulp: pulp.server.async.tasks:INFO: [6fb6d4ac] Task failed : [6fb6d4ac-8472-4a41-9cb0-92eccb08444d]
Feb 5 11:31:16 pulp-server pulp: celery.worker.job:ERROR: (9966-91776) Task pulp.server.managers.repo.unit_association.associate_from_repo[6fb6d4ac-8472-4a41-9cb0-92eccb08444d] raised unexpected: TypeError("unhashable type: 'list'",)