Issue #3344
closed
Copy content from one repo to another fails
Description
Hi,
I am hitting an issue while trying to copy content between RPM repositories.
I call Pulp API and send multiple tasks to copy content from multiple (3+) RPM repositories to 1 destination.
API calls are done immediatelly. I don't wait for the previous copy tasks to finish.
I get the below error for random source repos each try. I can reproduce it everytime.
Sometimes only 1 task fail, sometimes more tasks fail.
Environment:
RHEL 7.4
Pulp 2.15
API call:
url: /pulp/api/v2/repositories/destination_repo/actions/associate/
json body: {'source_repo_id': source, 'override_config': {'recursive': True}, 'criteria': { 'type_ids' : ['rpm','drpm','package_group','package_category','package_environment']}}
Logs:
Feb 5 11:28:51 pulp-server pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.unit_association.associate_from_repo[6fb6d4ac-8472-4a41-9cb0-92eccb08444d]
Feb 5 11:31:05 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing>
Feb 5 11:31:05 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776) "MongoClient opened before fork. Create MongoClient "
Feb 5 11:31:05 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776)
Feb 5 11:31:06 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776) /usr/lib/python2.7/site-packages/mongoengine/document.py:324: DeprecationWarning: save is deprecated. Use insert_one or replace_one instead
Feb 5 11:31:06 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776) object_id = collection.save(doc, **write_concern)
Feb 5 11:31:06 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776)
Feb 5 11:31:13 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776) /usr/lib/python2.7/site-packages/mongoengine/document.py:367: DeprecationWarning: update is deprecated. Use replace_one, update_one or update_many instead.
Feb 5 11:31:13 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776) upsert=upsert, **write_concern)
Feb 5 11:31:13 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776)
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) Exception from importer [yum_importer] while importing units into repository [destination_repo]
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) Traceback (most recent call last):
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 273, in associate_from_repo
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) units=transfer_units)
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py", line 58, in import_units
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) return associate.associate(source_repo, dest_repo, import_conduit, config, units)
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py", line 88, in associate
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) associate(source_repo, dest_repo, import_conduit, config, group_units))
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) TypeError: unhashable type: 'list'
Feb 5 11:31:16 pulp-server pulp: pulp.server.async.tasks:INFO: [6fb6d4ac] Task failed : [6fb6d4ac-8472-4a41-9cb0-92eccb08444d]
Feb 5 11:31:16 pulp-server pulp: celery.worker.job:ERROR: (9966-91776) Task pulp.server.managers.repo.unit_association.associate_from_repo[6fb6d4ac-8472-4a41-9cb0-92eccb08444d] raised unexpected: TypeError("unhashable type: 'list'",)
Updated by balonik about 6 years ago
The destination_repo is created by following json in API call:
body={'display_name': new_bundlename, 'id': new_bundlename, 'importer_type_id': 'yum_importer', 'importer_config': {}, 'distributors': [{'distributor_type_id': 'yum_distributor', 'distributor_config': { 'http': False, 'https': True, 'relative_url': "content/{0}".format(new_bundlename), 'generate_sqlite': True}, 'auto_publish': True}]}
Updated by bmbouter about 5 years ago
- Status changed from NEW to CLOSED - WONTFIX
Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.
.