Project

Profile

Help

Issue #3344

Copy content from one repo to another fails

Added by balonik over 3 years ago. Updated over 2 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Hi,
I am hitting an issue while trying to copy content between RPM repositories.
I call Pulp API and send multiple tasks to copy content from multiple (3+) RPM repositories to 1 destination.
API calls are done immediatelly. I don't wait for the previous copy tasks to finish.

I get the below error for random source repos each try. I can reproduce it everytime.
Sometimes only 1 task fail, sometimes more tasks fail.

Environment:
RHEL 7.4
Pulp 2.15

API call:
url: /pulp/api/v2/repositories/destination_repo/actions/associate/
json body: {'source_repo_id': source, 'override_config': {'recursive': True}, 'criteria': { 'type_ids' : ['rpm','drpm','package_group','package_category','package_environment']}}

Logs:
Feb 5 11:28:51 pulp-server pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.unit_association.associate_from_repo[6fb6d4ac-8472-4a41-9cb0-92eccb08444d]
Feb 5 11:31:05 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=False, or create client after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#using-pymongo-with-multiprocessing>
Feb 5 11:31:05 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776) "MongoClient opened before fork. Create MongoClient "
Feb 5 11:31:05 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776)
Feb 5 11:31:06 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776) /usr/lib/python2.7/site-packages/mongoengine/document.py:324: DeprecationWarning: save is deprecated. Use insert_one or replace_one instead
Feb 5 11:31:06 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776) object_id = collection.save(doc, **write_concern)
Feb 5 11:31:06 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776)
Feb 5 11:31:13 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776) /usr/lib/python2.7/site-packages/mongoengine/document.py:367: DeprecationWarning: update is deprecated. Use replace_one, update_one or update_many instead.
Feb 5 11:31:13 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776) upsert=upsert, **write_concern)
Feb 5 11:31:13 pulp-server pulp: py.warnings:WARNING: [6fb6d4ac] (12260-91776)
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) Exception from importer [yum_importer] while importing units into repository [destination_repo]
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) Traceback (most recent call last):
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 273, in associate_from_repo
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) units=transfer_units)
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py", line 58, in import_units
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) return associate.associate(source_repo, dest_repo, import_conduit, config, units)
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/associate.py", line 88, in associate
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) associate(source_repo, dest_repo, import_conduit, config, group_units))
Feb 5 11:31:16 pulp-server pulp: pulp.server.managers.repo.unit_association:ERROR: [6fb6d4ac] (12260-91776) TypeError: unhashable type: 'list'
Feb 5 11:31:16 pulp-server pulp: pulp.server.async.tasks:INFO: [6fb6d4ac] Task failed : [6fb6d4ac-8472-4a41-9cb0-92eccb08444d]
Feb 5 11:31:16 pulp-server pulp: celery.worker.job:ERROR: (9966-91776) Task pulp.server.managers.repo.unit_association.associate_from_repo[6fb6d4ac-8472-4a41-9cb0-92eccb08444d] raised unexpected: TypeError("unhashable type: 'list'",)

History

#1 Updated by balonik over 3 years ago

The destination_repo is created by following json in API call:
body={'display_name': new_bundlename, 'id': new_bundlename, 'importer_type_id': 'yum_importer', 'importer_config': {}, 'distributors': [{'distributor_type_id': 'yum_distributor', 'distributor_config': { 'http': False, 'https': True, 'relative_url': "content/{0}".format(new_bundlename), 'generate_sqlite': True}, 'auto_publish': True}]}

#2 Updated by dalley over 3 years ago

  • Triaged changed from No to Yes

#4 Updated by bmbouter over 2 years ago

  • Status changed from NEW to CLOSED - WONTFIX

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

#5 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF