Issue #1162: Importing ostree repo fails with ImporterConduitException - OSTree Support - Pulp

Actions

Send by e-mail Copy link

Issue #1162

closed

Importing ostree repo fails with ImporterConduitException

Added by jcline@redhat.com over 8 years ago. Updated about 5 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

High

Assignee:

jortel@redhat.com

Sprint/Milestone:

Start date:

Due date:

Estimated time:

Severity:

3. High

Version - OSTree:

master

Platform Release:

Target Release - OSTree:

1.0.0

OS:

RHEL 7

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Pulp 2

Sprint:

Quarter:

Description

I recall talking with Jeff Ortel about this, but I couldn't find a bug tracking it or anything. Syncing some OSTree repos fails with the traceback shown below. If I had to guess, it's some metadata that rpm-ostree creates that Mongo can't handle because of the '.'. This should be reproducible when syncing http://cdn.stage.redhat.com/content/dist/rhel/atomic/7/7Server/x86_64/ostree/repo/.

To reproduce:

Create a ostree repo with a feed of http://cdn.stage.redhat.com/content/dist/rhel/atomic/7/7Server/x86_64/ostree/repo/ and the branch 'rhel-atomic-host/7/x86_64/standard'
Sync it

I'm using pulp_ostree 'master'.

Jul 30 13:02:26 rhua.example.com pulp[31753]: pulp.server.async.tasks:INFO: Task failed : [d0df6999-4758-4aa9-a277-22ced8ceffba]
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328) Task pulp.server.managers.repo.sync.sync[d0df6999-4758-4aa9-a277-22ced8ceffba] raised unexpected: ImporterConduitException(Inv
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328) Traceback (most recent call last):
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     R = retval = fun(*args, **kwargs)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 394, in __call__
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     return super(Task, self).__call__(*args, **kwargs)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     return self.run(*args, **kwargs)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py", line 93, in sync
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     call_config)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py", line 147, in _do_sync
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     sync_report = sync_repo(transfer_repo, conduit, call_config)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 559, in wrap_f
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     return f(*args, **kwargs)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/web.py", line 91, in sync_repo
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     report = step.process_lifecycle()
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 501, in process_lifecycle
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     super(PluginStep, self).process_lifecycle()
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 125, in process_lifecycle
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     step.process()
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 202, in process
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     self._process_block()
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 244, in _process_block
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     self.process_main()
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/steps.py", line 165, in process_main
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     conduit.save_unit(_unit)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 487, in save_unit
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     unit.id = self._update_unit(unit, pulp_unit)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 521, in _update_unit
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     return self._add_unit(unit, pulp_unit)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 543, in _add_unit
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     unit_id = content_manager.add_content_unit(unit.type_id, None, pulp_unit)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/cud.py", line 35, in add_content_unit
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     collection.insert(unit_doc, safe=True)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/server/db/connection.py", line 167, in retry
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     return method(*args, **kwargs)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 415, in insert
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     self.uuid_subtype, client)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328) ImporterConduitException: key 'rpmostree.inputhash' must not contain '.'

Actions

Copy link

Updated by mhrivnak over 8 years ago

Priority changed from Normal to High
Severity changed from 2. Medium to 3. High
Triaged changed from No to Yes

Actions

Copy link

Updated by jortel@redhat.com over 8 years ago

I'm pretty sure the issue here is that commit metadata contains keys with ".". We just need to either strip those keys or make the keys safe. Like substitute "." with something like "-". Probably should be done by the importer step.