Project

Profile

Help

Issue #1162

Importing ostree repo fails with ImporterConduitException

Added by jcline@redhat.com over 6 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version - OSTree:
master
Platform Release:
Target Release - OSTree:
1.0.0
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

I recall talking with Jeff Ortel about this, but I couldn't find a bug tracking it or anything. Syncing some OSTree repos fails with the traceback shown below. If I had to guess, it's some metadata that rpm-ostree creates that Mongo can't handle because of the '.'. This should be reproducible when syncing http://cdn.stage.redhat.com/content/dist/rhel/atomic/7/7Server/x86_64/ostree/repo/.

To reproduce:

  1. Create a ostree repo with a feed of http://cdn.stage.redhat.com/content/dist/rhel/atomic/7/7Server/x86_64/ostree/repo/ and the branch 'rhel-atomic-host/7/x86_64/standard'
  2. Sync it

I'm using pulp_ostree 'master'.

Jul 30 13:02:26 rhua.example.com pulp[31753]: pulp.server.async.tasks:INFO: Task failed : [d0df6999-4758-4aa9-a277-22ced8ceffba]
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328) Task pulp.server.managers.repo.sync.sync[d0df6999-4758-4aa9-a277-22ced8ceffba] raised unexpected: ImporterConduitException(Inv
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328) Traceback (most recent call last):
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     R = retval = fun(*args, **kwargs)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 394, in __call__
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     return super(Task, self).__call__(*args, **kwargs)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     return self.run(*args, **kwargs)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py", line 93, in sync
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     call_config)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py", line 147, in _do_sync
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     sync_report = sync_repo(transfer_repo, conduit, call_config)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 559, in wrap_f
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     return f(*args, **kwargs)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/web.py", line 91, in sync_repo
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     report = step.process_lifecycle()
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 501, in process_lifecycle
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     super(PluginStep, self).process_lifecycle()
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 125, in process_lifecycle
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     step.process()
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 202, in process
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     self._process_block()
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 244, in _process_block
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     self.process_main()
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/steps.py", line 165, in process_main
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     conduit.save_unit(_unit)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 487, in save_unit
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     unit.id = self._update_unit(unit, pulp_unit)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 521, in _update_unit
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     return self._add_unit(unit, pulp_unit)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 543, in _add_unit
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     unit_id = content_manager.add_content_unit(unit.type_id, None, pulp_unit)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/cud.py", line 35, in add_content_unit
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     collection.insert(unit_doc, safe=True)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib/python2.7/site-packages/pulp/server/db/connection.py", line 167, in retry
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     return method(*args, **kwargs)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 415, in insert
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328)     self.uuid_subtype, client)
Jul 30 13:02:26 rhua.example.com pulp[31727]: celery.worker.job:ERROR: (31727-07328) ImporterConduitException: key 'rpmostree.inputhash' must not contain '.'

Associated revisions

Revision e6fcf226 View on GitHub
Added by jortel@redhat.com over 6 years ago

ref #1162 - replace (.) with (-) in commit metadata keys.

Revision a49c8be9 View on GitHub
Added by jortel@redhat.com over 6 years ago

Merge pull request #40 from jortel/issue-1162

ref #1162 - replace (.) with (-) in commit metadata keys.

History

#1 Updated by mhrivnak over 6 years ago

  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High
  • Triaged changed from No to Yes

#2 Updated by jortel@redhat.com over 6 years ago

I'm pretty sure the issue here is that commit metadata contains keys with ".". We just need to either strip those keys or make the keys safe. Like substitute "." with something like "-". Probably should be done by the importer step.

#3 Updated by jortel@redhat.com over 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com

#4 Updated by jortel@redhat.com over 6 years ago

  • Status changed from ASSIGNED to POST

#5 Updated by jortel@redhat.com over 6 years ago

  • Status changed from POST to MODIFIED

#6 Updated by jortel@redhat.com about 6 years ago

  • Target Release - OSTree set to 1.0.0

#7 Updated by rbarlow almost 6 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

#8 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF