Project

Profile

Help

Issue #2015

closed

rpm repo publish fails with "Incorrect length of data produced" error

Added by pthomas@redhat.com almost 8 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Urgent
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Master
Platform Release:
2.9.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 4
Quarter:

Description

Description

Description of problem:
Publishing errata is failing with fails with "Incorrect length of data produced" error

Version-Release number of selected component (if applicable):
2.9

How reproducible:
Always

Steps to Reproduce:

1.Create a rhel repo with feed

2. pulp-admin rpm repo sync run --repo-id=<repo-name>

Actual results:

[root@ibm-x3550m3-11 ~]# pulp-admin rpm repo create --repo-id rhel5-os --feed http://cdn.rcm-internal.redhat.com/content/dist/rhel/rhui/server/5/5.11/x86_64/os/ --download-policy on_demand
Successfully created repository [rhel5-os]

[root@ibm-x3550m3-11 ~]# pulp-admin rpm repo sync run --repo-id rhel5-os
+----------------------------------------------------------------------+
                  Synchronizing Repository [rhel5-os]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[|]
... completed

Downloading repository content...
[-]
[==================================================] 100%
RPMs:       13745/13745 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[\]
... completed

Importing package groups/categories...
[\]
... completed

Cleaning duplicate packages...
[\]
... completed

Task Succeeded

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
17117 of 17117 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[====                                              ] 9%
298 of 3135 items

Task Failed

Incorrect length of data produced

Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: py.warnings:WARNING: (32465-35008) /usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py:1109: DeprecationWarning: save is deprecated. Use insert_one or replace_one instead
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: py.warnings:WARNING: (32465-35008)   publish_result_coll.save(result)
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: py.warnings:WARNING: (32465-35008)
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008) Exception caught from plugin during publish for repo [rhel5-os]
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008) Traceback (most recent call last):
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1090, in _do_publish
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)     publish_report = publish_repo(transfer_repo, conduit, call_config)
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 658, in wrap_f
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)     return f(*args, **kwargs)
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 174, in publish_repo
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)     return self._publisher.process_lifecycle()
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 562, in process_lifecycle
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)     super(PluginStep, self).process_lifecycle()
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 159, in process_lifecycle
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)     step.process()
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 256, in process
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)     self.finalize()
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 608, in finalize
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)     self.context.checksum)
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/metadata/repomd.py", line 114, in add_metadata_file_metadata
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)     content = file_handle.read()
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)   File "/usr/lib64/python2.7/gzip.py", line 254, in read
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)     self._read(readsize)
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)   File "/usr/lib64/python2.7/gzip.py", line 325, in _read
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)     self._read_eof()
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)   File "/usr/lib64/python2.7/gzip.py", line 349, in _read_eof
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008)     raise IOError, "Incorrect length of data produced"
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.controllers.repository:ERROR: (32465-35008) IOError: Incorrect length of data produced
Jun 15 16:36:03 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32465]: pulp.server.async.tasks:INFO: Task failed : [1895b633-025d-4d26-987d-f3b900e4dfc3]
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008) Task pulp.server.managers.repo.publish.publish[1895b633-025d-4d26-987d-f3b900e4dfc3] raised unexpected: IOError('Incorrect length of data produced',)
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008) Traceback (most recent call last):
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)     R = retval = fun(*args, **kwargs)
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)     return super(Task, self).__call__(*args, **kwargs)
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)     return super(PulpTask, self).__call__(*args, **kwargs)
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)     return self.run(*args, **kwargs)
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 971, in publish
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)     result = check_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1038, in check_publish
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)     result = _do_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1090, in _do_publish
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)     publish_report = publish_repo(transfer_repo, conduit, call_config)
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 658, in wrap_f
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)     return f(*args, **kwargs)
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 174, in publish_repo
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)     return self._publisher.process_lifecycle()
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 562, in process_lifecycle
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)     super(PluginStep, self).process_lifecycle()
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 159, in process_lifecycle
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)     step.process()
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 256, in process
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)     self.finalize()
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 608, in finalize
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)     self.context.checksum)
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/metadata/repomd.py", line 114, in add_metadata_file_metadata
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)     content = file_handle.read()
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)   File "/usr/lib64/python2.7/gzip.py", line 254, in read
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)     self._read(readsize)
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)   File "/usr/lib64/python2.7/gzip.py", line 325, in _read
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)     self._read_eof()
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)   File "/usr/lib64/python2.7/gzip.py", line 349, in _read_eof
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008)     raise IOError, "Incorrect length of data produced"
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:ERROR: (32098-35008) IOError: Incorrect length of data produced
Jun 15 16:36:04 ibm-x3550m3-11.lab.eng.brq.redhat.com pulp[32098]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[646e9d1f-434c-4235-80c2-c8ebd0673483] succeeded in 0.00690435299475s: None

[root@ibm-x3550m3-11 ~]# rpm -qa |grep pulp
python-pulp-common-2.9.0-0.1.beta.git.286.fa65b74.el7.noarch
python-kombu-3.0.33-5.pulp.el7.noarch
python-pulp-docker-common-2.0.2-0.2.beta.git.48.0bab9ff.el7.noarch
pulp-selinux-2.9.0-0.1.beta.git.286.fa65b74.el7.noarch
pulp-docker-plugins-2.0.2-0.2.beta.git.48.0bab9ff.el7.noarch
python-pulp-client-lib-2.9.0-0.1.beta.git.286.fa65b74.el7.noarch
pulp-rpm-admin-extensions-2.9.0-0.1.beta.git.221.e93f53d.el7.noarch
pulp-python-admin-extensions-1.1.2-0.1.beta.git.12.7790d4a.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-pulp-puppet-common-2.9.0-0.1.beta.git.86.20ca773.el7.noarch
python-pulp-rpm-common-2.9.0-0.1.beta.git.221.e93f53d.el7.noarch
python-pulp-oid_validation-2.9.0-0.1.beta.git.286.fa65b74.el7.noarch
pulp-server-2.9.0-0.1.beta.git.286.fa65b74.el7.noarch
pulp-puppet-plugins-2.9.0-0.1.beta.git.86.20ca773.el7.noarch
python-pulp-bindings-2.9.0-0.1.beta.git.286.fa65b74.el7.noarch
pulp-puppet-admin-extensions-2.9.0-0.1.beta.git.86.20ca773.el7.noarch
pulp-docker-admin-extensions-2.0.2-0.2.beta.git.48.0bab9ff.el7.noarch
pulp-ostree-plugins-1.1.2-0.1.beta.git.11.9227383.el7.noarch
python-pulp-python-common-1.1.2-0.1.beta.git.12.7790d4a.el7.noarch
pulp-python-plugins-1.1.2-0.1.beta.git.12.7790d4a.el7.noarch
python-pulp-streamer-2.9.0-0.1.beta.git.286.fa65b74.el7.noarch
python-pulp-repoauth-2.9.0-0.1.beta.git.286.fa65b74.el7.noarch
pulp-rpm-plugins-2.9.0-0.1.beta.git.221.e93f53d.el7.noarch
pulp-admin-client-2.9.0-0.1.beta.git.286.fa65b74.el7.noarch
python-pulp-ostree-common-1.1.2-0.1.beta.git.11.9227383.el7.noarch
pulp-ostree-admin-extensions-1.1.2-0.1.beta.git.11.9227383.el7.noarch
[root@ibm-x3550m3-11 ~]#

Actions #1

Updated by bmbouter almost 8 years ago

This bug report is very difficult to read. Please use "pre" tags whenever linking to console output, configs, or logs.

Actions #2

Updated by pthomas@redhat.com almost 8 years ago

  • Description updated (diff)
Actions #3

Updated by pthomas@redhat.com almost 8 years ago

  • Description updated (diff)
Actions #4

Updated by bmbouter almost 8 years ago

Thanks @pthomas

Actions #5

Updated by semyers almost 8 years ago

Some IRC context:

12:42 <ipanova> preethi: okay, does it happen after some of my bug fix?
12:44 <ipanova> preethi: https://bugzilla.redhat.com/show_bug.cgi?id=1176698
12:44 <@pulpbot> Title: Bug 1176698 – rpm repo publish fails with "Incorrect length of data produced" error (at bugzilla.redhat.com)
12:49 <preethi> ipanova: this is on 2.9
12:53 <ipanova> preethi: i know but this commit removes the lines that were added to fix in 2.6
12:53 <ipanova> mhrivnak: you should probably take a look https://github.com/pulp/pulp_rpm/commit/d1491184a463ff30fd5923fcbc18e74005f6c3b7
12:53 <@pulpbot> Title: yum_distributor now uses configured checksum type for all metadata. · pulp/pulp_rpm@d149118 · GitHub (at github.com)
12:54 <ipanova> mhrivnak: seems like with ^ comoit this issue re-appears https://pulp.plan.io/issues/647
12:54 <@pulpbot> Title: Issue #647: rpm repo publish fails with "Incorrect length of data produced" error - Pulp (at pulp.plan.io)
13:22 <mhrivnak> ipanova, my commit just moved those lines. The three of them were consolidated to this one place: https://github.com/pulp/pulp_rpm/commit/d1491184a463ff30fd5923fcbc18e74005f6c3b7#diff-df55e4e5c922fdb2a0fefedecd7013c5R808
13:22 <@pulpbot> Title: yum_distributor now uses configured checksum type for all metadata. · pulp/pulp_rpm@d149118 · GitHub (at github.com)
13:23 <mhrivnak> And since it seems the error is happening during errata publish, I suspect the bug is elsewhere. Perhaps we need a similar line somewhere in the errata publish workflow.

My question is: Is this a regression in 2.9 that we need to call out in the release notes as a known issue, and potentially upgrade to a 2.9 blocker?

Actions #6

Updated by mhrivnak almost 8 years ago

This appears to be a regression in 2.9. Not being able to publish repos that contain errata is likely a 2.9 blocker.

Actions #7

Updated by semyers almost 8 years ago

  • Priority changed from Normal to Urgent
  • Severity changed from 2. Medium to 3. High
  • Platform Release set to 2.9.0
  • Triaged changed from No to Yes

I met with mhrivnak to do a blocker triage, and we both feel that the inability to sync errata from the CDN constitutes a release blocker: It is a regression in the most recent release, and breaks core pulp functionality.

The severity is high, not urgent, because the workaround is to not upgrade to pulp 2.9, which has not yet been released. :)

Preethi, can you check the same process against the latest 2.8 nightlies[0]? There's a very good chance that 2.8.5 introduced this regression, which was then merged forward to 2.9.

[0]: https://repos.fedorapeople.org/repos/pulp/pulp/testing/automation/2.8/stage/

Thanks!

Actions #8

Updated by mhrivnak almost 8 years ago

As Ina pointed out, this PR fixed a similar-looking issue in other metadata files: https://github.com/pulp/pulp_rpm/pull/605

Perhaps a similar change is required with updateinfo.

Actions #9

Updated by mhrivnak almost 8 years ago

I did this on master with Fedora 23:

pulp-admin rpm repo create --repo-id rhel5-os --feed http://cdn.rcm-internal.redhat.com/content/dist/rhel/rhui/server/5/5.11/x86_64/os/ --skip=rpm,distribution
pulp-admin rpm repo sync run --repo-id=rhel5-os

and saw this:

Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184) Task pulp.server.managers.repo.publish.publish[324e4e73-e733-49c8-8869-c49f67662e1d] raised unexpected: UnicodeEncodeError('ascii', u' title="Got Traceback while using \u201c/\u201d with \u2013serverurl register"', 34, 35, 'ordinal not in range(128)')
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184) Traceback (most recent call last):
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)     R = retval = fun(*args, **kwargs)
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)   File "/home/vagrant/devel/pulp/server/pulp/server/async/tasks.py", line 473, in __call__
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)     return super(Task, self).__call__(*args, **kwargs)
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)   File "/home/vagrant/devel/pulp/server/pulp/server/async/tasks.py", line 103, in __call__
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)     return super(PulpTask, self).__call__(*args, **kwargs)
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)     return self.run(*args, **kwargs)
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)   File "/home/vagrant/devel/pulp/server/pulp/server/controllers/repository.py", line 971, in publish
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)     result = check_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)   File "/home/vagrant/devel/pulp/server/pulp/server/controllers/repository.py", line 1038, in check_publish
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)     result = _do_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)   File "/home/vagrant/devel/pulp/server/pulp/server/controllers/repository.py", line 1090, in _do_publish
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)     publish_report = publish_repo(transfer_repo, conduit, call_config)
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)   File "/home/vagrant/devel/pulp/server/pulp/server/async/tasks.py", line 658, in wrap_f
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)     return f(*args, **kwargs)
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/distributors/yum/distributor.py", line 174, in publish_repo
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)     return self._publisher.process_lifecycle()
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)   File "/home/vagrant/devel/pulp/server/pulp/plugins/util/publish_step.py", line 562, in process_lifecycle
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)     super(PluginStep, self).process_lifecycle()
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)   File "/home/vagrant/devel/pulp/server/pulp/plugins/util/publish_step.py", line 159, in process_lifecycle
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)     step.process()
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)   File "/home/vagrant/devel/pulp/server/pulp/plugins/util/publish_step.py", line 232, in process
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)     self._process_block(item=item)
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)   File "/home/vagrant/devel/pulp/server/pulp/plugins/util/publish_step.py", line 291, in _process_block
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)     self.process_main(item=item)
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/distributors/yum/metadata/updateinfo.py", line 133, in add_unit_metadata
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)     self.xml_generator.completeElement('reference', reference_attributes, '')
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)   File "/home/vagrant/devel/pulp/server/pulp/plugins/util/saxwriter.py", line 79, in completeElement
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)     self.startElement(name, attrs)
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)   File "/home/vagrant/devel/pulp/server/pulp/plugins/util/saxwriter.py", line 114, in startElement
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)     self._write(' %s=%s' % (name, quoteattr(value)))
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)   File "/usr/lib64/python2.7/gzip.py", line 241, in write
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184)     self.fileobj.write(self.compress.compress(data))
Jun 15 19:00:48 dev pulp[3927]: celery.worker.job:ERROR: (3927-49184) UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' in position 34: ordinal not in range(128)
Actions #10

Updated by mhrivnak almost 8 years ago

Clearly it needs polish, but this rough patch to platform fixes the bug:

diff --git a/server/pulp/plugins/util/saxwriter.py b/server/pulp/plugins/util/saxwriter.py
index a860925..934401a 100644
--- a/server/pulp/plugins/util/saxwriter.py
+++ b/server/pulp/plugins/util/saxwriter.py
@@ -37,7 +37,7 @@ class XMLWriter(ContentHandler):
         :type  short_empty_elements: bool
         """
         ContentHandler.__init__(self)
-        self._write = stream.write
+        self._stream = stream
         self._flush = stream.flush
         self._encoding = encoding
         self._short_empty_elements = short_empty_elements
@@ -46,6 +46,11 @@ class XMLWriter(ContentHandler):
         self._indent_sep = '  '
         self._start_element = False

+    def _write(self, text):
+        if isinstance(text, unicode):
+            text = text.encode('utf-8')
+        self._stream.write(text)
+
     def _finish_pending_start_element(self):
         """
         Finish start of the element. Backported from Python 3.5.1 xml.sax.saxutils.XMLGenerator.
Actions #11

Updated by mhrivnak almost 8 years ago

  • Sprint/Milestone set to 22
Actions #12

Updated by mhrivnak almost 8 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to mhrivnak
Actions #13

Updated by mhrivnak almost 8 years ago

  • Status changed from ASSIGNED to POST

Added by mhrivnak almost 8 years ago

Revision 8661b9e5 | View on GitHub

saxwriter attributes and values can have utf-8 non-ascii chars

https://pulp.plan.io/issues/2015

fixes #2015

Added by mhrivnak almost 8 years ago

Revision 8661b9e5 | View on GitHub

saxwriter attributes and values can have utf-8 non-ascii chars

https://pulp.plan.io/issues/2015

fixes #2015

Actions #14

Updated by mhrivnak almost 8 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #15

Updated by pthomas@redhat.com over 7 years ago

  • Status changed from MODIFIED to 6

verified

[root@ibm-x3550m3-09 ~]# rpm -qa |grep pulp
python-pulp-common-2.9.0-0.2.beta.el7.noarch
python-kombu-3.0.33-5.pulp.el7.noarch
python-pulp-docker-common-2.0.2-0.2.beta.el7.noarch
python-pulp-repoauth-2.9.0-0.2.beta.el7.noarch
pulp-docker-plugins-2.0.2-0.2.beta.el7.noarch
python-pulp-client-lib-2.9.0-0.2.beta.el7.noarch
pulp-docker-admin-extensions-2.0.2-0.2.beta.el7.noarch
pulp-python-plugins-1.1.2-0.2.beta.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-pulp-puppet-common-2.9.0-0.2.beta.el7.noarch
python-pulp-rpm-common-2.9.0-0.2.beta.el7.noarch
python-pulp-oid_validation-2.9.0-0.2.beta.el7.noarch
pulp-server-2.9.0-0.2.beta.el7.noarch
pulp-puppet-plugins-2.9.0-0.2.beta.el7.noarch
python-pulp-bindings-2.9.0-0.2.beta.el7.noarch
pulp-puppet-admin-extensions-2.9.0-0.2.beta.el7.noarch
pulp-rpm-admin-extensions-2.9.0-0.2.beta.el7.noarch
pulp-ostree-plugins-1.1.2-0.1.beta.el7.noarch
python-pulp-python-common-1.1.2-0.2.beta.el7.noarch
pulp-python-admin-extensions-1.1.2-0.2.beta.el7.noarch
python-pulp-streamer-2.9.0-0.2.beta.el7.noarch
pulp-selinux-2.9.0-0.2.beta.el7.noarch
pulp-rpm-plugins-2.9.0-0.2.beta.el7.noarch
pulp-admin-client-2.9.0-0.2.beta.el7.noarch
python-pulp-ostree-common-1.1.2-0.1.beta.el7.noarch
pulp-ostree-admin-extensions-1.1.2-0.1.beta.el7.noarch
[root@ibm-x3550m3-09 ~]# 

[root@ibm-x3550m3-09 ~]#  pulp-admin rpm repo create --repo-id rhel5-os --feed http://cdn.rcm-internal.redhat.com/content/dist/rhel/rhui/server/5/5.11/x86_64/os/ --download-policy on_demand
Successfully created repository [rhel5-os]

[root@ibm-x3550m3-09 ~]# pulp-admin rpm repo sync run --repo-id=rhel5-os
+----------------------------------------------------------------------+
                  Synchronizing Repository [rhel5-os]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[-]
... completed

Downloading repository content...
[\]
[==================================================] 100%
RPMs:       17129/17129 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[\]
... completed

Cleaning duplicate packages...
[\]
... completed

Task Succeeded

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
17129 of 17129 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
3137 of 3137 items
... completed

Publishing Comps file
[==================================================] 100%
105 of 105 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Generating HTML files
... skipped

Publishing files to web
[|]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

[root@ibm-x3550m3-09 ~]# 
Actions #16

Updated by semyers over 7 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions #18

Updated by bmbouter about 6 years ago

  • Sprint set to Sprint 4
Actions #19

Updated by bmbouter about 6 years ago

  • Sprint/Milestone deleted (22)
Actions #20

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF