RPM Support

What is the output of the locale binary on this system?

[root@localhost ~]# locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Here is the error I see from the traceback.

"Traceback (most recent call last):\n File \"/usr/lib/python2.6/site-packages/celery/app/trace.py\",
line 240, in trace_task\n R = retval = fun(*args, **kwargs)\n File \"/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py\",
line 328, in call\n return super(Task, self).__call__(*args, **kwargs)\n
\ File \"/usr/lib/python2.6/site-packages/celery/app/trace.py\", line 437, in
protected_call\n return self.run(*args, **kwargs)\n File \"/usr/lib/python2.6/site-packages/pulp/server/managers/repo/sync.py\",
line 114, in sync\n raise PulpExecutionException(_('Importer indicated a failed
response'))\nPulpExecutionException: Importer indicated a failed response\n"

URL used to get this stacktrace http://dl.fedoraproject.org/pub/fedora/linux/updates/22/x86_64/

Sep 24 09:09:51 katello pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
Sep 24 09:09:51 katello pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata files.
Sep 24 09:09:51 katello pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Sep 24 09:09:51 katello pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Sep 24 09:09:51 katello pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Sep 24 09:09:51 katello pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Sep 24 09:09:51 katello pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Sep 24 09:10:05 katello pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Generating metadata databases.
Sep 24 09:10:43 katello pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Determining which units need to be downloaded.
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) sync failed
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) Traceback (most recent call last):
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 148, in run
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) self.update_content(metadata_files)
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 324, in update_content
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) rpms_to_download, drpms_to_download = self._decide_what_to_download(metadata_files)
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 342, in decide_what_to_download
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) rpms_to_download, rpms_count, rpms_total_size = self._decide_rpms_to_download(metadata_files)
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 374, in _decide_rpms_to_download
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) wanted = self._identify_wanted_versions(package_info_generator)
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 687, in _identify_wanted_versions
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) for model in package_info_generator:
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py", line 63, in package_list_generator
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) package_info = process_func(element)
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/primary.py", line 180, in process_package_element
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) model.raw_xml = utils.element_to_raw_xml(package_element, [rpm_namespace], COMMON_SPEC_URL)
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/utils.py", line 55, in element_to_raw_xml
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) tree.write(io)
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) File "/usr/lib64/python2.6/xml/etree/ElementTree.py", line 663, in write
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) self._write(file, self._root, encoding, {})
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) File "/usr/lib64/python2.6/xml/etree/ElementTree.py", line 707, in _write
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) self._write(file, n, encoding, namespaces)
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) File "/usr/lib64/python2.6/xml/etree/ElementTree.py", line 705, in _write
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) file.write(_escape_cdata(node.text, encoding))
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) File "/usr/lib64/python2.6/xml/etree/ElementTree.py", line 807, in _escape_cdata
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) return _encode_entity(text)
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) File "/usr/lib64/python2.6/xml/etree/ElementTree.py", line 792, in _encode_entity
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) return _encode(pattern.sub(escape_entities, text), "ascii")
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) File "/usr/lib64/python2.6/xml/etree/ElementTree.py", line 751, in _encode
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) return s.encode(encoding)
Sep 24 09:10:46 katello pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (20230-53344) UnicodeEncodeError: 'ascii' codec can't encode character u'\U000ff004' in position 145: ordinal not in range(128)
Sep 24 09:10:46 katello pulp: pulp.server.event.http:INFO: (20230-53344) {'call_report': {u'exception': None, u'task_type': u'pulp.server.managers.repo.sync.sync', u'task_id': u'837d0c2d-a5fb-4e10-b3ee-23be79469787', u'tags': [u'pulp:repository:Default_Organization-Fedora_22-updates', u'pulp:action:sync'], u'finish_time': None, u'_ns': u'task_status', u'start_time': u'2015-09-24T13:09:50Z', u'traceback': None, u'spawned_tasks': [], u'progress_report': {u'yum_importer': {u'content': {u'size_total': 0, u'items_left': 0, u'items_total': 0, u'state': u'FAILED', u'size_left': 0, u'details': {u'rpm_total': 0, u'rpm_done': 0, u'drpm_total': 0, u'drpm_done': 0}, u'error': u"'ascii' codec can't encode character u'\\U000ff004' in position 145: ordinal not in range(128)", u'error_details': []}, u'comps': {u'state': u'NOT_STARTED'}, u'distribution': {u'items_total': 0, u'state': u'NOT_STARTED', u'error_details': [], u'items_left': 0}, u'errata': {u'state': u'NOT_STARTED'}, u'metadata': {u'state': u'FINISHED'}}}, u'state': u'running', u'worker_name': u'reserved_resource_worker-0@katello.local', u'result': None, u'error': None, u'_id': ObjectId('5603f61e38ad1e31a61a0ecb'), u'id': u'5603f61efa74004ed4171677'}, 'event_type': 'repo.sync.finish', 'payload': {'importer_id': u'yum_importer', 'exception': None, 'repo_id': u'Default_Organization-Fedora_22-updates', 'traceback': None, 'started': '2015-09-24T13:09:50Z', '_ns': u'repo_sync_results', 'completed': '2015-09-24T13:10:46Z', 'importer_type_id': u'yum_importer', 'error_message': None, 'summary': {'content': {'state': 'FAILED'}, 'comps': {'state': 'NOT_STARTED'}, 'distribution': {'state': 'NOT_STARTED'}, 'errata': {'state': 'NOT_STARTED'}, 'metadata': {'state': 'FINISHED'}}, 'added_count': 0, 'result': 'failed', 'updated_count': 0, 'details': {'content': {'size_total': 0, 'items_left': 0, 'items_total': 0, 'state': 'FAILED', 'size_left': 0, 'details': {'rpm_total': 0, 'rpm_done': 0, 'drpm_total': 0, 'drpm_done': 0}, 'error': "'ascii' codec can't encode character u'\\U000ff004' in position 145: o
Sep 24 09:10:46 katello pulp: pulp.server.event.http:INFO: (20230-53344) rdinal not in range(128)", 'error_details': []}, 'comps': {'state': 'NOT_STARTED'}, 'distribution': {'items_total': 0, 'state': 'NOT_STARTED', 'error_details': [], 'items_left': 0}, 'errata': {'state': 'NOT_STARTED'}, 'metadata': {'state': 'FINISHED'}}, 'id': '5603f656fa74004f0671d6c7', 'removed_count': 0}}
Sep 24 09:10:46 katello pulp: celery.worker.job:ERROR: (20190-53344) Task pulp.server.managers.repo.sync.sync[837d0c2d-a5fb-4e10-b3ee-23be79469787] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Sep 24 09:10:46 katello pulp: celery.worker.job:ERROR: (20190-53344) Traceback (most recent call last):
Sep 24 09:10:46 katello pulp: celery.worker.job:ERROR: (20190-53344) File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task
Sep 24 09:10:46 katello pulp: celery.worker.job:ERROR: (20190-53344) R = retval = fun(*args, **kwargs)
Sep 24 09:10:46 katello pulp: celery.worker.job:ERROR: (20190-53344) File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 328, incall_
Sep 24 09:10:46 katello pulp: celery.worker.job:ERROR: (20190-53344) return super(Task, self).__call__(*args, **kwargs)
Sep 24 09:10:46 katello pulp: celery.worker.job:ERROR: (20190-53344) File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in protected_call
Sep 24 09:10:46 katello pulp: celery.worker.job:ERROR: (20190-53344) return self.run(*args, **kwargs)
Sep 24 09:10:46 katello pulp: celery.worker.job:ERROR: (20190-53344) File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/sync.py", line 114, in sync
Sep 24 09:10:46 katello pulp: celery.worker.job:ERROR: (20190-53344) raise PulpExecutionException(_('Importer indicated a failed response'))
Sep 24 09:10:46 katello pulp: celery.worker.job:ERROR: (20190-53344) PulpExecutionException: Importer indicated a failed response

The OP noted that this error happens in RHEL 6, but not in RHEL 7.

The issue lives here:

https://github.com/pulp/pulp_rpm/blob/2.6-release/plugins/pulp_rpm/plugins/importers/yum/utils.py#L53-L55

That code is writing the ElementTree to a StringIO object, but does not specify any encoding. According to the Python documentation, the default encoding for ElementTree.write() is US-ASCII:

https://docs.python.org/2/library/xml.etree.elementtree.html#xml.etree.ElementTree.ElementTree.write

Thus, it is impossible to represent non-ASCII characters when this write operation happens.

Changing the line tree.write(io) to tree.write(io, encoding='UTF-8') solves the issue here:

https://github.com/pulp/pulp_rpm/blob/2.6-release/plugins/pulp_rpm/plugins/importers/yum/utils.py#L55

Created pull request #715. This changed allowed Fedora 21 to sync without issue. Fedora 22 had an errata issue which I'm opening up another ticket for.

did like rbarlow suggested and it did not help:

Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144) sync failed
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144) Traceback (most recent call last):
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 115, i
n run
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)     self.update_content(metadata_files)
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 273, i
n update_content
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)     rpms_to_download, drpms_to_download = self._decide_what_to_download(metadata_files)
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 291, i
n _decide_what_to_download
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)     rpms_to_download, rpms_count, rpms_total_size = self._decide_rpms_to_download(metadata_fi
les)
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 323, i
n _decide_rpms_to_download
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)     wanted = self._identify_wanted_versions(package_info_generator)
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 636, i
n _identify_wanted_versions
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)     for model in package_info_generator:
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py", 
line 63, in package_list_generator
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)     package_info = process_func(element)
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/primary.py", line 180, in process_package_element
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)     model.raw_xml = utils.element_to_raw_xml(package_element, [rpm_namespace], COMMON_SPEC_URL)
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/utils.py", line 55, in element_to_raw_xml
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)     tree.write(io, encoding='UTF-8')
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)   File "/usr/lib/python2.6/xml/etree/ElementTree.py", line 663, in write
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)     self._write(file, self._root, encoding, {})
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)   File "/usr/lib/python2.6/xml/etree/ElementTree.py", line 707, in _write
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)     self._write(file, n, encoding, namespaces)
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)   File "/usr/lib/python2.6/xml/etree/ElementTree.py", line 705, in _write
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)     file.write(_escape_cdata(node.text, encoding))
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)   File "/usr/lib/python2.6/xml/etree/ElementTree.py", line 807, in _escape_cdata
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)     return _encode_entity(text)
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)   File "/usr/lib/python2.6/xml/etree/ElementTree.py", line 792, in _encode_entity
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)     return _encode(pattern.sub(escape_entities, text), "ascii")
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)   File "/usr/lib/python2.6/xml/etree/ElementTree.py", line 751, in _encode
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144)     return s.encode(encoding)
Sep 29 11:52:35 ec2-54-78-179-248 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (14036-02144) UnicodeEncodeError: 'ascii' codec can't encode character u'\U000ff004' in position 145: ordinal not in range(128)

Ina and I discussed this on IRC, and restarting the workers seemed to fix the issue she reported.

As I've done more testing, when I made the code change to my system, there were no errors in the publish, but when I tried to update my pulp host from it's own centos 6 repos, which were synced after the code change, I'm getting the error TypeError: Parsing primary.xml error: XML declaration allowed only at start of the document.