RPM Support

Let's say that one has created an RPM repository with an ID of "foo". One can search for content units in that RPM repository by making an HTTP POST request to /pulp/api/v2/repositories/foo/search/units/ with a body of at least {"criteria": {}}. (A more typical body is {"criteria": {"type_ids": ["rpm"]}}.) Unfortunately, executing such a search causes Pulp to return an HTTP 500. The following is logged:

Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: Unhandled Exception
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856) 'utf8' codec can't decode byte 0x9c in position 1: invalid start byte
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856) Traceback (most recent call last):
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)   File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)     response = wrapped_callback(request, *callback_args, **callback_kwargs)
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)     return self.dispatch(request, *args, **kwargs)
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)     return handler(request, *args, **kwargs)
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 241, in _auth_decorator
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)     return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 195, in _verify_auth
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)     value = method(self, *args, **kwargs)
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py", line 130, in wrapper
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)     return func(*args, **kwargs)
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/search.py", line 127, in post
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)     return self._generate_response(query, options, *args, **kwargs)
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py", line 294, in _generate_response
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)     return generate_json_response_with_pulp_encoder(units)
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py", line 52, in generate_json_response
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)     json_obj = json.dumps(content, default=default)
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)   File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)     sort_keys=sort_keys, **kw).encode(obj)
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)   File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)     chunks = self.iterencode(o, _one_shot=True)
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)   File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856)     return _iterencode(o, 0)
Mar 06 11:33:11 rhel-7-3-pulp-2-12 pulp[3309]: pulp.server.webservices.middleware.exception:ERROR: (3309-85856) UnicodeDecodeError: 'utf8' codec can't decode byte 0x9c in position 1: invalid start byte

A similar error is logged for all RPM repository searches. Triggering this failure is as easy as executing the following script:

pulp-admin rpm repo create --repo-id foo --feed https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-unsigned/
pulp-admin rpm repo sync run --repo-id foo
pulp-admin rpm repo content rpm --repo-id foo

The final command spits out an error:

$ pulp-admin rpm repo content rpm --repo-id foo                                                            
An internal error occurred on the Pulp server:

RequestException: POST request
on /pulp/api/v2/repositories/foo/search/units/ failed with 500 - 'utf8' codec
can't decode byte 0x9c in position 1: invalid start byte

This error is present for Pulp 2.12 and 2.13 nightlies on all supported platforms. The error is reproducible on both Jenkins and on my personal systems. Here's the packages installed on one of my systems:

[root@rhel-7-3-pulp-2-12 ~]# rpm -qa | grep -i pulp | sort
pulp-admin-client-2.12.2-0.1.alpha.git.17.8ca5cf2.el7.noarch
pulp-docker-admin-extensions-2.3.1-0.1.alpha.git.5.052c506.el7.noarch
pulp-docker-plugins-2.3.1-0.1.alpha.git.5.052c506.el7.noarch
pulp-ostree-admin-extensions-1.2.1-0.1.alpha.git.27.7f9a84b.el7.noarch
pulp-ostree-plugins-1.2.1-0.1.alpha.git.27.7f9a84b.el7.noarch
pulp-puppet-admin-extensions-2.12.2-0.1.alpha.git.2.f338f5d.el7.noarch
pulp-puppet-plugins-2.12.2-0.1.alpha.git.2.f338f5d.el7.noarch
pulp-python-admin-extensions-2.0.1-0.1.alpha.git.6.8c46f3f.el7.noarch
pulp-python-plugins-2.0.1-0.1.alpha.git.6.8c46f3f.el7.noarch
pulp-rpm-admin-extensions-2.12.2-0.1.alpha.git.19.da51b5f.el7.noarch
pulp-rpm-plugins-2.12.2-0.1.alpha.git.19.da51b5f.el7.noarch
pulp-selinux-2.12.2-0.1.alpha.git.17.8ca5cf2.el7.noarch
pulp-server-2.12.2-0.1.alpha.git.17.8ca5cf2.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-6.pulp.el7.noarch
python-pulp-bindings-2.12.2-0.1.alpha.git.17.8ca5cf2.el7.noarch
python-pulp-client-lib-2.12.2-0.1.alpha.git.17.8ca5cf2.el7.noarch
python-pulp-common-2.12.2-0.1.alpha.git.17.8ca5cf2.el7.noarch
python-pulp-docker-common-2.3.1-0.1.alpha.git.5.052c506.el7.noarch
python-pulp-oid_validation-2.12.2-0.1.alpha.git.17.8ca5cf2.el7.noarch
python-pulp-ostree-common-1.2.1-0.1.alpha.git.27.7f9a84b.el7.noarch
python-pulp-puppet-common-2.12.2-0.1.alpha.git.2.f338f5d.el7.noarch
python-pulp-python-common-2.0.1-0.1.alpha.git.6.8c46f3f.el7.noarch
python-pulp-repoauth-2.12.2-0.1.alpha.git.17.8ca5cf2.el7.noarch
python-pulp-rpm-common-2.12.2-0.1.alpha.git.19.da51b5f.el7.noarch
python-pulp-streamer-2.12.2-0.1.alpha.git.17.8ca5cf2.el7.noarch