When uploading an invalid rpm such as a non utf8 RPM, users should get a friendly error message
When users upload an invalid RPM such as one that has non-utf8 characters in their headers, they should get a 4xx response and a user friendly error message. Currently they receive a 500.
Steps to reproduce.
http POST http://localhost:8000/pulp/api/v3/repositories/ name=foo export REPO_HREF=$(http :8000/pulp/api/v3/repositories/ | jq -r '.results | select(.name == "foo") | ._href') wget https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-non-utf-8/rpm-with-non-utf-8-1-1.fc25.noarch.rpm http --form POST http://localhost:8000/pulp/api/v3/artifacts/ file@./rpm-with-non-utf-8-1-1.fc25.noarch.rpm http POST http://localhost:8000/pulp/api/v3/content/rpm/packages/ relative_path=rpm-with-non-utf-8-1-1.fc25.noarch.rpm artifact="/pulp/api/v3/artifacts/1/" filename=rpm-with-non-utf8.rpm
HTTP/1.1 500 Internal Server Error Content-Length: 20215 Content-Type: text/plain; charset=utf-8 Date: Thu, 29 Nov 2018 15:41:48 GMT Server: WSGIServer/0.2 CPython/3.7.1 Vary: Cookie X-Frame-Options: SAMEORIGIN UnicodeDecodeError at /pulp/api/v3/content/rpm/packages/ 'utf-8' codec can't decode byte 0x80 in position 168: invalid start byte Request Method: POST Request URL: http://localhost:8000/pulp/api/v3/content/rpm/packages/ Django Version: 2.1.3 Python Executable: /home/vagrant/.virtualenvs/pulp/bin/python3 Python Version: 3.7.1 Python Path: ['/home/vagrant/.virtualenvs/pulp/bin', '/usr/lib64/python37.zip', '/usr/lib64/python3.7', '/usr/lib64/python3.7/lib-dynload', '/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages', '/home/vagrant/devel/pulp/pulpcore', '/home/vagrant/devel/pulp/plugin', '/home/vagrant/pulp_rpm', '/home/vagrant/.virtualenvs/pulp/lib/python3.7/site-packages'] Server time: Thu, 29 Nov 2018 15:41:48 +0000 Installed Applications: ['django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'django_filters', 'drf_yasg', 'rest_framework', 'pulpcore.app', 'pulp_file.app.PulpFilePluginAppConfig', 'pulp_rpm.app.PulpRpmPluginAppConfig', 'crispy_forms', 'django_extensions'] Installed Middleware: ['django.middleware.security.SecurityMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware'] Traceback: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/core/handlers/exception.py" in inner 34. response = get_response(request) File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/core/handlers/base.py" in _get_response 126. response = self.process_exception_by_middleware(e, request) File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/core/handlers/base.py" in _get_response 124. response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/views/decorators/csrf.py" in wrapped_view 54. return view_func(*args, **kwargs) File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/viewsets.py" in view 116. return self.dispatch(request, *args, **kwargs) File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/views.py" in dispatch 495. response = self.handle_exception(exc) File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/views.py" in handle_exception 455. self.raise_uncaught_exception(exc) File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/views.py" in dispatch 492. response = handler(request, *args, **kwargs) File "/usr/lib64/python3.7/contextlib.py" in inner 74. return func(*args, **kwds) File "/home/vagrant/pulp_rpm/pulp_rpm/app/viewsets.py" in create 96. package = Package.createrepo_to_dict(cr_pkginfo) File "/home/vagrant/pulp_rpm/pulp_rpm/app/models.py" in createrepo_to_dict 243. 'description': getattr(package, CREATEREPO_PACKAGE_ATTRS.DESCRIPTION) or '', Exception Type: UnicodeDecodeError at /pulp/api/v3/content/rpm/packages/ Exception Value: 'utf-8' codec can't decode byte 0x80 in position 168: invalid start byte Request information: USER: admin GET: No GET data POST: No POST data FILES: No FILES data COOKIES: No cookie data META: CONTENT_LENGTH = '137' CONTENT_TYPE = 'application/json'
[vagrant@p3 ~]$ cat /etc/redhat-release Fedora release 29 (Twenty Nine)
#5 Updated by bmbouter over 1 year ago
The Pulp2 behavior for a RPM to Pulp2 with non utf-8 chars in certain fields would have those invalid chars removed. That's good for being able to sync repositories with invalidly formed RPMs. It's not good because Pulp is mutating the data in unexpected ways. For example now the checksum of that field won't validate correctly after passing through Pulp which is a problem. The real fix is for those RPMs to have their metadata correctly utf-8 encoded.
Note that per the spec in section 2.3.1, RPMs expect utf-8 encoding for various fields, so Pulp3 is correct in rejecting an encoding it can't interpret. Since we aren't getting active bug reports from users, I think we should close this for now. I'm leaving open to go back through triage.
Other ideas and thoughts/concerns are welcome.
#6 Updated by daviddavis over 1 year ago
- Subject changed from Pulp does not support upload of non utf8 RPM to When uploading an invalid rpm such as a non utf8 RPM, users should get a friendly error message
- Description updated (diff)
- Triaged changed from No to Yes
Updating issue to better handle invalid RPMs.
Please register to edit this issue