Project

Profile

Help

Issue #4215

When uploading an invalid rpm such as a non utf8 RPM, users should get a friendly error message

Added by kersom almost 2 years ago. Updated over 1 year ago.

Status:
NEW
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

When users upload an invalid RPM such as one that has non-utf8 characters in their headers, they should get a 4xx response and a user friendly error message. Currently they receive a 500.

Steps to reproduce.

http POST http://localhost:8000/pulp/api/v3/repositories/ name=foo

export REPO_HREF=$(http :8000/pulp/api/v3/repositories/ | jq -r '.results[] | select(.name == "foo") | ._href')

wget https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-non-utf-8/rpm-with-non-utf-8-1-1.fc25.noarch.rpm

http --form POST http://localhost:8000/pulp/api/v3/artifacts/ file@./rpm-with-non-utf-8-1-1.fc25.noarch.rpm

http POST http://localhost:8000/pulp/api/v3/content/rpm/packages/ relative_path=rpm-with-non-utf-8-1-1.fc25.noarch.rpm artifact="/pulp/api/v3/artifacts/1/" filename=rpm-with-non-utf8.rpm

Traceback

HTTP/1.1 500 Internal Server Error
Content-Length: 20215
Content-Type: text/plain; charset=utf-8
Date: Thu, 29 Nov 2018 15:41:48 GMT
Server: WSGIServer/0.2 CPython/3.7.1
Vary: Cookie
X-Frame-Options: SAMEORIGIN

UnicodeDecodeError at /pulp/api/v3/content/rpm/packages/
'utf-8' codec can't decode byte 0x80 in position 168: invalid start byte

Request Method: POST
Request URL: http://localhost:8000/pulp/api/v3/content/rpm/packages/
Django Version: 2.1.3
Python Executable: /home/vagrant/.virtualenvs/pulp/bin/python3
Python Version: 3.7.1
Python Path: ['/home/vagrant/.virtualenvs/pulp/bin', '/usr/lib64/python37.zip', '/usr/lib64/python3.7', '/usr/lib64/python3.7/lib-dynload', '/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages', '/home/vagrant/devel/pulp/pulpcore', '/home/vagrant/devel/pulp/plugin', '/home/vagrant/pulp_rpm', '/home/vagrant/.virtualenvs/pulp/lib/python3.7/site-packages']
Server time: Thu, 29 Nov 2018 15:41:48 +0000
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django_filters',
 'drf_yasg',
 'rest_framework',
 'pulpcore.app',
 'pulp_file.app.PulpFilePluginAppConfig',
 'pulp_rpm.app.PulpRpmPluginAppConfig',
 'crispy_forms',
 'django_extensions']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']

Traceback:

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/core/handlers/exception.py" in inner
  34.             response = get_response(request)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/core/handlers/base.py" in _get_response
  126.                 response = self.process_exception_by_middleware(e, request)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/core/handlers/base.py" in _get_response
  124.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
  54.         return view_func(*args, **kwargs)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/viewsets.py" in view
  116.             return self.dispatch(request, *args, **kwargs)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/views.py" in dispatch
  495.             response = self.handle_exception(exc)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/views.py" in handle_exception
  455.             self.raise_uncaught_exception(exc)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/views.py" in dispatch
  492.             response = handler(request, *args, **kwargs)

File "/usr/lib64/python3.7/contextlib.py" in inner
  74.                 return func(*args, **kwds)

File "/home/vagrant/pulp_rpm/pulp_rpm/app/viewsets.py" in create
  96.             package = Package.createrepo_to_dict(cr_pkginfo)

File "/home/vagrant/pulp_rpm/pulp_rpm/app/models.py" in createrepo_to_dict
  243.             'description': getattr(package, CREATEREPO_PACKAGE_ATTRS.DESCRIPTION) or '',

Exception Type: UnicodeDecodeError at /pulp/api/v3/content/rpm/packages/
Exception Value: 'utf-8' codec can't decode byte 0x80 in position 168: invalid start byte
Request information:
USER: admin

GET: No GET data

POST: No POST data

FILES: No FILES data

COOKIES: No cookie data

META:
CONTENT_LENGTH = '137'
CONTENT_TYPE = 'application/json'
[vagrant@p3 ~]$ cat /etc/redhat-release 
Fedora release 29 (Twenty Nine)

Related issues

Related to RPM Support - Test #4210: Test upload of RPM with invalid byte sequences embedded in their headersCLOSED - COMPLETE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 2b13963a View on GitHub
Added by kersom over 1 year ago

Verify HTTP exception when uploading nonutf8 rpm

Add test to assert that HTTP exception is raised when uploading a non-utf8 RPM.

https://pulp.plan.io/issues/4210 closes: #4210

https://pulp.plan.io/issues/4215 ref: #4215

History

#1 Updated by kersom almost 2 years ago

  • Description updated (diff)

#2 Updated by kersom almost 2 years ago

  • Subject changed from Pulp does not support upload of non ascii RPM to Pulp does not support upload of non utf8 RPM
  • Description updated (diff)

#3 Updated by amacdona@redhat.com almost 2 years ago

  • Project changed from Pulp to RPM Support

#4 Updated by kersom almost 2 years ago

  • Related to Test #4210: Test upload of RPM with invalid byte sequences embedded in their headers added

#5 Updated by bmbouter almost 2 years ago

The Pulp2 behavior for a RPM to Pulp2 with non utf-8 chars in certain fields would have those invalid chars removed. That's good for being able to sync repositories with invalidly formed RPMs. It's not good because Pulp is mutating the data in unexpected ways. For example now the checksum of that field won't validate correctly after passing through Pulp which is a problem. The real fix is for those RPMs to have their metadata correctly utf-8 encoded.

Note that per the spec in section 2.3.1, RPMs expect utf-8 encoding for various fields, so Pulp3 is correct in rejecting an encoding it can't interpret. Since we aren't getting active bug reports from users, I think we should close this for now. I'm leaving open to go back through triage.

Other ideas and thoughts/concerns are welcome.

#6 Updated by daviddavis almost 2 years ago

  • Subject changed from Pulp does not support upload of non utf8 RPM to When uploading an invalid rpm such as a non utf8 RPM, users should get a friendly error message
  • Description updated (diff)
  • Triaged changed from No to Yes

Updating issue to better handle invalid RPMs.

#7 Updated by bmbouter over 1 year ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF