Project

Profile

Help

Issue #4215

closed

When uploading an invalid rpm such as a non utf8 RPM, users should get a friendly error message

Added by kersom over 5 years ago. Updated over 2 years ago.

Status:
CLOSED - DUPLICATE
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Ticket moved to GitHub: "pulp/pulp_rpm/2211":https://github.com/pulp/pulp_rpm/issues/2211


When users upload an invalid RPM such as one that has non-utf8 characters in their headers, they should get a 4xx response and a user friendly error message. Currently they receive a 500.

Steps to reproduce.

http POST http://localhost:8000/pulp/api/v3/repositories/ name=foo

export REPO_HREF=$(http :8000/pulp/api/v3/repositories/ | jq -r '.results[] | select(.name == "foo") | ._href')

wget https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-non-utf-8/rpm-with-non-utf-8-1-1.fc25.noarch.rpm

http --form POST http://localhost:8000/pulp/api/v3/artifacts/ file@./rpm-with-non-utf-8-1-1.fc25.noarch.rpm

http POST http://localhost:8000/pulp/api/v3/content/rpm/packages/ relative_path=rpm-with-non-utf-8-1-1.fc25.noarch.rpm artifact="/pulp/api/v3/artifacts/1/" filename=rpm-with-non-utf8.rpm

Traceback

HTTP/1.1 500 Internal Server Error
Content-Length: 20215
Content-Type: text/plain; charset=utf-8
Date: Thu, 29 Nov 2018 15:41:48 GMT
Server: WSGIServer/0.2 CPython/3.7.1
Vary: Cookie
X-Frame-Options: SAMEORIGIN

UnicodeDecodeError at /pulp/api/v3/content/rpm/packages/
'utf-8' codec can't decode byte 0x80 in position 168: invalid start byte

Request Method: POST
Request URL: http://localhost:8000/pulp/api/v3/content/rpm/packages/
Django Version: 2.1.3
Python Executable: /home/vagrant/.virtualenvs/pulp/bin/python3
Python Version: 3.7.1
Python Path: ['/home/vagrant/.virtualenvs/pulp/bin', '/usr/lib64/python37.zip', '/usr/lib64/python3.7', '/usr/lib64/python3.7/lib-dynload', '/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages', '/home/vagrant/devel/pulp/pulpcore', '/home/vagrant/devel/pulp/plugin', '/home/vagrant/pulp_rpm', '/home/vagrant/.virtualenvs/pulp/lib/python3.7/site-packages']
Server time: Thu, 29 Nov 2018 15:41:48 +0000
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django_filters',
 'drf_yasg',
 'rest_framework',
 'pulpcore.app',
 'pulp_file.app.PulpFilePluginAppConfig',
 'pulp_rpm.app.PulpRpmPluginAppConfig',
 'crispy_forms',
 'django_extensions']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']

Traceback:

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/core/handlers/exception.py" in inner
  34.             response = get_response(request)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/core/handlers/base.py" in _get_response
  126.                 response = self.process_exception_by_middleware(e, request)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/core/handlers/base.py" in _get_response
  124.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
  54.         return view_func(*args, **kwargs)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/viewsets.py" in view
  116.             return self.dispatch(request, *args, **kwargs)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/views.py" in dispatch
  495.             response = self.handle_exception(exc)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/views.py" in handle_exception
  455.             self.raise_uncaught_exception(exc)

File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/views.py" in dispatch
  492.             response = handler(request, *args, **kwargs)

File "/usr/lib64/python3.7/contextlib.py" in inner
  74.                 return func(*args, **kwds)

File "/home/vagrant/pulp_rpm/pulp_rpm/app/viewsets.py" in create
  96.             package = Package.createrepo_to_dict(cr_pkginfo)

File "/home/vagrant/pulp_rpm/pulp_rpm/app/models.py" in createrepo_to_dict
  243.             'description': getattr(package, CREATEREPO_PACKAGE_ATTRS.DESCRIPTION) or '',

Exception Type: UnicodeDecodeError at /pulp/api/v3/content/rpm/packages/
Exception Value: 'utf-8' codec can't decode byte 0x80 in position 168: invalid start byte
Request information:
USER: admin

GET: No GET data

POST: No POST data

FILES: No FILES data

COOKIES: No cookie data

META:
CONTENT_LENGTH = '137'
CONTENT_TYPE = 'application/json'
[vagrant@p3 ~]$ cat /etc/redhat-release 
Fedora release 29 (Twenty Nine)

Related issues

Related to RPM Support - Test #4210: Test upload of RPM with invalid byte sequences embedded in their headersCLOSED - COMPLETEActions
Actions #1

Updated by kersom over 5 years ago

  • Description updated (diff)
Actions #2

Updated by kersom over 5 years ago

  • Subject changed from Pulp does not support upload of non ascii RPM to Pulp does not support upload of non utf8 RPM
  • Description updated (diff)
Actions #3

Updated by amacdona@redhat.com over 5 years ago

  • Project changed from Pulp to RPM Support
Actions #4

Updated by kersom over 5 years ago

  • Related to Test #4210: Test upload of RPM with invalid byte sequences embedded in their headers added
Actions #5

Updated by bmbouter over 5 years ago

The Pulp2 behavior for a RPM to Pulp2 with non utf-8 chars in certain fields would have those invalid chars removed. That's good for being able to sync repositories with invalidly formed RPMs. It's not good because Pulp is mutating the data in unexpected ways. For example now the checksum of that field won't validate correctly after passing through Pulp which is a problem. The real fix is for those RPMs to have their metadata correctly utf-8 encoded.

Note that per the spec in section 2.3.1, RPMs expect utf-8 encoding for various fields, so Pulp3 is correct in rejecting an encoding it can't interpret. Since we aren't getting active bug reports from users, I think we should close this for now. I'm leaving open to go back through triage.

Other ideas and thoughts/concerns are welcome.

Actions #6

Updated by daviddavis over 5 years ago

  • Subject changed from Pulp does not support upload of non utf8 RPM to When uploading an invalid rpm such as a non utf8 RPM, users should get a friendly error message
  • Description updated (diff)
  • Triaged changed from No to Yes

Updating issue to better handle invalid RPMs.

Added by kersom about 5 years ago

Revision 2b13963a | View on GitHub

Verify HTTP exception when uploading nonutf8 rpm

Add test to assert that HTTP exception is raised when uploading a non-utf8 RPM.

https://pulp.plan.io/issues/4210 closes: #4210

https://pulp.plan.io/issues/4215 ref: #4215

Actions #7

Updated by bmbouter almost 5 years ago

  • Tags deleted (Pulp 3)
Actions #8

Updated by pulpbot over 2 years ago

  • Description updated (diff)
  • Status changed from NEW to CLOSED - DUPLICATE

Also available in: Atom PDF