Project

Profile

Help

Issue #8628

closed

Need better error message on invalid chunk-size for export

Added by paji@redhat.com almost 3 years ago. Updated almost 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 97
Quarter:

Description

The chunking parameter in the export call is unable to handle super large files. There is a possibility that the user may accidentally give bytes for megabytes. We need pulp to fail more gracefully for unreasonable values.

Try to export stuff in chunks in terms of unreasonably large number like 10000000000000M. Looks like the split command used by pulp internally is going to fail along

 
split: 104857600000000000000: invalid number of bytes
 Try 'split --help' for more information.
Apr 26 11:38:57 pulpcore-worker-4[1148]: pulp [2880fe6a-6428-407f-bd20-c5215be001e3]: rq.worker:INFO: 1148@dhcp-2-120.vms.sat.rdu2.redhat.com: Job OK (81ac65b1-d349-481d-88db-6df45499e1f3)
Apr 26 11:38:57 pulpcore-worker-4[1148]: pulp [None]: rq.worker:INFO: 1148@dhcp-2-120.vms.sat.rdu2.redhat.com: 6003a481-77bf-4f4c-b045-3d234e56872c
Apr 26 11:38:57 pulpcore-worker-4[1148]: pulp [None]: rq.worker:INFO: 1148@dhcp-2-120.vms.sat.rdu2.redhat.com: Job OK (6003a481-77bf-4f4c-b045-3d234e56872c)
 pulp [2880fe6a-6428-407f-bd20-c5215be001e3]: rq.worker:ERROR: Traceback (most recent call last):
 File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 193, in pulp_export
 _do_export(pulp_exporter, tar, the_export)
 File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 299, in _do_export
 export_artifacts(the_export, artifacts)
 File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 88, in export_artifacts
 export.tarfile.add(artifact.file.path, dest)
 File "/usr/lib64/python3.6/tarfile.py", line 1954, in add
 self.addfile(tarinfo, f)
 File "/usr/lib64/python3.6/tarfile.py", line 1982, in addfile
 copyfileobj(fileobj, self.fileobj, tarinfo.size, bufsize=bufsize)
 File "/usr/lib64/python3.6/tarfile.py", line 258, in copyfileobj
 dst.write(buf)
 File "/usr/lib64/python3.6/tarfile.py", line 443, in write
 self.__write(s)
 File "/usr/lib64/python3.6/tarfile.py", line 451, in __write
 self.fileobj.write(self.buf[:self.bufsize])
 BrokenPipeError: [Errno 32] Broken pipe
 During handling of the above exception, another exception occurred:
 Traceback (most recent call last):
 File "/usr/lib/python3.6/site-packages/rq/worker.py", line 975, in perform_job
 rv = job.perform()
 File "/usr/lib/python3.6/site-packages/rq/job.py", line 696, in perform
 self._result = self._execute()
 File "/usr/lib/python3.6/site-packages/rq/job.py", line 719, in _execute
 return self.func(*self.args, **self.kwargs)
 File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 193, in pulp_export
 _do_export(pulp_exporter, tar, the_export)
 File "/usr/lib64/python3.6/tarfile.py", line 2441, in __exit__
 self.fileobj.close()
 File "/usr/lib64/python3.6/tarfile.py", line 467, in close
 self.fileobj.write(self.buf)
 BrokenPipeError: [Errno 32] Broken pipe
 Traceback (most recent call last):
 File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 193, in pulp_export
 _do_export(pulp_exporter, tar, the_export)
 File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 299, in _do_export
 export_artifacts(the_export, artifacts)
 File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 88, in export_artifacts
 export.tarfile.add(artifact.file.path, dest)
 File "/usr/lib64/python3.6/tarfile.py", line 1954, in add
 self.addfile(tarinfo, f)
 File "/usr/lib64/python3.6/tarfile.py", line 1982, in addfile
 copyfileobj(fileobj, self.fileobj, tarinfo.size, bufsize=bufsize)
 File "/usr/lib64/python3.6/tarfile.py", line 258, in copyfileobj
 dst.write(buf)
 File "/usr/lib64/python3.6/tarfile.py", line 443, in write
 self.__write(s)
 File "/usr/lib64/python3.6/tarfile.py", line 451, in __write
 self.fileobj.write(self.buf[:self.bufsize])
 BrokenPipeError: [Errno 32] Broken pipe

I noticed the same when I ran the command directly

# split -b 10000000000000M foo.txt 
split: 10000000000000M: invalid number of bytes: File too large

Looks like split command has a maximum

.
$ split -b 1000000000000M foo.txt 
$ split -b 10000000000000M foo.txt 
split: invalid number of bytes: ‘10000000000000M’: Value too large for defined data type

pulp needs to error better and reflect it. Instead we get an error along " description: "[Errno 32] Broken pipe"

Actions #2

Updated by fao89 almost 3 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 95
Actions #3

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 95 to Sprint 96
Actions #4

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 96 to Sprint 97
Actions #5

Updated by pulpbot almost 3 years ago

  • Status changed from NEW to POST
Actions #6

Updated by dalley almost 3 years ago

  • Assignee set to ggainey

Added by ggainey almost 3 years ago

Revision 78f7609f | View on GitHub

Taught PulpExportSerializer to validate chunk-size-max.

fixes #8628

Actions #7

Updated by ggainey almost 3 years ago

  • Status changed from POST to MODIFIED
Actions #8

Updated by dalley almost 3 years ago

  • Sprint/Milestone set to 3.13.0
Actions #9

Updated by pulpbot almost 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF