Project

Profile

Help

Issue #8628

Need better error message on invalid chunk-size for export

Added by paji@redhat.com 5 months ago. Updated 4 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 97
Quarter:

Description

The chunking parameter in the export call is unable to handle super large files. There is a possibility that the user may accidentally give bytes for megabytes. We need pulp to fail more gracefully for unreasonable values.

Try to export stuff in chunks in terms of unreasonably large number like 10000000000000M. Looks like the split command used by pulp internally is going to fail along

 
split: 104857600000000000000: invalid number of bytes
 Try 'split --help' for more information.
Apr 26 11:38:57 pulpcore-worker-4[1148]: pulp [2880fe6a-6428-407f-bd20-c5215be001e3]: rq.worker:INFO: 1148@dhcp-2-120.vms.sat.rdu2.redhat.com: Job OK (81ac65b1-d349-481d-88db-6df45499e1f3)
Apr 26 11:38:57 pulpcore-worker-4[1148]: pulp [None]: rq.worker:INFO: 1148@dhcp-2-120.vms.sat.rdu2.redhat.com: 6003a481-77bf-4f4c-b045-3d234e56872c
Apr 26 11:38:57 pulpcore-worker-4[1148]: pulp [None]: rq.worker:INFO: 1148@dhcp-2-120.vms.sat.rdu2.redhat.com: Job OK (6003a481-77bf-4f4c-b045-3d234e56872c)
 pulp [2880fe6a-6428-407f-bd20-c5215be001e3]: rq.worker:ERROR: Traceback (most recent call last):
 File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 193, in pulp_export
 _do_export(pulp_exporter, tar, the_export)
 File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 299, in _do_export
 export_artifacts(the_export, artifacts)
 File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 88, in export_artifacts
 export.tarfile.add(artifact.file.path, dest)
 File "/usr/lib64/python3.6/tarfile.py", line 1954, in add
 self.addfile(tarinfo, f)
 File "/usr/lib64/python3.6/tarfile.py", line 1982, in addfile
 copyfileobj(fileobj, self.fileobj, tarinfo.size, bufsize=bufsize)
 File "/usr/lib64/python3.6/tarfile.py", line 258, in copyfileobj
 dst.write(buf)
 File "/usr/lib64/python3.6/tarfile.py", line 443, in write
 self.__write(s)
 File "/usr/lib64/python3.6/tarfile.py", line 451, in __write
 self.fileobj.write(self.buf[:self.bufsize])
 BrokenPipeError: [Errno 32] Broken pipe
 During handling of the above exception, another exception occurred:
 Traceback (most recent call last):
 File "/usr/lib/python3.6/site-packages/rq/worker.py", line 975, in perform_job
 rv = job.perform()
 File "/usr/lib/python3.6/site-packages/rq/job.py", line 696, in perform
 self._result = self._execute()
 File "/usr/lib/python3.6/site-packages/rq/job.py", line 719, in _execute
 return self.func(*self.args, **self.kwargs)
 File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 193, in pulp_export
 _do_export(pulp_exporter, tar, the_export)
 File "/usr/lib64/python3.6/tarfile.py", line 2441, in __exit__
 self.fileobj.close()
 File "/usr/lib64/python3.6/tarfile.py", line 467, in close
 self.fileobj.write(self.buf)
 BrokenPipeError: [Errno 32] Broken pipe
 Traceback (most recent call last):
 File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 193, in pulp_export
 _do_export(pulp_exporter, tar, the_export)
 File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 299, in _do_export
 export_artifacts(the_export, artifacts)
 File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 88, in export_artifacts
 export.tarfile.add(artifact.file.path, dest)
 File "/usr/lib64/python3.6/tarfile.py", line 1954, in add
 self.addfile(tarinfo, f)
 File "/usr/lib64/python3.6/tarfile.py", line 1982, in addfile
 copyfileobj(fileobj, self.fileobj, tarinfo.size, bufsize=bufsize)
 File "/usr/lib64/python3.6/tarfile.py", line 258, in copyfileobj
 dst.write(buf)
 File "/usr/lib64/python3.6/tarfile.py", line 443, in write
 self.__write(s)
 File "/usr/lib64/python3.6/tarfile.py", line 451, in __write
 self.fileobj.write(self.buf[:self.bufsize])
 BrokenPipeError: [Errno 32] Broken pipe

I noticed the same when I ran the command directly

# split -b 10000000000000M foo.txt 
split: 10000000000000M: invalid number of bytes: File too large

Looks like split command has a maximum

.
$ split -b 1000000000000M foo.txt 
$ split -b 10000000000000M foo.txt 
split: invalid number of bytes: ‘10000000000000M’: Value too large for defined data type

pulp needs to error better and reflect it. Instead we get an error along " description: "[Errno 32] Broken pipe"

Associated revisions

Revision 78f7609f View on GitHub
Added by ggainey 4 months ago

Taught PulpExportSerializer to validate chunk-size-max.

fixes #8628

History

#2 Updated by fao89 5 months ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 95

#3 Updated by rchan 5 months ago

  • Sprint changed from Sprint 95 to Sprint 96

#4 Updated by rchan 4 months ago

  • Sprint changed from Sprint 96 to Sprint 97

#5 Updated by pulpbot 4 months ago

  • Status changed from NEW to POST

#6 Updated by dalley 4 months ago

  • Assignee set to ggainey

#7 Updated by ggainey 4 months ago

  • Status changed from POST to MODIFIED

#8 Updated by dalley 4 months ago

  • Sprint/Milestone set to 3.13.0

#9 Updated by pulpbot 4 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF