Pulp

The chunking parameter in the export call is unable to handle super large files. There is a possibility that the user may accidentally give bytes for megabytes. We need pulp to fail more gracefully for unreasonable values.

Try to export stuff in chunks in terms of unreasonably large number like 10000000000000M. Looks like the split command used by pulp internally is going to fail along

 
split: 104857600000000000000: invalid number of bytes
 Try 'split --help' for more information.
Apr 26 11:38:57 pulpcore-worker-4[1148]: pulp [2880fe6a-6428-407f-bd20-c5215be001e3]: rq.worker:INFO: 1148@dhcp-2-120.vms.sat.rdu2.redhat.com: Job OK (81ac65b1-d349-481d-88db-6df45499e1f3)
Apr 26 11:38:57 pulpcore-worker-4[1148]: pulp [None]: rq.worker:INFO: 1148@dhcp-2-120.vms.sat.rdu2.redhat.com: 6003a481-77bf-4f4c-b045-3d234e56872c
Apr 26 11:38:57 pulpcore-worker-4[1148]: pulp [None]: rq.worker:INFO: 1148@dhcp-2-120.vms.sat.rdu2.redhat.com: Job OK (6003a481-77bf-4f4c-b045-3d234e56872c)
 pulp [2880fe6a-6428-407f-bd20-c5215be001e3]: rq.worker:ERROR: Traceback (most recent call last):
 File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 193, in pulp_export
 _do_export(pulp_exporter, tar, the_export)
 File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 299, in _do_export
 export_artifacts(the_export, artifacts)
 File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 88, in export_artifacts
 export.tarfile.add(artifact.file.path, dest)
 File "/usr/lib64/python3.6/tarfile.py", line 1954, in add
 self.addfile(tarinfo, f)
 File "/usr/lib64/python3.6/tarfile.py", line 1982, in addfile
 copyfileobj(fileobj, self.fileobj, tarinfo.size, bufsize=bufsize)
 File "/usr/lib64/python3.6/tarfile.py", line 258, in copyfileobj
 dst.write(buf)
 File "/usr/lib64/python3.6/tarfile.py", line 443, in write
 self.__write(s)
 File "/usr/lib64/python3.6/tarfile.py", line 451, in __write
 self.fileobj.write(self.buf[:self.bufsize])
 BrokenPipeError: [Errno 32] Broken pipe
 During handling of the above exception, another exception occurred:
 Traceback (most recent call last):
 File "/usr/lib/python3.6/site-packages/rq/worker.py", line 975, in perform_job
 rv = job.perform()
 File "/usr/lib/python3.6/site-packages/rq/job.py", line 696, in perform
 self._result = self._execute()
 File "/usr/lib/python3.6/site-packages/rq/job.py", line 719, in _execute
 return self.func(*self.args, **self.kwargs)
 File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 193, in pulp_export
 _do_export(pulp_exporter, tar, the_export)
 File "/usr/lib64/python3.6/tarfile.py", line 2441, in __exit__
 self.fileobj.close()
 File "/usr/lib64/python3.6/tarfile.py", line 467, in close
 self.fileobj.write(self.buf)
 BrokenPipeError: [Errno 32] Broken pipe
 Traceback (most recent call last):
 File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 193, in pulp_export
 _do_export(pulp_exporter, tar, the_export)
 File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 299, in _do_export
 export_artifacts(the_export, artifacts)
 File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 88, in export_artifacts
 export.tarfile.add(artifact.file.path, dest)
 File "/usr/lib64/python3.6/tarfile.py", line 1954, in add
 self.addfile(tarinfo, f)
 File "/usr/lib64/python3.6/tarfile.py", line 1982, in addfile
 copyfileobj(fileobj, self.fileobj, tarinfo.size, bufsize=bufsize)
 File "/usr/lib64/python3.6/tarfile.py", line 258, in copyfileobj
 dst.write(buf)
 File "/usr/lib64/python3.6/tarfile.py", line 443, in write
 self.__write(s)
 File "/usr/lib64/python3.6/tarfile.py", line 451, in __write
 self.fileobj.write(self.buf[:self.bufsize])
 BrokenPipeError: [Errno 32] Broken pipe

I noticed the same when I ran the command directly

# split -b 10000000000000M foo.txt 
split: 10000000000000M: invalid number of bytes: File too large

Looks like split command has a maximum

.
$ split -b 1000000000000M foo.txt 
$ split -b 10000000000000M foo.txt 
split: invalid number of bytes: ‘10000000000000M’: Value too large for defined data type

pulp needs to error better and reflect it. Instead we get an error along " description: "[Errno 32] Broken pipe"