Project

Profile

Help

Issue #8610

PulpImporter assumes tempfiles can always go to /tmp

Added by ggainey 15 days ago. Updated 7 days ago.

Status:
NEW
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 96
Quarter:

Description

importer.pulp_import uses tempfile.TemporaryDirectory() in places like this:

https://github.com/pulp/pulpcore/blob/master/pulpcore/app/tasks/importer.py#L118

If your /tmp is small, and your export is Large, this can cause Bad Things to happen.

We should perhas set dir= to the workers work-directory?

History

#1 Updated by ggainey 15 days ago

And by "Bad Things", I mean the following stacktrace:

Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: pulp [ff6805b69c554c78ac05ef53f096ea3d]: rq.worker:ERROR: Traceback (most recent call last):
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/worker.py", line 1008, in perform_job
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     rv = job.perform()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 706, in perform
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self._result = self._execute()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 729, in _execute
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     result = self.func(*self.args, **self.kwargs)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/home/vagrant/devel/pulpcore/pulpcore/app/tasks/importer.py", line 387, in pulp_import
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     tar.extractall(path=temp_dir)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2036, in extractall
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self.extract(tarinfo, path, set_attrs=not tarinfo.isdir(),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2077, in extract
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2150, in _extract_member
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self.makefile(tarinfo, targetpath)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2199, in makefile
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 250, in copyfileobj
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     dst.write(buf)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: OSError: [Errno 28] No space left on device
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: Traceback (most recent call last):
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/worker.py", line 1008, in perform_job
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     rv = job.perform()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 706, in perform
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self._result = self._execute()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 729, in _execute
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     result = self.func(*self.args, **self.kwargs)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/home/vagrant/devel/pulpcore/pulpcore/app/tasks/importer.py", line 387, in pulp_import
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     tar.extractall(path=temp_dir)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2036, in extractall
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self.extract(tarinfo, path, set_attrs=not tarinfo.isdir(),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2077, in extract
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2150, in _extract_member
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self.makefile(tarinfo, targetpath)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2199, in makefile
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 250, in copyfileobj
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     dst.write(buf)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: OSError: [Errno 28] No space left on device

(Note: trying to import a ~17GB export file, on a system with a 2.5GB /tmp, is...suboptimal.)

#2 Updated by fao89 14 days ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 95

#3 Updated by rchan 7 days ago

  • Sprint changed from Sprint 95 to Sprint 96

Please register to edit this issue

Also available in: Atom PDF