Project

Profile

Help

Issue #8610

PulpImporter assumes tempfiles can always go to /tmp

Added by ggainey 5 months ago. Updated 7 days ago.

Status:
NEW
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 105
Quarter:

Description

importer.pulp_import uses tempfile.TemporaryDirectory() in places like this:

https://github.com/pulp/pulpcore/blob/master/pulpcore/app/tasks/importer.py#L118

If your /tmp is small, and your export is Large, this can cause Bad Things to happen.

We should perhas set dir= to the workers work-directory?

History

#1 Updated by ggainey 5 months ago

And by "Bad Things", I mean the following stacktrace:

Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: pulp [ff6805b69c554c78ac05ef53f096ea3d]: rq.worker:ERROR: Traceback (most recent call last):
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/worker.py", line 1008, in perform_job
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     rv = job.perform()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 706, in perform
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self._result = self._execute()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 729, in _execute
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     result = self.func(*self.args, **self.kwargs)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/home/vagrant/devel/pulpcore/pulpcore/app/tasks/importer.py", line 387, in pulp_import
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     tar.extractall(path=temp_dir)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2036, in extractall
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self.extract(tarinfo, path, set_attrs=not tarinfo.isdir(),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2077, in extract
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2150, in _extract_member
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self.makefile(tarinfo, targetpath)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2199, in makefile
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 250, in copyfileobj
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     dst.write(buf)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: OSError: [Errno 28] No space left on device
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: Traceback (most recent call last):
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/worker.py", line 1008, in perform_job
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     rv = job.perform()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 706, in perform
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self._result = self._execute()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 729, in _execute
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     result = self.func(*self.args, **self.kwargs)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/home/vagrant/devel/pulpcore/pulpcore/app/tasks/importer.py", line 387, in pulp_import
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     tar.extractall(path=temp_dir)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2036, in extractall
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self.extract(tarinfo, path, set_attrs=not tarinfo.isdir(),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2077, in extract
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2150, in _extract_member
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self.makefile(tarinfo, targetpath)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2199, in makefile
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 250, in copyfileobj
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     dst.write(buf)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: OSError: [Errno 28] No space left on device

(Note: trying to import a ~17GB export file, on a system with a 2.5GB /tmp, is...suboptimal.)

#2 Updated by fao89 5 months ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 95

#3 Updated by rchan 5 months ago

  • Sprint changed from Sprint 95 to Sprint 96

#4 Updated by rchan 4 months ago

  • Sprint changed from Sprint 96 to Sprint 97

#5 Updated by rchan 4 months ago

  • Sprint changed from Sprint 97 to Sprint 98

#6 Updated by rchan 3 months ago

  • Sprint changed from Sprint 98 to Sprint 99

#7 Updated by rchan 3 months ago

  • Sprint changed from Sprint 99 to Sprint 100

#8 Updated by rchan 2 months ago

  • Sprint changed from Sprint 100 to Sprint 101

#9 Updated by ipanova@redhat.com about 2 months ago

  • Sprint changed from Sprint 101 to Sprint 102

#10 Updated by rchan about 1 month ago

  • Sprint changed from Sprint 102 to Sprint 103

#11 Updated by rchan 21 days ago

  • Sprint changed from Sprint 103 to Sprint 104

#12 Updated by rchan 7 days ago

  • Sprint changed from Sprint 104 to Sprint 105

Please register to edit this issue

Also available in: Atom PDF