Actions
Issue #8610
closed
PulpImporter assumes tempfiles can always go to /tmp
Added by ggainey over 3 years ago. Updated almost 3 years ago.
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 111
Quarter:
Description
importer.pulp_import uses tempfile.TemporaryDirectory() in places like this:
https://github.com/pulp/pulpcore/blob/master/pulpcore/app/tasks/importer.py#L118
If your /tmp is small, and your export is Large, this can cause Bad Things to happen.
We should perhas set dir= to the workers work-directory?
Related issues
Updated by ggainey over 3 years ago
And by "Bad Things", I mean the following stacktrace:
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: pulp [ff6805b69c554c78ac05ef53f096ea3d]: rq.worker:ERROR: Traceback (most recent call last):
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/worker.py", line 1008, in perform_job
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: rv = job.perform()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 706, in perform
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: self._result = self._execute()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 729, in _execute
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: result = self.func(*self.args, **self.kwargs)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: File "/home/vagrant/devel/pulpcore/pulpcore/app/tasks/importer.py", line 387, in pulp_import
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: tar.extractall(path=temp_dir)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: File "/usr/lib64/python3.9/tarfile.py", line 2036, in extractall
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: self.extract(tarinfo, path, set_attrs=not tarinfo.isdir(),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: File "/usr/lib64/python3.9/tarfile.py", line 2077, in extract
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: File "/usr/lib64/python3.9/tarfile.py", line 2150, in _extract_member
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: self.makefile(tarinfo, targetpath)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: File "/usr/lib64/python3.9/tarfile.py", line 2199, in makefile
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: File "/usr/lib64/python3.9/tarfile.py", line 250, in copyfileobj
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: dst.write(buf)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: OSError: [Errno 28] No space left on device
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: Traceback (most recent call last):
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/worker.py", line 1008, in perform_job
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: rv = job.perform()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 706, in perform
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: self._result = self._execute()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 729, in _execute
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: result = self.func(*self.args, **self.kwargs)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: File "/home/vagrant/devel/pulpcore/pulpcore/app/tasks/importer.py", line 387, in pulp_import
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: tar.extractall(path=temp_dir)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: File "/usr/lib64/python3.9/tarfile.py", line 2036, in extractall
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: self.extract(tarinfo, path, set_attrs=not tarinfo.isdir(),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: File "/usr/lib64/python3.9/tarfile.py", line 2077, in extract
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: File "/usr/lib64/python3.9/tarfile.py", line 2150, in _extract_member
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: self.makefile(tarinfo, targetpath)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: File "/usr/lib64/python3.9/tarfile.py", line 2199, in makefile
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: File "/usr/lib64/python3.9/tarfile.py", line 250, in copyfileobj
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: dst.write(buf)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: OSError: [Errno 28] No space left on device
(Note: trying to import a ~17GB export file, on a system with a 2.5GB /tmp, is...suboptimal.)
Updated by fao89 over 3 years ago
- Triaged changed from No to Yes
- Sprint set to Sprint 95
Updated by ipanova@redhat.com over 3 years ago
- Sprint changed from Sprint 101 to Sprint 102
Updated by rchan about 3 years ago
- Sprint changed from Sprint 103 to Sprint 104
Updated by rchan about 3 years ago
- Sprint changed from Sprint 104 to Sprint 105
Updated by rchan about 3 years ago
- Sprint changed from Sprint 105 to Sprint 106
Updated by rchan about 3 years ago
- Sprint changed from Sprint 106 to Sprint 107
Updated by rchan about 3 years ago
- Sprint changed from Sprint 107 to Sprint 108
Updated by rchan about 3 years ago
- Sprint changed from Sprint 108 to Sprint 109
Updated by ttereshc about 3 years ago
- Related to Task #9569: Eliminate all instances when Pulp writes to /tmp and always use the worker working directory added
Updated by rchan about 3 years ago
- Sprint changed from Sprint 109 to Sprint 110
Updated by rchan almost 3 years ago
- Sprint changed from Sprint 110 to Sprint 111
Updated by ggainey almost 3 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to ggainey
Updated by pulpbot almost 3 years ago
- Status changed from ASSIGNED to POST
Added by ggainey almost 3 years ago
Updated by ggainey almost 3 years ago
- Status changed from POST to MODIFIED
Applied in changeset pulpcore|2f565c936d03f26c61c4b992c5e98a2b835d7cc7.
Updated by pulpbot almost 3 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions
.
Taught import-task to extract into current-working-dir.
fixes #8610 [nocoverage]