Project

Profile

Help

Issue #8610

closed

PulpImporter assumes tempfiles can always go to /tmp

Added by ggainey over 1 year ago. Updated 10 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 111
Quarter:

Description

importer.pulp_import uses tempfile.TemporaryDirectory() in places like this:

https://github.com/pulp/pulpcore/blob/master/pulpcore/app/tasks/importer.py#L118

If your /tmp is small, and your export is Large, this can cause Bad Things to happen.

We should perhas set dir= to the workers work-directory?


Related issues

Related to Pulp - Task #9569: Eliminate all instances when Pulp writes to /tmp and always use the worker working directoryCLOSED - DUPLICATE

Actions
Actions #1

Updated by ggainey over 1 year ago

And by "Bad Things", I mean the following stacktrace:

Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: pulp [ff6805b69c554c78ac05ef53f096ea3d]: rq.worker:ERROR: Traceback (most recent call last):
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/worker.py", line 1008, in perform_job
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     rv = job.perform()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 706, in perform
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self._result = self._execute()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 729, in _execute
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     result = self.func(*self.args, **self.kwargs)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/home/vagrant/devel/pulpcore/pulpcore/app/tasks/importer.py", line 387, in pulp_import
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     tar.extractall(path=temp_dir)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2036, in extractall
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self.extract(tarinfo, path, set_attrs=not tarinfo.isdir(),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2077, in extract
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2150, in _extract_member
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self.makefile(tarinfo, targetpath)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2199, in makefile
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 250, in copyfileobj
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     dst.write(buf)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: OSError: [Errno 28] No space left on device
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: Traceback (most recent call last):
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/worker.py", line 1008, in perform_job
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     rv = job.perform()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 706, in perform
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self._result = self._execute()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 729, in _execute
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     result = self.func(*self.args, **self.kwargs)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/home/vagrant/devel/pulpcore/pulpcore/app/tasks/importer.py", line 387, in pulp_import
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     tar.extractall(path=temp_dir)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2036, in extractall
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self.extract(tarinfo, path, set_attrs=not tarinfo.isdir(),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2077, in extract
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2150, in _extract_member
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self.makefile(tarinfo, targetpath)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2199, in makefile
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 250, in copyfileobj
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     dst.write(buf)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: OSError: [Errno 28] No space left on device

(Note: trying to import a ~17GB export file, on a system with a 2.5GB /tmp, is...suboptimal.)

Actions #2

Updated by fao89 over 1 year ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 95
Actions #3

Updated by rchan over 1 year ago

  • Sprint changed from Sprint 95 to Sprint 96
Actions #4

Updated by rchan over 1 year ago

  • Sprint changed from Sprint 96 to Sprint 97
Actions #5

Updated by rchan over 1 year ago

  • Sprint changed from Sprint 97 to Sprint 98
Actions #6

Updated by rchan over 1 year ago

  • Sprint changed from Sprint 98 to Sprint 99
Actions #7

Updated by rchan about 1 year ago

  • Sprint changed from Sprint 99 to Sprint 100
Actions #8

Updated by rchan about 1 year ago

  • Sprint changed from Sprint 100 to Sprint 101
Actions #9

Updated by ipanova@redhat.com about 1 year ago

  • Sprint changed from Sprint 101 to Sprint 102
Actions #10

Updated by rchan about 1 year ago

  • Sprint changed from Sprint 102 to Sprint 103
Actions #11

Updated by rchan about 1 year ago

  • Sprint changed from Sprint 103 to Sprint 104
Actions #12

Updated by rchan about 1 year ago

  • Sprint changed from Sprint 104 to Sprint 105
Actions #13

Updated by rchan about 1 year ago

  • Sprint changed from Sprint 105 to Sprint 106
Actions #14

Updated by rchan 12 months ago

  • Sprint changed from Sprint 106 to Sprint 107
Actions #15

Updated by rchan 11 months ago

  • Sprint changed from Sprint 107 to Sprint 108
Actions #16

Updated by rchan 11 months ago

  • Sprint changed from Sprint 108 to Sprint 109
Actions #17

Updated by ttereshc 11 months ago

  • Related to Task #9569: Eliminate all instances when Pulp writes to /tmp and always use the worker working directory added
Actions #18

Updated by rchan 10 months ago

  • Sprint changed from Sprint 109 to Sprint 110
Actions #19

Updated by rchan 10 months ago

  • Sprint changed from Sprint 110 to Sprint 111
Actions #20

Updated by ggainey 10 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ggainey
Actions #21

Updated by pulpbot 10 months ago

  • Status changed from ASSIGNED to POST

Added by ggainey 10 months ago

Revision 2f565c93

Taught import-task to extract into current-working-dir.

fixes #8610 [nocoverage]

Actions #22

Updated by ggainey 10 months ago

  • Status changed from POST to MODIFIED
Actions #23

Updated by pulpbot 10 months ago

  • Sprint/Milestone set to 3.17.0
Actions #24

Updated by pulpbot 10 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF