Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #8610

closed

PulpImporter assumes tempfiles can always go to /tmp

Added by ggainey about 3 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
ggainey
Category:
-
Sprint/Milestone:
3.17.0
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 111
Quarter:

Description

importer.pulp_import uses tempfile.TemporaryDirectory() in places like this:

https://github.com/pulp/pulpcore/blob/master/pulpcore/app/tasks/importer.py#L118

If your /tmp is small, and your export is Large, this can cause Bad Things to happen.

We should perhas set dir= to the workers work-directory?

Related issues

Related to Pulp - Task #9569: Eliminate all instances when Pulp writes to /tmp and always use the worker working directoryCLOSED - DUPLICATE

Actions
Actions
Copy link
 #1

Updated by ggainey about 3 years ago

And by "Bad Things", I mean the following stacktrace:

Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: pulp [ff6805b69c554c78ac05ef53f096ea3d]: rq.worker:ERROR: Traceback (most recent call last):
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/worker.py", line 1008, in perform_job
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     rv = job.perform()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 706, in perform
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self._result = self._execute()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 729, in _execute
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     result = self.func(*self.args, **self.kwargs)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/home/vagrant/devel/pulpcore/pulpcore/app/tasks/importer.py", line 387, in pulp_import
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     tar.extractall(path=temp_dir)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2036, in extractall
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self.extract(tarinfo, path, set_attrs=not tarinfo.isdir(),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2077, in extract
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2150, in _extract_member
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self.makefile(tarinfo, targetpath)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2199, in makefile
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 250, in copyfileobj
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     dst.write(buf)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: OSError: [Errno 28] No space left on device
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: Traceback (most recent call last):
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/worker.py", line 1008, in perform_job
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     rv = job.perform()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 706, in perform
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self._result = self._execute()
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/rq/job.py", line 729, in _execute
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     result = self.func(*self.args, **self.kwargs)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/home/vagrant/devel/pulpcore/pulpcore/app/tasks/importer.py", line 387, in pulp_import
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     tar.extractall(path=temp_dir)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2036, in extractall
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self.extract(tarinfo, path, set_attrs=not tarinfo.isdir(),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2077, in extract
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2150, in _extract_member
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     self.makefile(tarinfo, targetpath)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 2199, in makefile
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:   File "/usr/lib64/python3.9/tarfile.py", line 250, in copyfileobj
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]:     dst.write(buf)
Apr 22 01:56:30 pulp3-source-fedora33.padre-fedora.example.com rq[163654]: OSError: [Errno 28] No space left on device

(Note: trying to import a ~17GB export file, on a system with a 2.5GB /tmp, is...suboptimal.)

Actions
Copy link
 #2

Updated by fao89 about 3 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 95
Actions
Copy link
 #3

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 95 to Sprint 96
Actions
Copy link
 #4

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 96 to Sprint 97
Actions
Copy link
 #5

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 97 to Sprint 98
Actions
Copy link
 #6

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 98 to Sprint 99
Actions
Copy link
 #7

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 99 to Sprint 100
Actions
Copy link
 #8

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 100 to Sprint 101
Actions
Copy link
 #9

Updated by ipanova@redhat.com over 2 years ago

  • Sprint changed from Sprint 101 to Sprint 102
Actions
Copy link
 #10

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 102 to Sprint 103
Actions
Copy link
 #11

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 103 to Sprint 104
Actions
Copy link
 #12

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 104 to Sprint 105
Actions
Copy link
 #13

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 105 to Sprint 106
Actions
Copy link
 #14

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 106 to Sprint 107
Actions
Copy link
 #15

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 107 to Sprint 108
Actions
Copy link
 #16

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 108 to Sprint 109
Actions
Copy link
 #17

Updated by ttereshc over 2 years ago

  • Related to Task #9569: Eliminate all instances when Pulp writes to /tmp and always use the worker working directory added
Actions
Copy link
 #18

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 109 to Sprint 110
Actions
Copy link
 #19

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 110 to Sprint 111
Actions
Copy link
 #20

Updated by ggainey over 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ggainey
Actions
Copy link
 #21

Updated by pulpbot over 2 years ago

  • Status changed from ASSIGNED to POST

Added by ggainey over 2 years ago

Revision 2f565c93 | View on GitHub

Taught import-task to extract into current-working-dir.

fixes #8610 [nocoverage]

Actions
Copy link
 #22

Updated by ggainey over 2 years ago

  • Status changed from POST to MODIFIED
Actions
Copy link
 #23

Updated by pulpbot over 2 years ago

  • Sprint/Milestone set to 3.17.0
Actions
Copy link
 #24

Updated by pulpbot over 2 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions
Send by e-mail Copy link

Also available in: Atom PDF