Project

Profile

Help

Issue #8595

closed

pulp-python publication faling

Added by pmishra over 3 years ago. Updated over 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Easy Fix
Sprint:
Quarter:

Description

Hi,

I'm running pulp in a Kubernetes cluster and sometimes python publications are failing with the error '[Errno 39] Directory not empty' , after some debugging the error leads to https://github.com/pulp/pulpcore/blob/master/pulpcore/tasking/storage.py#L60

pulp-cli:

# pulp python publication create --repository foo --version 4
..................Error: Task /pulp/api/v3/tasks/11b036e7-3799-4806-a29b-9a784b6f4857/ failed: '[Errno 39] Directory not empty: 'wheel''

worker log:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/rq/worker.py", line 1008, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.9/site-packages/rq/job.py", line 706, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.9/site-packages/rq/job.py", line 729, in _execute
    result = self.func(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/tasks/publish.py", line 65, in publish
    write_simple_api(publication)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/tasking/storage.py", line 98, in __exit__
    os.chdir(self._prev_dir)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/tasking/storage.py", line 55, in delete
    self._delete()
  File "/usr/local/lib/python3.9/site-packages/pulpcore/tasking/storage.py", line 65, in _delete
    #shutil.rmtree(self.path, ignore_errors=True)
  File "/usr/local/lib/python3.9/shutil.py", line 718, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/local/lib/python3.9/shutil.py", line 655, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/local/lib/python3.9/shutil.py", line 659, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/usr/local/lib/python3.9/shutil.py", line 657, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'wheel'

One quick fix maybe ignore 'OSError' or pass 'shutil.rmtree(self.path, ignore_errors=True)' ref: https://docs.python.org/3/library/shutil.html#shutil.rmtree

Actions #1

Updated by pmishra over 3 years ago

# ls -laR 6@pulp-d7cb84bff-bxtlq/
6@pulp-d7cb84bff-bxtlq/:
total 0
drwx------ 3 nobody nobody 50 Apr 20 01:45 .
drwxrwxrwx 7 root   root   98 Apr 20 01:43 ..
drwx------ 3 nobody nobody 30 Apr 20 01:45 32f29a54-4ce4-4684-a192-127e06321051

6@pulp-d7cb84bff-bxtlq/32f29a54-4ce4-4684-a192-127e06321051:
total 0
drwx------ 3 nobody nobody 30 Apr 20 01:45 .
drwx------ 3 nobody nobody 50 Apr 20 01:45 ..
drwx------ 3 nobody nobody 20 Apr 20 01:45 workdir_hlo8pjqn

6@pulp-d7cb84bff-bxtlq/32f29a54-4ce4-4684-a192-127e06321051/workdir_hlo8pjqn:
total 0
drwx------ 3 nobody nobody 20 Apr 20 01:45 .
drwx------ 3 nobody nobody 30 Apr 20 01:45 ..
drwxr-xr-x 3 nobody nobody 19 Apr 20 01:45 simple

6@pulp-d7cb84bff-bxtlq/32f29a54-4ce4-4684-a192-127e06321051/workdir_hlo8pjqn/simple:
total 0
drwxr-xr-x 3 nobody nobody 19 Apr 20 01:45 .
drwx------ 3 nobody nobody 20 Apr 20 01:45 ..
drwxr-xr-x 2 nobody nobody  6 Apr 20 01:45 wheel

6@pulp-d7cb84bff-bxtlq/32f29a54-4ce4-4684-a192-127e06321051/workdir_hlo8pjqn/simple/wheel:
total 0
drwxr-xr-x 2 nobody nobody  6 Apr 20 01:45 .
drwxr-xr-x 3 nobody nobody 19 Apr 20 01:45 ..
Actions #2

Updated by daviddavis over 3 years ago

  • Triaged changed from No to Yes
Actions #3

Updated by pulpbot over 3 years ago

  • Status changed from NEW to POST

Added by pmishra over 3 years ago

Revision dd749995 | View on GitHub

fix directory not empty error

fix issue with shutil.rmtree sometime failing with error [Errno 39]Directory not empty, added new exception block to cath OSError and retry the rmtree with 'ignore_errors' set to True https://docs.python.org/3/library/shutil.html#shutil.rmtree

closes #8595

Update CHANGES/8595.bugfix

Co-authored-by: David Davis

Actions #4

Updated by pmishra over 3 years ago

  • Status changed from POST to MODIFIED
Actions #5

Updated by dalley over 3 years ago

  • Sprint/Milestone set to 3.13.0
Actions #6

Updated by pulpbot over 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF