Issue #8595: pulp-python publication faling - Pulp

Actions

Send by e-mail Copy link

Issue #8595

closed

pulp-python publication faling

Added by pmishra over 3 years ago. Updated over 3 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

Normal

Assignee:

Category:

Sprint/Milestone:

3.13.0

Start date:

Due date:

Estimated time:

Severity:

2. Medium

Version:

Platform Release:

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Easy Fix

Sprint:

Quarter:

Description

Hi,

I'm running pulp in a Kubernetes cluster and sometimes python publications are failing with the error '[Errno 39] Directory not empty' , after some debugging the error leads to https://github.com/pulp/pulpcore/blob/master/pulpcore/tasking/storage.py#L60

pulp-cli:

# pulp python publication create --repository foo --version 4
..................Error: Task /pulp/api/v3/tasks/11b036e7-3799-4806-a29b-9a784b6f4857/ failed: '[Errno 39] Directory not empty: 'wheel''

worker log:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/rq/worker.py", line 1008, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.9/site-packages/rq/job.py", line 706, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.9/site-packages/rq/job.py", line 729, in _execute
    result = self.func(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/tasks/publish.py", line 65, in publish
    write_simple_api(publication)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/tasking/storage.py", line 98, in __exit__
    os.chdir(self._prev_dir)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/tasking/storage.py", line 55, in delete
    self._delete()
  File "/usr/local/lib/python3.9/site-packages/pulpcore/tasking/storage.py", line 65, in _delete
    #shutil.rmtree(self.path, ignore_errors=True)
  File "/usr/local/lib/python3.9/shutil.py", line 718, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/local/lib/python3.9/shutil.py", line 655, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/local/lib/python3.9/shutil.py", line 659, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/usr/local/lib/python3.9/shutil.py", line 657, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'wheel'

One quick fix maybe ignore 'OSError' or pass 'shutil.rmtree(self.path, ignore_errors=True)' ref: https://docs.python.org/3/library/shutil.html#shutil.rmtree

Actions

Copy link

Updated by pmishra over 3 years ago

# ls -laR 6@pulp-d7cb84bff-bxtlq/
6@pulp-d7cb84bff-bxtlq/:
total 0
drwx------ 3 nobody nobody 50 Apr 20 01:45 .
drwxrwxrwx 7 root   root   98 Apr 20 01:43 ..
drwx------ 3 nobody nobody 30 Apr 20 01:45 32f29a54-4ce4-4684-a192-127e06321051

6@pulp-d7cb84bff-bxtlq/32f29a54-4ce4-4684-a192-127e06321051:
total 0
drwx------ 3 nobody nobody 30 Apr 20 01:45 .
drwx------ 3 nobody nobody 50 Apr 20 01:45 ..
drwx------ 3 nobody nobody 20 Apr 20 01:45 workdir_hlo8pjqn

6@pulp-d7cb84bff-bxtlq/32f29a54-4ce4-4684-a192-127e06321051/workdir_hlo8pjqn:
total 0
drwx------ 3 nobody nobody 20 Apr 20 01:45 .
drwx------ 3 nobody nobody 30 Apr 20 01:45 ..
drwxr-xr-x 3 nobody nobody 19 Apr 20 01:45 simple

6@pulp-d7cb84bff-bxtlq/32f29a54-4ce4-4684-a192-127e06321051/workdir_hlo8pjqn/simple:
total 0
drwxr-xr-x 3 nobody nobody 19 Apr 20 01:45 .
drwx------ 3 nobody nobody 20 Apr 20 01:45 ..
drwxr-xr-x 2 nobody nobody  6 Apr 20 01:45 wheel

6@pulp-d7cb84bff-bxtlq/32f29a54-4ce4-4684-a192-127e06321051/workdir_hlo8pjqn/simple/wheel:
total 0
drwxr-xr-x 2 nobody nobody  6 Apr 20 01:45 .
drwxr-xr-x 3 nobody nobody 19 Apr 20 01:45 ..

Actions

Copy link

Updated by daviddavis over 3 years ago

Triaged changed from No to Yes

Actions

Copy link

Updated by pulpbot over 3 years ago

Status changed from NEW to POST

PR: https://github.com/pulp/pulpcore/pull/1267

Added by pmishra over 3 years ago

Revision dd749995 | View on GitHub

fix directory not empty error

fix issue with shutil.rmtree sometime failing with error [Errno 39]Directory not empty, added new exception block to cath OSError and retry the rmtree with 'ignore_errors' set to True https://docs.python.org/3/library/shutil.html#shutil.rmtree

closes #8595

Update CHANGES/8595.bugfix

Co-authored-by: David Davis daviddavis@users.noreply.github.com

Actions

Copy link