Project

Profile

Help

Issue #7719

'Directory not empty' error in publishing

Added by adam.winberg@smhi.se about 1 year ago. Updated 10 months ago.

Status:
CLOSED - WORKSFORME
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Running a pulp2 migration always results in an error when the migration runs the publishing jobs:

    "error": {
        "description": "[Errno 39] Directory not empty: '/var/lib/pulp/tmp/2287063@lxserv2285.smhi.se/7b022048-0d39-4d6d-a278-91ecb3c4779c'",
        "traceback": "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n    rv = job.perform()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n    self._result = self._execute()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py\", line 142, in migrate_from_pulp2\n    create_repoversions_publications_distributions(plan)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 293, in create_repoversions_publications_distributions\n    task_func(*task_args)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 187, in simple_plugin_migration\n    migrate_repo_distributor(dist_migrator, progress_dist, pulp2_dist)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 391, in migrate_repo_distributor\n    pulp2dist, repo_version)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py\", line 74, in migrate_to_pulp3\n    publish(repo_version.pk, checksum_types=checksum_types)\n  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py\", line 300, in publish\n    metadata_signing_service=metadata_signing_service\n  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py\", line 178, in __exit__\n    self.delete()\n  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py\", line 80, in delete\n    self._delete()\n  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py\", line 90, in _delete\n    shutil.rmtree(self.path)\n  File \"/usr/lib64/python3.6/shutil.py\", line 490, in rmtree\n    onerror(os.rmdir, path, sys.exc_info())\n  File \"/usr/lib64/python3.6/shutil.py\", line 488, in rmtree\n    os.rmdir(path)\n"
    },

/var/lib/pulp/ is a NFS share. Checking the directory in the error message shows that it is empty. Maybe an issue with NFS lock files?

This is on an rpm based installation on RHEL8:

python3-pulpcore-3.7.1-3.el8.noarch
python3-pulp-rpm-3.7.0-1.el8.noarch
python3-pulp-2to3-migration-0.4.0-1.el8.noarch

History

#1 Updated by adam.winberg@smhi.se about 1 year ago

Sorry about the formatting, I can't edit the original description apparantly. Here is the error output in a slightly more readable form:

    "error": {
        "description": "[Errno 39] Directory not empty: '/var/lib/pulp/tmp/2287063@lxserv2285.ex.com/7b022048-0d39-4d6d-a278-91ecb3c4779c'",
        "traceback": "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job
    rv = job.perform()
  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform
    self._result = self._execute()
  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py\", line 142, in migrate_from_pulp2
    create_repoversions_publications_distributions(plan)
  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 293, in create_repoversions_publications_distributions
    task_func(*task_args)
  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 187, in simple_plugin_migration
    migrate_repo_distributor(dist_migrator, progress_dist, pulp2_dist)
  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 391, in migrate_repo_distributor
    pulp2dist, repo_version)
  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py\", line 74, in migrate_to_pulp3
    publish(repo_version.pk, checksum_types=checksum_types)
  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py\", line 300, in publish
    metadata_signing_service=metadata_signing_service
  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py\", line 178, in __exit__
    self.delete()
  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py\", line 80, in delete
    self._delete()
  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py\", line 90, in _delete
    shutil.rmtree(self.path)
  File \"/usr/lib64/python3.6/shutil.py\", line 490, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File \"/usr/lib64/python3.6/shutil.py\", line 488, in rmtree
    os.rmdir(path)
"
    },

#2 Updated by adam.winberg@smhi.se about 1 year ago

I have updated to migration 0.5.0 and no longer see this issue.

#3 Updated by dalley about 1 year ago

  • Status changed from NEW to CLOSED - WORKSFORME

Great to hear. I'm going to close this issue, but if you encounter it again, feel free to reopen.

#4 Updated by adam.winberg@smhi.se about 1 year ago

This error re-appeared today while I was running 2to3-migration.

Nov 05 08:49:19  rq[2064]: pulp: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=rhel8-rh-codeready, version=1
Nov 05 08:50:09  rq[2064]: pulp: rq.worker:ERROR: Traceback (most recent call last):
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Nov 05 08:50:09  rq[2064]:     rv = job.perform()
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Nov 05 08:50:09  rq[2064]:     self._result = self._execute()
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Nov 05 08:50:09  rq[2064]:     return self.func(*self.args, **self.kwargs)
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 140, in migrate_from_pulp2
Nov 05 08:50:09  rq[2064]:     create_repoversions_publications_distributions(plan)
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 293, in create_repoversions_publications_distributions
Nov 05 08:50:09  rq[2064]:     task_func(*task_args)
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 187, in simple_plugin_migration
Nov 05 08:50:09  rq[2064]:     migrate_repo_distributor(dist_migrator, progress_dist, pulp2_dist)
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 391, in migrate_repo_distributor
Nov 05 08:50:09  rq[2064]:     pulp2dist, repo_version)
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 74, in migrate_to_pulp3
Nov 05 08:50:09  rq[2064]:     publish(repo_version.pk, checksum_types=checksum_types)
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 300, in publish
Nov 05 08:50:09  rq[2064]:     metadata_signing_service=metadata_signing_service
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 178, in __exit__
Nov 05 08:50:09  rq[2064]:     self.delete()
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 80, in delete
Nov 05 08:50:09  rq[2064]:     self._delete()
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 90, in _delete
Nov 05 08:50:09  rq[2064]:     shutil.rmtree(self.path)
Nov 05 08:50:09  rq[2064]:   File "/usr/lib64/python3.6/shutil.py", line 490, in rmtree
Nov 05 08:50:09  rq[2064]:     onerror(os.rmdir, path, sys.exc_info())
Nov 05 08:50:09  rq[2064]:   File "/usr/lib64/python3.6/shutil.py", line 488, in rmtree
Nov 05 08:50:09  rq[2064]:     os.rmdir(path)
Nov 05 08:50:09  rq[2064]: OSError: [Errno 39] Directory not empty: '/var/lib/pulp/tmp/2064@lxserv2285/4c0cc0a8-f4d2-4a8c-a199-9f2a67a51368'

The migration was nearly done, just the publishing left.

#5 Updated by adam.winberg@smhi.se about 1 year ago

A common cause for this error seems to be a process having an open file handle to a file in the directory, when the file is removed and there is still an existing open filehandle to it a 'replacement' file is created with the name '.nfsXXXXXXXXXXX'. This file exists until the filehandle is released. So maybe some process writing to this tmp dir is not properly releasing the file handle?

#6 Updated by adam.winberg@smhi.se about 1 year ago

Trying to workaround this by resorting to a 'rm -rf' on the directory if rmtree fails, and now I get this error:

rm: cannot remove '/var/lib/pulp/tmp/164660@lxserv2285.smhi.se/9f2684d0-3b5a-4046-9fda-4e694adaf2c0/.nfs00000000004d1f0800000005': Device or resource busy

So it is more or less confirmed that the problem is that files in the tmp directory is being removed while a running process has an open filehandle to it, and that that filehandle is not being released before trying to remove the directory.

#7 Updated by adam.winberg@smhi.se about 1 year ago

To make the migration work I had to add exception handling ('pass') of OSerror in the '_delete' function in storage.py, and also add the flag 'exist_ok=True' to os.makedirs in the 'create' function. Ugly workaround but makes my migration able to finish.

#8 Updated by adam.winberg@smhi.se 10 months ago

I updated to python3-pulpcore-3.9.1-1.el8.noarch and I still need the workaround to avoid following error when doing publish:

[Errno 39] Directory not empty: '/var/lib/pulp/tmp/3976957@lxserv2285/4f8caff2-522f-46bf-aae5-f67ca8611bee'"

I'm no longer doing 2to3 migrations, so this is not related to the migration plugin.

#9 Updated by adam.winberg@smhi.se 10 months ago

Can this be re-opened or do I need to create a new issue?

Please register to edit this issue

Also available in: Atom PDF