Migration Plugin

Sorry about the formatting, I can't edit the original description apparantly. Here is the error output in a slightly more readable form:

    "error": {
        "description": "[Errno 39] Directory not empty: '/var/lib/pulp/tmp/2287063@lxserv2285.ex.com/7b022048-0d39-4d6d-a278-91ecb3c4779c'",
        "traceback": "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job
    rv = job.perform()
  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform
    self._result = self._execute()
  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py\", line 142, in migrate_from_pulp2
    create_repoversions_publications_distributions(plan)
  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 293, in create_repoversions_publications_distributions
    task_func(*task_args)
  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 187, in simple_plugin_migration
    migrate_repo_distributor(dist_migrator, progress_dist, pulp2_dist)
  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 391, in migrate_repo_distributor
    pulp2dist, repo_version)
  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py\", line 74, in migrate_to_pulp3
    publish(repo_version.pk, checksum_types=checksum_types)
  File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py\", line 300, in publish
    metadata_signing_service=metadata_signing_service
  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py\", line 178, in __exit__
    self.delete()
  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py\", line 80, in delete
    self._delete()
  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py\", line 90, in _delete
    shutil.rmtree(self.path)
  File \"/usr/lib64/python3.6/shutil.py\", line 490, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File \"/usr/lib64/python3.6/shutil.py\", line 488, in rmtree
    os.rmdir(path)
"
    },

I have updated to migration 0.5.0 and no longer see this issue.

This error re-appeared today while I was running 2to3-migration.

Nov 05 08:49:19  rq[2064]: pulp: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=rhel8-rh-codeready, version=1
Nov 05 08:50:09  rq[2064]: pulp: rq.worker:ERROR: Traceback (most recent call last):
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Nov 05 08:50:09  rq[2064]:     rv = job.perform()
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Nov 05 08:50:09  rq[2064]:     self._result = self._execute()
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Nov 05 08:50:09  rq[2064]:     return self.func(*self.args, **self.kwargs)
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 140, in migrate_from_pulp2
Nov 05 08:50:09  rq[2064]:     create_repoversions_publications_distributions(plan)
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 293, in create_repoversions_publications_distributions
Nov 05 08:50:09  rq[2064]:     task_func(*task_args)
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 187, in simple_plugin_migration
Nov 05 08:50:09  rq[2064]:     migrate_repo_distributor(dist_migrator, progress_dist, pulp2_dist)
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 391, in migrate_repo_distributor
Nov 05 08:50:09  rq[2064]:     pulp2dist, repo_version)
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 74, in migrate_to_pulp3
Nov 05 08:50:09  rq[2064]:     publish(repo_version.pk, checksum_types=checksum_types)
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 300, in publish
Nov 05 08:50:09  rq[2064]:     metadata_signing_service=metadata_signing_service
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 178, in __exit__
Nov 05 08:50:09  rq[2064]:     self.delete()
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 80, in delete
Nov 05 08:50:09  rq[2064]:     self._delete()
Nov 05 08:50:09  rq[2064]:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 90, in _delete
Nov 05 08:50:09  rq[2064]:     shutil.rmtree(self.path)
Nov 05 08:50:09  rq[2064]:   File "/usr/lib64/python3.6/shutil.py", line 490, in rmtree
Nov 05 08:50:09  rq[2064]:     onerror(os.rmdir, path, sys.exc_info())
Nov 05 08:50:09  rq[2064]:   File "/usr/lib64/python3.6/shutil.py", line 488, in rmtree
Nov 05 08:50:09  rq[2064]:     os.rmdir(path)
Nov 05 08:50:09  rq[2064]: OSError: [Errno 39] Directory not empty: '/var/lib/pulp/tmp/2064@lxserv2285/4c0cc0a8-f4d2-4a8c-a199-9f2a67a51368'

The migration was nearly done, just the publishing left.

A common cause for this error seems to be a process having an open file handle to a file in the directory, when the file is removed and there is still an existing open filehandle to it a 'replacement' file is created with the name '.nfsXXXXXXXXXXX'. This file exists until the filehandle is released. So maybe some process writing to this tmp dir is not properly releasing the file handle?

Trying to workaround this by resorting to a 'rm -rf' on the directory if rmtree fails, and now I get this error:

rm: cannot remove '/var/lib/pulp/tmp/164660@lxserv2285.smhi.se/9f2684d0-3b5a-4046-9fda-4e694adaf2c0/.nfs00000000004d1f0800000005': Device or resource busy

So it is more or less confirmed that the problem is that files in the tmp directory is being removed while a running process has an open filehandle to it, and that that filehandle is not being released before trying to remove the directory.

To make the migration work I had to add exception handling ('pass') of OSerror in the '_delete' function in storage.py, and also add the flag 'exist_ok=True' to os.makedirs in the 'create' function. Ugly workaround but makes my migration able to finish.

I updated to python3-pulpcore-3.9.1-1.el8.noarch and I still need the workaround to avoid following error when doing publish:

[Errno 39] Directory not empty: '/var/lib/pulp/tmp/3976957@lxserv2285/4f8caff2-522f-46bf-aae5-f67ca8611bee'"

I'm no longer doing 2to3 migrations, so this is not related to the migration plugin.

Can this be re-opened or do I need to create a new issue?