Issue #988
closedCannot sync: No such file or directory: '/var/cache/pulp/reserved_resource_worker-X…'
Description
I noticed that I cannot sync with Pulp master as of last week. It seems to be that nothing is creating the required structure under /var/cache/pulp/, but the sync code is assuming that structure has been created. If you do a quick developer install with master pulp and pulp_rpm, and then sync a repo, you will see this:
celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[8c52597c-8659-481f-a56a-2c45bfc969de]
celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[9e8610f6-fc8d-40e5-a32c-49775cbc65cc]
celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[afc8a5db-93cd-4346-9af7-3b5dd6a1913a] succeeded in 29.11072141s: None
pulp.server.async.tasks:INFO: Task failed : [8c52597c-8659-481f-a56a-2c45bfc969de]
celery.worker.job:ERROR: (16705-96320) Task pulp.server.managers.repo.sync.sync[8c52597c-8659-481f-a56a-2c45bfc969de] raised unexpected: OSError(2, 'No such file or direct
celery.worker.job:ERROR: (16705-96320) Traceback (most recent call last):
celery.worker.job:ERROR: (16705-96320) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
celery.worker.job:ERROR: (16705-96320) R = retval = fun(*args, **kwargs)
celery.worker.job:ERROR: (16705-96320) File "/home/vagrant/devel/pulp/server/pulp/server/async/tasks.py", line 392, in __call__
celery.worker.job:ERROR: (16705-96320) return super(Task, self).__call__(*args, **kwargs)
celery.worker.job:ERROR: (16705-96320) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
celery.worker.job:ERROR: (16705-96320) return self.run(*args, **kwargs)
celery.worker.job:ERROR: (16705-96320) File "/home/vagrant/devel/pulp/server/pulp/server/managers/repo/sync.py", line 87, in sync
celery.worker.job:ERROR: (16705-96320) transfer_repo.working_dir = common_utils.get_working_directory()
celery.worker.job:ERROR: (16705-96320) File "/home/vagrant/devel/pulp/server/pulp/server/managers/repo/_common.py", line 195, in get_working_directory
celery.worker.job:ERROR: (16705-96320) os.mkdir(working_dir_root)
celery.worker.job:ERROR: (16705-96320) OSError: [Errno 2] No such file or directory: '/var/cache/pulp/reserved_resource_worker-1@pulp-devel/8c52597c-8659-481f-a56a-2c45bfc
Steps to reproduce:
1) Check out pulp and pulp_rpm at the latest revision of master
2) Use vagrant to build a development environment (vagrant syncs the zoo repo as the final step)
Expected behavior:
Zoo should sync
Actual behavior:
You will receive the above error in the log and a similar error from the CLI, since /var/cache/pulp/reserved_resource_worker-X@pulp-devel will not exist.
I am not sure if this issue only affects master, or if it is present in 2.7. I haven't heard anybody else talk about it being broken in 2.7 and I believe most of us are working in that branch these days so it may only be broken on master.
Perform all Worker setup in the same signal.
We had two different Celery startup signal handlers registered, and it seemed that the second was no longer being sent. This commit ensures that all worker startup code happens in the same signal handler, and that it occurs in the correct order.
https://pulp.plan.io/issues/988
fixed #988