Issue #988


Cannot sync: No such file or directory: '/var/cache/pulp/reserved_resource_worker-X…'

Added by rbarlow over 8 years ago. Updated over 3 years ago.

Start date:
Due date:
Estimated time:
3. High
Platform Release:
Sprint Candidate:
Pulp 2


I noticed that I cannot sync with Pulp master as of last week. It seems to be that nothing is creating the required structure under /var/cache/pulp/, but the sync code is assuming that structure has been created. If you do a quick developer install with master pulp and pulp_rpm, and then sync a repo, you will see this:

celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[8c52597c-8659-481f-a56a-2c45bfc969de]
celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[9e8610f6-fc8d-40e5-a32c-49775cbc65cc]
celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[afc8a5db-93cd-4346-9af7-3b5dd6a1913a] succeeded in 29.11072141s: None
pulp.server.async.tasks:INFO: Task failed : [8c52597c-8659-481f-a56a-2c45bfc969de]
celery.worker.job:ERROR: (16705-96320) Task pulp.server.managers.repo.sync.sync[8c52597c-8659-481f-a56a-2c45bfc969de] raised unexpected: OSError(2, 'No such file or direct
celery.worker.job:ERROR: (16705-96320) Traceback (most recent call last):
celery.worker.job:ERROR: (16705-96320)   File "/usr/lib/python2.7/site-packages/celery/app/", line 240, in trace_task
celery.worker.job:ERROR: (16705-96320)     R = retval = fun(*args, **kwargs)
celery.worker.job:ERROR: (16705-96320)   File "/home/vagrant/devel/pulp/server/pulp/server/async/", line 392, in __call__
celery.worker.job:ERROR: (16705-96320)     return super(Task, self).__call__(*args, **kwargs)
celery.worker.job:ERROR: (16705-96320)   File "/usr/lib/python2.7/site-packages/celery/app/", line 437, in __protected_call__
celery.worker.job:ERROR: (16705-96320)     return*args, **kwargs)
celery.worker.job:ERROR: (16705-96320)   File "/home/vagrant/devel/pulp/server/pulp/server/managers/repo/", line 87, in sync
celery.worker.job:ERROR: (16705-96320)     transfer_repo.working_dir = common_utils.get_working_directory()
celery.worker.job:ERROR: (16705-96320)   File "/home/vagrant/devel/pulp/server/pulp/server/managers/repo/", line 195, in get_working_directory
celery.worker.job:ERROR: (16705-96320)     os.mkdir(working_dir_root)
celery.worker.job:ERROR: (16705-96320) OSError: [Errno 2] No such file or directory: '/var/cache/pulp/reserved_resource_worker-1@pulp-devel/8c52597c-8659-481f-a56a-2c45bfc

Steps to reproduce:
1) Check out pulp and pulp_rpm at the latest revision of master
2) Use vagrant to build a development environment (vagrant syncs the zoo repo as the final step)

Expected behavior:
Zoo should sync

Actual behavior:
You will receive the above error in the log and a similar error from the CLI, since /var/cache/pulp/reserved_resource_worker-X@pulp-devel will not exist.

I am not sure if this issue only affects master, or if it is present in 2.7. I haven't heard anybody else talk about it being broken in 2.7 and I believe most of us are working in that branch these days so it may only be broken on master.

Actions #1

Updated by rbarlow over 8 years ago

  • Subject changed from Cannot sync with Pulp master to Cannot sync: No such file or directory: '/var/cache/pulp/reserved_resource_worker-X…'
Actions #2

Updated by mhrivnak over 8 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to rbarlow
  • Priority changed from Normal to High
  • Triaged changed from No to Yes
Actions #3

Updated by rbarlow over 8 years ago

  • Status changed from ASSIGNED to POST

Added by rbarlow over 8 years ago

Revision 951cf3f3 | View on GitHub

Perform all Worker setup in the same signal.

We had two different Celery startup signal handlers registered, and it seemed that the second was no longer being sent. This commit ensures that all worker startup code happens in the same signal handler, and that it occurs in the correct order.

fixed #988

Actions #4

Updated by rbarlow over 8 years ago

  • Status changed from POST to MODIFIED

I typo'd my commit message ☹ s/fixed/fixes/

Actions #5

Updated by bmbouter over 8 years ago

I manually associated the revision using redmine. FYI, these are all editable outside of commit messages.

Actions #6

Updated by over 8 years ago

  • Platform Release set to 2.8.0
Actions #7

Updated by almost 8 years ago

  • Status changed from MODIFIED to 5
Actions #9

Updated by rbarlow almost 8 years ago

  • Status changed from 5 to 6
Actions #10

Updated by over 7 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions #12

Updated by bmbouter over 4 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF