Project

Profile

Help

Issue #988

Cannot sync: No such file or directory: '/var/cache/pulp/reserved_resource_worker-X…'

Added by rbarlow over 5 years ago. Updated 7 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Category:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Master
Platform Release:
2.8.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

I noticed that I cannot sync with Pulp master as of last week. It seems to be that nothing is creating the required structure under /var/cache/pulp/, but the sync code is assuming that structure has been created. If you do a quick developer install with master pulp and pulp_rpm, and then sync a repo, you will see this:

celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[8c52597c-8659-481f-a56a-2c45bfc969de]
celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[9e8610f6-fc8d-40e5-a32c-49775cbc65cc]
celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[afc8a5db-93cd-4346-9af7-3b5dd6a1913a] succeeded in 29.11072141s: None
pulp.server.async.tasks:INFO: Task failed : [8c52597c-8659-481f-a56a-2c45bfc969de]
celery.worker.job:ERROR: (16705-96320) Task pulp.server.managers.repo.sync.sync[8c52597c-8659-481f-a56a-2c45bfc969de] raised unexpected: OSError(2, 'No such file or direct
celery.worker.job:ERROR: (16705-96320) Traceback (most recent call last):
celery.worker.job:ERROR: (16705-96320)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
celery.worker.job:ERROR: (16705-96320)     R = retval = fun(*args, **kwargs)
celery.worker.job:ERROR: (16705-96320)   File "/home/vagrant/devel/pulp/server/pulp/server/async/tasks.py", line 392, in __call__
celery.worker.job:ERROR: (16705-96320)     return super(Task, self).__call__(*args, **kwargs)
celery.worker.job:ERROR: (16705-96320)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
celery.worker.job:ERROR: (16705-96320)     return self.run(*args, **kwargs)
celery.worker.job:ERROR: (16705-96320)   File "/home/vagrant/devel/pulp/server/pulp/server/managers/repo/sync.py", line 87, in sync
celery.worker.job:ERROR: (16705-96320)     transfer_repo.working_dir = common_utils.get_working_directory()
celery.worker.job:ERROR: (16705-96320)   File "/home/vagrant/devel/pulp/server/pulp/server/managers/repo/_common.py", line 195, in get_working_directory
celery.worker.job:ERROR: (16705-96320)     os.mkdir(working_dir_root)
celery.worker.job:ERROR: (16705-96320) OSError: [Errno 2] No such file or directory: '/var/cache/pulp/reserved_resource_worker-1@pulp-devel/8c52597c-8659-481f-a56a-2c45bfc

Steps to reproduce:
1) Check out pulp and pulp_rpm at the latest revision of master
2) Use vagrant to build a development environment (vagrant syncs the zoo repo as the final step)

Expected behavior:
Zoo should sync

Actual behavior:
You will receive the above error in the log and a similar error from the CLI, since /var/cache/pulp/reserved_resource_worker-X@pulp-devel will not exist.

I am not sure if this issue only affects master, or if it is present in 2.7. I haven't heard anybody else talk about it being broken in 2.7 and I believe most of us are working in that branch these days so it may only be broken on master.

Associated revisions

Revision 951cf3f3 View on GitHub
Added by rbarlow over 5 years ago

Perform all Worker setup in the same signal.

We had two different Celery startup signal handlers registered, and it seemed that the second was no longer being sent. This commit ensures that all worker startup code happens in the same signal handler, and that it occurs in the correct order.

https://pulp.plan.io/issues/988

fixed #988

History

#1 Updated by rbarlow over 5 years ago

  • Subject changed from Cannot sync with Pulp master to Cannot sync: No such file or directory: '/var/cache/pulp/reserved_resource_worker-X…'

#2 Updated by mhrivnak over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to rbarlow
  • Priority changed from Normal to High
  • Triaged changed from No to Yes

#3 Updated by rbarlow over 5 years ago

  • Status changed from ASSIGNED to POST

#4 Updated by rbarlow over 5 years ago

  • Status changed from POST to MODIFIED

I typo'd my commit message ☹ s/fixed/fixes/

#5 Updated by bmbouter over 5 years ago

I manually associated the revision using redmine. FYI, these are all editable outside of commit messages.

#6 Updated by dkliban@redhat.com over 5 years ago

  • Platform Release set to 2.8.0

#7 Updated by dkliban@redhat.com over 4 years ago

  • Status changed from MODIFIED to 5

#9 Updated by rbarlow over 4 years ago

  • Status changed from 5 to 6

#10 Updated by dkliban@redhat.com over 4 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#12 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF