OSTree Support

It's currently impossible to sync an OSTree repository. I know https://pulp.plan.io/issues/1731 notes the same thing, but the exact set of errors are different, so I'm listing this in a separate bug.

This bug is currently only exhibited on Fedora 23. However, that's because the OSTree plugin isn't being installed on Fedora 22 or RHEL 6. The same issue might occur if we installed the OSTree plugin on other platforms.

Let's say you create an OSTree Pulp repository and sync it as follows. The procedure will fail:

# pulp-admin ostree repo create --repo-id ichi10 --branch 'fedora-atomic/f21/x86_64/updates/docker-host' --feed 'https://repos.fedorapeople.org/pulp/pulp/demo_repos/test-ostree-small' 
Repository [ichi10] successfully created

# pulp-admin ostree repo sync run --repo-id ichi10
+----------------------------------------------------------------------+
                   Synchronizing Repository [ichi10]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Task Failed

[Errno 2] No such file or directory

This failure also occurs via the API. Here's the relevant system log lines:

May 20 16:14:10 pulp.example.com pulp[16491]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[a0edd8a3-1bb4-4d5d-877a-6cac25da2beb]
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[709e729c-29db-48dc-83b4-64944857d7a0]
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[51cc89eb-1b48-496d-abab-bab392f93984]
May 20 16:14:10 pulp.example.com pulp[16491]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[a0edd8a3-1bb4-4d5d-877a-6cac25da2beb] succeeded in 0.0251055580447s: None
May 20 16:14:10 pulp.example.com pulp[16898]: pulp.server.async.tasks:INFO: Task failed : [709e729c-29db-48dc-83b4-64944857d7a0]
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408) Task pulp.server.managers.repo.sync.sync[709e729c-29db-48dc-83b4-64944857d7a0] raised unexpected: OSError(2, 'No such file or directory')
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408) Traceback (most recent call last):
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)     R = retval = fun(*args, **kwargs)
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)     return super(Task, self).__call__(*args, **kwargs)
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)     return super(PulpTask, self).__call__(*args, **kwargs)
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)     return self.run(*args, **kwargs)
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 760, in sync
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)     sync_report = sync_repo(transfer_repo, conduit, call_config)
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 658, in wrap_f
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)     return f(*args, **kwargs)
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)   File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/web.py", line 91, in sync_repo
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)     report = step.process_lifecycle()
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 562, in process_lifecycle
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)     super(PluginStep, self).process_lifecycle()
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 159, in process_lifecycle
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)     step.process()
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 249, in process
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)     self._process_block()
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 293, in _process_block
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)     self.process_main()
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)   File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/steps.py", line 93, in process_main
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)     remote.add()
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)   File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/steps.py", line 414, in add
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)     path, key_ids = self.gpg_keys
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)   File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/steps.py", line 385, in gpg_keys
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)     gpg = GPG(gnupghome=home)
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)   File "/usr/lib/python2.7/site-packages/gnupg.py", line 733, in __init__
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)     p = self._open_subprocess(["--version"])
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)   File "/usr/lib/python2.7/site-packages/gnupg.py", line 786, in _open_subprocess
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)     startupinfo=si)
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)   File "/usr/lib64/python2.7/subprocess.py", line 710, in __init__
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)     errread, errwrite)
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)   File "/usr/lib64/python2.7/subprocess.py", line 1335, in _execute_child
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408)     raise child_exception
May 20 16:14:10 pulp.example.com pulp[16641]: celery.worker.job:ERROR: (16641-37408) OSError: [Errno 2] No such file or directory
May 20 16:14:11 pulp.example.com pulp[16641]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[51cc89eb-1b48-496d-abab-bab392f93984] succeeded in 0.0370001610136s: None

Here's what things look like to a Pulp Smash user:

$ python -m unittest2 pulp_smash.tests.ostree.api_v2.test_sync_publish.SyncTestCase
E
======================================================================
ERROR: setUpClass (pulp_smash.tests.ostree.api_v2.test_sync_publish.SyncTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/tests/ostree/api_v2/test_sync_publish.py", line 118, in setUpClass
    cls.report = utils.sync_repo(cls.cfg, repo['_href'])
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/utils.py", line 418, in sync_repo
    return api.Client(server_config).post(urljoin(href, 'actions/sync/'))
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/api.py", line 279, in post
    return self.request('POST', url, **kwargs)
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/api.py", line 319, in request
    requests.request(method, **request_kwargs),
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/api.py", line 109, in safe_handler
    _handle_202(server_config, response)
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/api.py", line 86, in _handle_202
    _check_tasks(tasks)
  File "/home/ichimonji10/code/pulp-smash/pulp_smash/api.py", line 76, in _check_tasks
    raise exceptions.TaskReportError(msg)
pulp_smash.exceptions.TaskReportError: Task report /pulp/api/v2/tasks/3b60b573-e217-49f9-828e-de722bb0c2b5/ contains a error: {'sub_errors': [], 'data': {}, 'description': '[Errno 2] No such file or directory', 'code': 'PLP0000'}
Full task report: {'exception': None, 'finish_time': '2016-05-20T20:24:43Z', 'spawned_tasks': [], 'traceback': 'Traceback (most recent call last):\n  File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__\n    return super(Task, self).__call__(*args, **kwargs)\n  File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__\n    return super(PulpTask, self).__call__(*args, **kwargs)\n  File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__\n    return self.run(*args, **kwargs)\n  File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 760, in sync\n    sync_report = sync_repo(transfer_repo, conduit, call_config)\n  File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 658, in wrap_f\n    return f(*args, **kwargs)\n  File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/web.py", line 91, in sync_repo\n    report = step.process_lifecycle()\n  File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 562, in process_lifecycle\n    super(PluginStep, self).process_lifecycle()\n  File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 159, in process_lifecycle\n    step.process()\n  File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 249, in process\n    self._process_block()\n  File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 293, in _process_block\n    self.process_main()\n  File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/steps.py", line 93, in process_main\n    remote.add()\n  File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/steps.py", line 414, in add\n    path, key_ids = self.gpg_keys\n  File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/steps.py", line 385, in gpg_keys\n    gpg = GPG(gnupghome=home)\n  File "/usr/lib/python2.7/site-packages/gnupg.py", line 733, in __init__\n    p = self._open_subprocess(["--version"])\n  File "/usr/lib/python2.7/site-packages/gnupg.py", line 786, in _open_subprocess\n    startupinfo=si)\n  File "/usr/lib64/python2.7/subprocess.py", line 710, in __init__\n    errread, errwrite)\n  File "/usr/lib64/python2.7/subprocess.py", line 1335, in _execute_child\n    raise child_exception\nOSError: [Errno 2] No such file or directory\n', 'start_time': '2016-05-20T20:24:43Z', 'worker_name': 'reserved_resource_worker-2@pulp.example.com', 'id': '573f728be64aa79aacd764a9', 'queue': 'reserved_resource_worker-2@pulp.example.com.dq', '_ns': 'task_status', 'state': 'error', 'task_type': 'pulp.server.managers.repo.sync.sync', 'result': None, '_id': {'$oid': '573f728be64aa79aacd764a9'}, 'tags': ['pulp:repository:7d416b69-8b77-4740-b763-820a91c7a852', 'pulp:action:sync'], '_href': '/pulp/api/v2/tasks/3b60b573-e217-49f9-828e-de722bb0c2b5/', 'progress_report': {'ostree_web_importer': [{'description': 'Create Local Repository', 'num_success': 0, 'num_processed': 1, 'items_total': 1, 'num_failures': 1, 'step_type': 'import_create_repository', 'error_details': [{'error': '[Errno 2] No such file or directory', 'traceback': '  File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 249, in process\n    self._process_block()\n\n  File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 293, in _process_block\n    self.process_main()\n\n  File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/steps.py", line 93, in process_main\n    remote.add()\n\n  File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/steps.py", line 414, in add\n    path, key_ids = self.gpg_keys\n\n  File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/steps.py", line 385, in gpg_keys\n    gpg = GPG(gnupghome=home)\n\n  File "/usr/lib/python2.7/site-packages/gnupg.py", line 733, in __init__\n    p = self._open_subprocess(["--version"])\n\n  File "/usr/lib/python2.7/site-packages/gnupg.py", line 786, in _open_subprocess\n    startupinfo=si)\n\n  File "/usr/lib64/python2.7/subprocess.py", line 710, in __init__\n    errread, errwrite)\n\n  File "/usr/lib64/python2.7/subprocess.py", line 1335, in _execute_child\n    raise child_exception\n'}], 'step_id': '20504bb8-927f-449e-82d6-75e8c80b7f49', 'details': '', 'state': 'FAILED'}, {'description': 'Update Summary', 'num_success': 0, 'num_processed': 0, 'items_total': 1, 'num_failures': 0, 'step_type': 'import_summary', 'error_details': [], 'step_id': 'c01fd9bc-87ff-4275-aeb5-d613889b61de', 'details': '', 'state': 'NOT_STARTED'}, {'description': 'Pull Remote Branches', 'num_success': 0, 'num_processed': 0, 'items_total': 1, 'num_failures': 0, 'step_type': 'import_pull', 'error_details': [], 'step_id': 'd9317432-ed53-4680-bec3-c5de48c825ae', 'details': '', 'state': 'NOT_STARTED'}, {'description': 'Add Content Units', 'num_success': 0, 'num_processed': 0, 'items_total': 1, 'num_failures': 0, 'step_type': 'import_add_unit', 'error_details': [], 'step_id': '1b7b4e89-422f-4f65-a6ac-36ae2e039814', 'details': '', 'state': 'NOT_STARTED'}, {'description': 'Clean', 'num_success': 0, 'num_processed': 0, 'items_total': 1, 'num_failures': 0, 'step_type': 'import_clean', 'error_details': [], 'step_id': '3495709c-1e81-4184-9910-6d1cd110a539', 'details': '', 'state': 'NOT_STARTED'}]}, 'task_id': '3b60b573-e217-49f9-828e-de722bb0c2b5', 'error': {'sub_errors': [], 'data': {}, 'description': '[Errno 2] No such file or directory', 'code': 'PLP0000'}}

----------------------------------------------------------------------
Ran 0 tests in 3.196s

FAILED (errors=1)

Finally, for the record:

$ ssh $hostname rpm -qa | grep pulp | sort
pulp-admin-client-2.8.3-0.4.rc.git.1.72869b6.fc23.noarch
pulp-docker-admin-extensions-2.0.1-0.4.rc.git.1.bab7be4.fc23.noarch
pulp-docker-plugins-2.0.1-0.4.rc.git.1.bab7be4.fc23.noarch
pulp-ostree-admin-extensions-1.1.1-0.4.rc.git.7.bd7b441.fc23.noarch
pulp-ostree-plugins-1.1.1-0.4.rc.git.7.bd7b441.fc23.noarch
pulp-puppet-admin-extensions-2.8.3-0.4.rc.git.1.ffbcab2.fc23.noarch
pulp-puppet-plugins-2.8.3-0.4.rc.git.1.ffbcab2.fc23.noarch
pulp-python-admin-extensions-1.1.1-0.4.rc.git.1.e375cd0.fc23.noarch
pulp-python-plugins-1.1.1-0.4.rc.git.1.e375cd0.fc23.noarch
pulp-rpm-admin-extensions-2.8.3-0.4.rc.git.1.2634acc.fc23.noarch
pulp-rpm-plugins-2.8.3-0.4.rc.git.1.2634acc.fc23.noarch
pulp-selinux-2.8.3-0.4.rc.git.1.72869b6.fc23.noarch
pulp-server-2.8.3-0.4.rc.git.1.72869b6.fc23.noarch
python-kombu-3.0.33-5.pulp.fc23.noarch
python-pulp-bindings-2.8.3-0.4.rc.git.1.72869b6.fc23.noarch
python-pulp-client-lib-2.8.3-0.4.rc.git.1.72869b6.fc23.noarch
python-pulp-common-2.8.3-0.4.rc.git.1.72869b6.fc23.noarch
python-pulp-docker-common-2.0.1-0.4.rc.git.1.bab7be4.fc23.noarch
python-pulp-oid_validation-2.8.3-0.4.rc.git.1.72869b6.fc23.noarch
python-pulp-ostree-common-1.1.1-0.4.rc.git.7.bd7b441.fc23.noarch
python-pulp-puppet-common-2.8.3-0.4.rc.git.1.ffbcab2.fc23.noarch
python-pulp-python-common-1.1.1-0.4.rc.git.1.e375cd0.fc23.noarch
python-pulp-repoauth-2.8.3-0.4.rc.git.1.72869b6.fc23.noarch
python-pulp-rpm-common-2.8.3-0.4.rc.git.1.2634acc.fc23.noarch
python-pulp-streamer-2.8.3-0.4.rc.git.1.72869b6.fc23.noarch