Ansible Plugin

When attempting to sync a repo to a CollectionRemote that has a 'url' pointing to automation-hub, the sync fails in pulp_ansible.tasks.collections.CollectionSyncFirstStage._fetch_collections().

For automation-hub / v3 collections API, the 'versions_url' field no longer includes a scheme. Previously a 'versions_url' would be a full url like https://galaxy.ansible.com/api/v3/collections/some_namespace/some_collection/versions/, but the v3 api uses '/api/automation-hub/v3/collections/some_namespace/some_collection/versions'.

When the CollectionSync tasks uses 'version_url' (https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/tasks/collections.py#L349) it eventually causes pulpcore DownloadFactory to hit a KeyError looking up the '' scheme (https://github.com/pulp/pulpcore/blob/master/pulpcore/download/factory.py#L135-L138)

https://gist.github.com/alikins/b6349ad49a5ce1b25fa6c9d938a01764#file-gistfile2-txt has some logs pasted here for reference:

pulp-worker_1            | pulp: rq.worker:ERROR: ValueError: URL: /api/automation-hub/v3/collections/autohubtest2/collection_dep_a_jbkboxzc/versions/ not supported.
pulp-worker_1            | Traceback (most recent call last):
pulp-worker_1            |   File "/venv/lib64/python3.6/site-packages/pulpcore/download/factory.py", line 137, in build
pulp-worker_1            |     builder = self._handler_map[scheme]
pulp-worker_1            | KeyError: ''
pulp-worker_1            | 
pulp-worker_1            | During handling of the above exception, another exception occurred:
pulp-worker_1            | 
pulp-worker_1            | Traceback (most recent call last):
pulp-worker_1            |   File "/venv/lib64/python3.6/site-packages/rq/worker.py", line 884, in perform_job
pulp-worker_1            |     rv = job.perform()
pulp-worker_1            |   File "/venv/lib64/python3.6/site-packages/rq/job.py", line 664, in perform
pulp-worker_1            |     self._result = self._execute()
pulp-worker_1            |   File "/venv/lib64/python3.6/site-packages/rq/job.py", line 670, in _execute
pulp-worker_1            |     return self.func(*self.args, **self.kwargs)
pulp-worker_1            |   File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 80, in sync
pulp-worker_1            |     d_version.create()
pulp-worker_1            |   File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
pulp-worker_1            |     loop.run_until_complete(pipeline)
pulp-worker_1            |   File "/opt/rh/rh-python36/root/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
pulp-worker_1            |     return future.result()
pulp-worker_1            |   File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 228, in create_pipeline
pulp-worker_1            |     await asyncio.gather(*futures)
pulp-worker_1            |   File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
pulp-worker_1            |     await self.run()
pulp-worker_1            |   File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 274, in run
pulp-worker_1            |     async for metadata in self._fetch_collections():
pulp-worker_1            |   File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 351, in _fetch_collections
pulp-worker_1            |     [remote.get_downloader(url=result["versions_url"]).run()]
pulp-worker_1            |   File "/venv/lib64/python3.6/site-packages/pulpcore/app/models/repository.py", line 288, in get_downloader
pulp-worker_1            |     return self.download_factory.build(url, **kwargs)
pulp-worker_1            |   File "/venv/lib64/python3.6/site-packages/pulpcore/download/factory.py", line 140, in build
pulp-worker_1            |     raise ValueError(_('URL: {u} not supported.'.format(u=url)))
pulp-worker_1            | ValueError: URL: /api/automation-hub/v3/collections/autohubtest2/collection_dep_a_jbkboxzc/versions/ not supported.
pulp-worker_1            | Traceback (most recent call last):
pulp-worker_1            |   File "/venv/lib64/python3.6/site-packages/pulpcore/download/factory.py", line 137, in build
pulp-worker_1            |     builder = self._handler_map[scheme]
pulp-worker_1            | KeyError: ''
pulp-worker_1            | 
pulp-worker_1            | During handling of the above exception, another exception occurred:
pulp-worker_1            | 
pulp-worker_1            | Traceback (most recent call last):
pulp-worker_1            |   File "/venv/lib64/python3.6/site-packages/rq/worker.py", line 884, in perform_job
pulp-worker_1            |     rv = job.perform()
pulp-worker_1            |   File "/venv/lib64/python3.6/site-packages/rq/job.py", line 664, in perform
pulp-worker_1            |     self._result = self._execute()
pulp-worker_1            |   File "/venv/lib64/python3.6/site-packages/rq/job.py", line 670, in _execute
pulp-worker_1            |     return self.func(*self.args, **self.kwargs)
pulp-worker_1            |   File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 80, in sync
pulp-worker_1            |     d_version.create()
pulp-worker_1            |   File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
pulp-worker_1            |     loop.run_until_complete(pipeline)
pulp-worker_1            |   File "/opt/rh/rh-python36/root/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
pulp-worker_1            |     return future.result()
pulp-worker_1            |   File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 228, in create_pipeline
pulp-worker_1            |     await asyncio.gather(*futures)
pulp-worker_1            |   File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
pulp-worker_1            |     await self.run()
pulp-worker_1            |   File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 274, in run
pulp-worker_1            |     async for metadata in self._fetch_collections():
pulp-worker_1            |   File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 351, in _fetch_collections
pulp-worker_1            |     [remote.get_downloader(url=result["versions_url"]).run()]
pulp-worker_1            |   File "/venv/lib64/python3.6/site-packages/pulpcore/app/models/repository.py", line 288, in get_downloader
pulp-worker_1            |     return self.download_factory.build(url, **kwargs)
pulp-worker_1            |   File "/venv/lib64/python3.6/site-packages/pulpcore/download/factory.py", line 140, in build
pulp-worker_1            |     raise ValueError(_('URL: {u} not supported.'.format(u=url)))
pulp-worker_1            | ValueError: URL: /api/automation-hub/v3/collections/autohubtest2/collection_dep_a_jbkboxzc/versions/ not supported.