Project

Profile

Help

Actions
Send by e-mail Copy link

Story #5288

closed

As a user, I can sync collections from another Pulp server

Added by sajha over 5 years ago. Updated about 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
daviddavis
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 58
Quarter:

Description

This is from before the requirements.yaml PR went in..

STEPS TO REPRODUCE:

#Create repo
http POST $BASE_ADDR/pulp/api/v3/repositories/ name=foo
export REPO_HREF=$(http $BASE_ADDR/pulp/api/v3/repositories/ | \
  jq -r '.results[] | select(.name == "foo") | ._href')

#Create remote
http POST $BASE_ADDR/pulp/api/v3/remotes/ansible/collection/     name='bar'     url='https://galaxy-dev.ansible.com/api/v2/collections/testing/ansible_testing_content'
export REMOTE_HREF=$(http $BASE_ADDR/pulp/api/v3/remotes/ansible/collection/ | jq -r '.results[] | select(.name == "bar") | ._href')

#Sync repo with Remote
export TASK_URL=$(http POST $BASE_ADDR$REMOTE_HREF'sync/' repository=$REPO_HREF \
  | jq -r '.task')
export REPOVERSION_HREF=$(http $BASE_ADDR$TASK_URL | jq -r '.created_resources[]')

#Create Distribution for the synced Repo
export TASK_URL=$(http POST $BASE_ADDR/pulp/api/v3/distributions/ansible/ansible/ \
  name='baz' \
  base_path='foo' \
  repository=${REPO_HREF} | jq -r '.task')
export DIST_PATH=$(http $BASE_ADDR$TASK_URL | jq -r '.created_resources[]')

#Form SYNC_URL
export GALAXY_URL=$(http $BASE_ADDR$DIST_PATH | jq -r '.mazer_url')
export SYNC_URL=$GALAXY_URL'/api/v2/collections/testing/ansible_testing_content'
http $SYNC_URL
# Notice Error in /var/log/messages around multiple collection versions being returned.
http $SYNC_URL'/versions'

#-----------------Try to use $SYNC_URL as remote for 2nd repo----------
#Create 2nd repo to sync from the sync url
export REPO_HREF=$(http $BASE_ADDR/pulp/api/v3/repositories/ | \
  jq -r '.results[] | select(.name == "foo1") | ._href')

#Create remote with url pointing to pulp's galaxy endpoint of 1st repo
http POST $BASE_ADDR/pulp/api/v3/remotes/ansible/collection/     name='bar1'     url=$SYNC_URL
export REMOTE_HREF=$(http $BASE_ADDR/pulp/api/v3/remotes/ansible/collection/ | jq -r '.results[] | select(.name == "bar1") | ._href')

#Notice url in the Remote
http $BASE_HREF$REMOTE_HREF

#Sync repo2 with remote poinitng to 1st repos pulp-galaxy url
export TASK_URL=$(http POST $BASE_ADDR$REMOTE_HREF'sync/' repository=$REPO_HREF \
  | jq -r '.task')
#Notice failure in /var/log/messages
http $BASE_HREF$TASK_URL

#http://localhost:24817/pulp_ansible/galaxy/foo/api/v2/collections/testing/ansible_testing_content/versions/4.0.4/ as sync URL fails too

Tried this with a collection https://galaxy.ansible.com/api/v2/collections/devoperate/base/versions/ having only one version. Got the error:

"error": {
        "code": null,
        "description": "string indices must be integers",
        "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 822, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 605, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 611, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py\", line 66, in sync\n    d_version.create()\n  File \"/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py\", line 169, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py\", line 209, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py\", line 195, in run\n    async for metadata in self._fetch_collections():\n  File \"/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py\", line 242, in _fetch_collections\n    \"namespace\": result[\"namespace\"][\"name\"],\n"
    },

Related issues

Related to Ansible Plugin - Test #5333: As a user, I can sync collections from another Pulp serverCLOSED - COMPLETEkersomActions
Actions
Copy link
 #1

Updated by sajha over 5 years ago

Because I updated my system to bring in the latest changes around collection sync, I can't vocuh for these steps. But the error I was seeing on a GET from the /pulp_ansible/galaxy/foo/api/v2/collections/testing/ansible_testing_content was

Aug 15 19:29:22 ansible-demo gunicorn: 127.0.0.1 - - [15/Aug/2019:19:29:22 +0000] "GET /pulp_ansible/galaxy/foo/api/v2/collections/testing/ansible_testing_content?page=1 HTTP/1.1" 301 0 "-" "Python/3.6 aiohttp/3.5.4"
Aug 15 19:29:22 ansible-demo gunicorn: pulp: django.request:ERROR: Internal Server Error: /pulp_ansible/galaxy/foo/api/v2/collections/testing/ansible_testing_content/
Aug 15 19:29:22 ansible-demo gunicorn: Traceback (most recent call last):
Aug 15 19:29:22 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
Aug 15 19:29:22 ansible-demo gunicorn: response = get_response(request)
Aug 15 19:29:22 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
Aug 15 19:29:22 ansible-demo gunicorn: response = self.process_exception_by_middleware(e, request)
Aug 15 19:29:22 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
Aug 15 19:29:22 ansible-demo gunicorn: response = wrapped_callback(request, *callback_args, **callback_kwargs)
Aug 15 19:29:22 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Aug 15 19:29:22 ansible-demo gunicorn: return view_func(*args, **kwargs)
Aug 15 19:29:22 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/views/generic/base.py", line 71, in view
Aug 15 19:29:22 ansible-demo gunicorn: return self.dispatch(request, *args, **kwargs)
Aug 15 19:29:22 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/views.py", line 497, in dispatch
Aug 15 19:29:22 ansible-demo gunicorn: response = self.handle_exception(exc)
Aug 15 19:29:22 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/views.py", line 457, in handle_exception
Aug 15 19:29:22 ansible-demo gunicorn: self.raise_uncaught_exception(exc)
Aug 15 19:29:22 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/views.py", line 468, in raise_uncaught_exception
Aug 15 19:29:22 ansible-demo gunicorn: raise exc
Aug 15 19:29:22 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/views.py", line 494, in dispatch
Aug 15 19:29:22 ansible-demo gunicorn: response = handler(request, *args, **kwargs)
Aug 15 19:29:22 ansible-demo gunicorn: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/galaxy/views.py", line 114, in get
Aug 15 19:29:22 ansible-demo gunicorn: CollectionVersion, collection__namespace=namespace, collection__name=name
Aug 15 19:29:22 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/shortcuts.py", line 93, in get_object_or_404
Aug 15 19:29:22 ansible-demo gunicorn: return queryset.get(*args, **kwargs)
Aug 15 19:29:22 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/query.py", line 412, in get
Aug 15 19:29:22 ansible-demo rq: Giving up _run(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 500, message='Internal Server Error')
Aug 15 19:29:22 ansible-demo rq: pulp: backoff:ERROR: Giving up _run(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 500, message='Internal Server Error')
Aug 15 19:29:22 ansible-demo gunicorn: (self.model._meta.object_name, num)
Aug 15 19:29:22 ansible-demo gunicorn: pulp_ansible.app.models.CollectionVersion.MultipleObjectsReturned: get() returned more than one CollectionVersion -- it returned 2!
Aug 15 19:29:22 ansible-demo gunicorn: 127.0.0.1 - - [15/Aug/2019:19:29:22 +0000] "GET /pulp_ansible/galaxy/foo/api/v2/collections/testing/ansible_testing_content/?page=1 HTTP/1.1" 500 27 "-" "Python/3.6 aiohttp/3.5.4"
Aug 15 19:29:22 ansible-demo rq: pulp: rq.worker:ERROR: aiohttp.client_exceptions.ClientResponseError: 500, message='Internal Server Error'
Aug 15 19:29:22 ansible-demo rq: Traceback (most recent call last):
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 822, in perform_job
Aug 15 19:29:22 ansible-demo rq: rv = job.perform()
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 605, in perform
Aug 15 19:29:22 ansible-demo rq: self._result = self._execute()
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 611, in _execute
Aug 15 19:29:22 ansible-demo rq: return self.func(*self.args, **self.kwargs)
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 66, in sync
Aug 15 19:29:22 ansible-demo rq: d_version.create()
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py", line 169, in create
Aug 15 19:29:22 ansible-demo rq: loop.run_until_complete(pipeline)
Aug 15 19:29:22 ansible-demo rq: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Aug 15 19:29:22 ansible-demo rq: return future.result()
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py", line 209, in create_pipeline
Aug 15 19:29:22 ansible-demo rq: await asyncio.gather(*futures)
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py", line 43, in __call__
Aug 15 19:29:22 ansible-demo rq: await self.run()
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 195, in run
Aug 15 19:29:22 ansible-demo rq: async for metadata in self._fetch_collections():
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 232, in _fetch_collections
Aug 15 19:29:22 ansible-demo rq: async for metadata in self._fetch_galaxy_pages():
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 260, in _fetch_galaxy_pages
Aug 15 19:29:22 ansible-demo rq: metadata = parse_metadata(await downloader.run())
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/download/base.py", line 212, in run
Aug 15 19:29:22 ansible-demo rq: return await self._run(extra_data=extra_data)
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py", line 131, in retry
Aug 15 19:29:22 ansible-demo rq: ret = await target(*args, **kwargs)
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/download/http.py", line 183, in _run
Aug 15 19:29:22 ansible-demo rq: response.raise_for_status()
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client_reqrep.py", line 942, in raise_for_status
Aug 15 19:29:22 ansible-demo rq: headers=self.headers)
Aug 15 19:29:22 ansible-demo rq: aiohttp.client_exceptions.ClientResponseError: 500, message='Internal Server Error'
Aug 15 19:29:22 ansible-demo rq: Traceback (most recent call last):
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 822, in perform_job
Aug 15 19:29:22 ansible-demo rq: rv = job.perform()
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 605, in perform
Aug 15 19:29:22 ansible-demo rq: self._result = self._execute()
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 611, in _execute
Aug 15 19:29:22 ansible-demo rq: return self.func(*self.args, **self.kwargs)
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 66, in sync
Aug 15 19:29:22 ansible-demo rq: d_version.create()
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py", line 169, in create
Aug 15 19:29:22 ansible-demo rq: loop.run_until_complete(pipeline)
Aug 15 19:29:22 ansible-demo rq: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Aug 15 19:29:22 ansible-demo rq: return future.result()
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py", line 209, in create_pipeline
Aug 15 19:29:22 ansible-demo rq: await asyncio.gather(*futures)
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py", line 43, in __call__
Aug 15 19:29:22 ansible-demo rq: await self.run()
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 195, in run
Aug 15 19:29:22 ansible-demo rq: async for metadata in self._fetch_collections():
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 232, in _fetch_collections
Aug 15 19:29:22 ansible-demo rq: async for metadata in self._fetch_galaxy_pages():
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 260, in _fetch_galaxy_pages
Aug 15 19:29:22 ansible-demo rq: metadata = parse_metadata(await downloader.run())
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/download/base.py", line 212, in run
Aug 15 19:29:22 ansible-demo rq: return await self._run(extra_data=extra_data)
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py", line 131, in retry
Aug 15 19:29:22 ansible-demo rq: ret = await target(*args, **kwargs)
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/download/http.py", line 183, in _run
Aug 15 19:29:22 ansible-demo rq: response.raise_for_status()
Aug 15 19:29:22 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client_reqrep.py", line 942, in raise_for_status
Aug 15 19:29:22 ansible-demo rq: headers=self.headers)
Aug 15 19:29:22 ansible-demo rq: aiohttp.client_exceptions.ClientResponseError: 500, message='Internal Server Error'
Aug

This is separate from the error log in the description which is from collection with 1 version.

Actions
Copy link
 #2

Updated by daviddavis over 5 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 57
Actions
Copy link
 #3

Updated by sajha over 5 years ago

Tested the steps with:

 "versions": [
        {
            "component": "pulpcore", 
            "version": "3.0.0rc5.dev0"
        }, 
        {
            "component": "pulpcore-plugin", 
            "version": "0.1.0rc5.dev0"
        }, 
        {
            "component": "pulp_docker", 
            "version": "4.0.0b6.dev0"
        }, 
        {
            "component": "pulp_file", 
            "version": "0.1.0b2.dev0"
        }, 
        {
            "component": "pulp_ansible", 
            "version": "0.2.0b3.dev0"

Initial Repo used to test was https://galaxy-dev.ansible.com/api/v2/collections/testing/ansible_testing_content resulting in download of 2 units.

Aug 20 19:47:13 ansible-demo pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Aug 20 19:47:13 ansible-demo pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[c031c36c-64b7-4a5b-9270-6fcba120ff4c]
Aug 20 19:47:13 ansible-demo pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[0732516f-2bb3-43cb-8301-65f63c8981e8]
Aug 20 19:47:13 ansible-demo pulp: celery.app.trace:INFO: [c031c36c] Task pulp.server.controllers.repository.queue_download_deferred[c031c36c-64b7-4a5b-9270-6fcba120ff4c] succeeded in 0.00802982199821s: None
Aug 20 19:47:13 ansible-demo pulp: celery.app.trace:INFO: [0732516f] Task pulp.server.controllers.repository.download_deferred[0732516f-2bb3-43cb-8301-65f63c8981e8] succeeded in 0.0089367920009s: None
Aug 20 19:47:14 ansible-demo rq: pulp: rq.worker:INFO: resource-manager: pulpcore.tasking.tasks._queue_reserved_task(<function sync at 0x7f67d56f46a8>, '7959a43e-4815-4374-b043-e576911e5ac2', ['/pulp/api/v3/repositories/7a04e1b7-f741-4058-8fe6-5ae5734d45db/', '/pulp/api/v3/remotes/ansible/collection/e6eb7bad-e7e8-4712-9db2-89b980af5894/'], (), {'remote_pk': UUID('e6eb7bad-e7e8-4712-9db2-89b980af5894'), 'repository_pk': UUID('7a04e1b7-f741-4058-8fe6-5ae5734d45db'), 'mirror': False}, {}) (cc86f234-6d14-4aa4-95fd-28441ff53c51)
Aug 20 19:47:14 ansible-demo gunicorn: 127.0.0.1 - admin [20/Aug/2019:19:47:14 +0000] "POST /pulp/api/v3/remotes/ansible/collection/e6eb7bad-e7e8-4712-9db2-89b980af5894/sync/ HTTP/1.1" 202 67 "-" "HTTPie/0.9.4"
Aug 20 19:47:14 ansible-demo rq: pulp: rq.worker:INFO: reserved-resource-worker-1@ansible-demo.samir.example.com: pulp_ansible.app.tasks.collections.sync(mirror=False, remote_pk=UUID('e6eb7bad-e7e8-4712-9db2-89b980af5894'), repository_pk=UUID('7a04e1b7-f741-4058-8fe6-5ae5734d45db')) (7959a43e-4815-4374-b043-e576911e5ac2)
Aug 20 19:47:14 ansible-demo rq: pulp: rq.worker:INFO: resource-manager: Job OK (cc86f234-6d14-4aa4-95fd-28441ff53c51)
Aug 20 19:47:14 ansible-demo gunicorn: 127.0.0.1 - - [20/Aug/2019:19:47:14 +0000] "GET /pulp_ansible/galaxy/foo/api/v2/collections/testing/ansible_testing_content?page=1 HTTP/1.1" 301 0 "-" "Python/3.6 aiohttp/3.5.4"
Aug 20 19:47:14 ansible-demo gunicorn: pulp: django.request:ERROR: Internal Server Error: /pulp_ansible/galaxy/foo/api/v2/collections/testing/ansible_testing_content/
Aug 20 19:47:14 ansible-demo gunicorn: Traceback (most recent call last):
Aug 20 19:47:14 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
Aug 20 19:47:14 ansible-demo gunicorn: response = get_response(request)
Aug 20 19:47:14 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
Aug 20 19:47:14 ansible-demo gunicorn: response = self.process_exception_by_middleware(e, request)
Aug 20 19:47:14 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
Aug 20 19:47:14 ansible-demo gunicorn: response = wrapped_callback(request, *callback_args, **callback_kwargs)
Aug 20 19:47:14 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Aug 20 19:47:14 ansible-demo gunicorn: return view_func(*args, **kwargs)
Aug 20 19:47:14 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/views/generic/base.py", line 71, in view
Aug 20 19:47:14 ansible-demo gunicorn: return self.dispatch(request, *args, **kwargs)
Aug 20 19:47:14 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/views.py", line 497, in dispatch
Aug 20 19:47:14 ansible-demo gunicorn: response = self.handle_exception(exc)
Aug 20 19:47:14 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/views.py", line 457, in handle_exception
Aug 20 19:47:14 ansible-demo gunicorn: self.raise_uncaught_exception(exc)
Aug 20 19:47:14 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/views.py", line 468, in raise_uncaught_exception
Aug 20 19:47:14 ansible-demo gunicorn: raise exc
Aug 20 19:47:14 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/views.py", line 494, in dispatch
Aug 20 19:47:14 ansible-demo gunicorn: response = handler(request, *args, **kwargs)
Aug 20 19:47:14 ansible-demo gunicorn: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/galaxy/views.py", line 114, in get
Aug 20 19:47:14 ansible-demo gunicorn: CollectionVersion, collection__namespace=namespace, collection__name=name
Aug 20 19:47:14 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/shortcuts.py", line 93, in get_object_or_404
Aug 20 19:47:14 ansible-demo gunicorn: return queryset.get(*args, **kwargs)
Aug 20 19:47:14 ansible-demo gunicorn: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/query.py", line 412, in get
Aug 20 19:47:14 ansible-demo gunicorn: (self.model._meta.object_name, num)
Aug 20 19:47:14 ansible-demo gunicorn: pulp_ansible.app.models.CollectionVersion.MultipleObjectsReturned: get() returned more than one CollectionVersion -- it returned 2!
Aug 20 19:47:14 ansible-demo gunicorn: 127.0.0.1 - - [20/Aug/2019:19:47:14 +0000] "GET /pulp_ansible/galaxy/foo/api/v2/collections/testing/ansible_testing_content/?page=1 HTTP/1.1" 500 27 "-" "Python/3.6 aiohttp/3.5.4"
Aug 20 19:47:14 ansible-demo rq: Giving up _run(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 500, message='Internal Server Error')
Aug 20 19:47:14 ansible-demo rq: pulp: backoff:ERROR: Giving up _run(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 500, message='Internal Server Error')
Aug 20 19:47:14 ansible-demo rq: pulp: rq.worker:ERROR: aiohttp.client_exceptions.ClientResponseError: 500, message='Internal Server Error'
Aug 20 19:47:14 ansible-demo rq: Traceback (most recent call last):
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 822, in perform_job
Aug 20 19:47:14 ansible-demo rq: rv = job.perform()
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 605, in perform
Aug 20 19:47:14 ansible-demo rq: self._result = self._execute()
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 611, in _execute
Aug 20 19:47:14 ansible-demo rq: return self.func(*self.args, **self.kwargs)
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 70, in sync
Aug 20 19:47:14 ansible-demo rq: d_version.create()
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py", line 169, in create
Aug 20 19:47:14 ansible-demo rq: loop.run_until_complete(pipeline)
Aug 20 19:47:14 ansible-demo rq: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Aug 20 19:47:14 ansible-demo rq: return future.result()
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py", line 209, in create_pipeline
Aug 20 19:47:14 ansible-demo rq: await asyncio.gather(*futures)
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py", line 43, in __call__
Aug 20 19:47:14 ansible-demo rq: await self.run()
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 200, in run
Aug 20 19:47:14 ansible-demo rq: async for metadata in self._fetch_collections():
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 254, in _fetch_collections
Aug 20 19:47:14 ansible-demo rq: initial_data = parse_metadata(await downloader.run())
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/download/base.py", line 212, in run
Aug 20 19:47:14 ansible-demo rq: return await self._run(extra_data=extra_data)
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py", line 131, in retry
Aug 20 19:47:14 ansible-demo rq: ret = await target(*args, **kwargs)
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/download/http.py", line 183, in _run
Aug 20 19:47:14 ansible-demo rq: response.raise_for_status()
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client_reqrep.py", line 942, in raise_for_status
Aug 20 19:47:14 ansible-demo rq: headers=self.headers)
Aug 20 19:47:14 ansible-demo rq: aiohttp.client_exceptions.ClientResponseError: 500, message='Internal Server Error'
Aug 20 19:47:14 ansible-demo rq: Traceback (most recent call last):
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 822, in perform_job
Aug 20 19:47:14 ansible-demo rq: rv = job.perform()
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 605, in perform
Aug 20 19:47:14 ansible-demo rq: self._result = self._execute()
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 611, in _execute
Aug 20 19:47:14 ansible-demo rq: return self.func(*self.args, **self.kwargs)
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 70, in sync
Aug 20 19:47:14 ansible-demo rq: d_version.create()
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py", line 169, in create
Aug 20 19:47:14 ansible-demo rq: loop.run_until_complete(pipeline)
Aug 20 19:47:14 ansible-demo rq: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Aug 20 19:47:14 ansible-demo rq: return future.result()
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py", line 209, in create_pipeline
Aug 20 19:47:14 ansible-demo rq: await asyncio.gather(*futures)
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py", line 43, in __call__
Aug 20 19:47:14 ansible-demo rq: await self.run()
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 200, in run
Aug 20 19:47:14 ansible-demo rq: async for metadata in self._fetch_collections():
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 254, in _fetch_collections
Aug 20 19:47:14 ansible-demo rq: initial_data = parse_metadata(await downloader.run())
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/download/base.py", line 212, in run
Aug 20 19:47:14 ansible-demo rq: return await self._run(extra_data=extra_data)
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py", line 131, in retry
Aug 20 19:47:14 ansible-demo rq: ret = await target(*args, **kwargs)
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/download/http.py", line 183, in _run
Aug 20 19:47:14 ansible-demo rq: response.raise_for_status()
Aug 20 19:47:14 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client_reqrep.py", line 942, in raise_for_status
Aug 20 19:47:14 ansible-demo rq: headers=self.headers)
Aug 20 19:47:14 ansible-demo rq: aiohttp.client_exceptions.ClientResponseError: 500, message='Internal Server Error'
Aug 20 19:47:14 ansible-demo rq: pulp: rq.worker:INFO: reserved-resource-worker-1@ansible-demo.samir.example.com: pulpcore.tasking.tasks._release_resources('7959a43e-4815-4374-b043-e576911e5ac2') (2993bf1f-86bb-40e1-a4c8-048fb38607fa)
Aug 20 19:47:14 ansible-demo rq: pulp: rq.worker:INFO: reserved-resource-worker-1@ansible-demo.samir.example.com: Job OK (2993bf1f-86bb-40e1-a4c8-048fb38607fa)
Actions
Copy link
 #4

Updated by sajha over 5 years ago

With the master repo synced with https://galaxy.ansible.com/api/v2/collections/devoperate/base, I see the below:

Aug 20 20:03:14 ansible-demo rq: pulp: rq.worker:INFO: resource-manager: pulpcore.tasking.tasks._queue_reserved_task(<function sync at 0x7f67d56f46a8>, 'ede8da6b-5f6a-4b83-809f-740c40916dcb', ['/pulp/api/v3/repositories/c20f8e0a-b5c5-460e-800a-7a798c5a8fd6/', '/pulp/api/v3/remotes/ansible/collection/a80361fe-47f3-4a11-8310-b11656cff7f8/'], (), {'remote_pk': UUID('a80361fe-47f3-4a11-8310-b11656cff7f8'), 'repository_pk': UUID('c20f8e0a-b5c5-460e-800a-7a798c5a8fd6'), 'mirror': False}, {}) (26fd790a-8691-4e43-8e78-7cbb5c1099ba)
Aug 20 20:03:14 ansible-demo gunicorn: 127.0.0.1 - admin [20/Aug/2019:20:03:14 +0000] "POST /pulp/api/v3/remotes/ansible/collection/a80361fe-47f3-4a11-8310-b11656cff7f8/sync/ HTTP/1.1" 202 67 "-" "HTTPie/0.9.4"
Aug 20 20:03:14 ansible-demo rq: pulp: rq.worker:INFO: reserved-resource-worker-2@ansible-demo.samir.example.com: pulp_ansible.app.tasks.collections.sync(mirror=False, remote_pk=UUID('a80361fe-47f3-4a11-8310-b11656cff7f8'), repository_pk=UUID('c20f8e0a-b5c5-460e-800a-7a798c5a8fd6')) (ede8da6b-5f6a-4b83-809f-740c40916dcb)
Aug 20 20:03:14 ansible-demo rq: pulp: rq.worker:INFO: resource-manager: Job OK (26fd790a-8691-4e43-8e78-7cbb5c1099ba)
Aug 20 20:03:14 ansible-demo gunicorn: 127.0.0.1 - - [20/Aug/2019:20:03:14 +0000] "GET /pulp_ansible/galaxy/fooagain/api/v2/collections/devoperate/base?page=1 HTTP/1.1" 301 0 "-" "Python/3.6 aiohttp/3.5.4"
Aug 20 20:03:14 ansible-demo gunicorn: 127.0.0.1 - - [20/Aug/2019:20:03:14 +0000] "GET /pulp_ansible/galaxy/fooagain/api/v2/collections/devoperate/base/?page=1 HTTP/1.1" 200 284 "-" "Python/3.6 aiohttp/3.5.4"
Aug 20 20:03:14 ansible-demo gunicorn: 127.0.0.1 - - [20/Aug/2019:20:03:14 +0000] "GET /pulp_ansible/galaxy/fooagain/api/v2/collections/devoperate/base?page=1 HTTP/1.1" 301 0 "-" "Python/3.6 aiohttp/3.5.4"
Aug 20 20:03:14 ansible-demo gunicorn: 127.0.0.1 - - [20/Aug/2019:20:03:14 +0000] "GET /pulp_ansible/galaxy/fooagain/api/v2/collections/devoperate/base/?page=1 HTTP/1.1" 200 284 "-" "Python/3.6 aiohttp/3.5.4"
Aug 20 20:03:14 ansible-demo gunicorn: 127.0.0.1 - - [20/Aug/2019:20:03:14 +0000] "GET /pulp_ansible/galaxy/fooagain/api/v2/collections/devoperate/base/versions/0.1.0/ HTTP/1.1" 200 85 "-" "Python/3.6 aiohttp/3.5.4"
Aug 20 20:03:14 ansible-demo gunicorn: 127.0.0.1 - - [20/Aug/2019:20:03:14 +0000] "GET /pulp_ansible/galaxy/fooagain/api/v2/collections/devoperate/base/versions/ HTTP/1.1" 200 336 "-" "Python/3.6 aiohttp/3.5.4"
Aug 20 20:03:14 ansible-demo rq: pulp: rq.worker:ERROR: KeyError: 'namespace'
Aug 20 20:03:14 ansible-demo rq: Traceback (most recent call last):
Aug 20 20:03:14 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 822, in perform_job
Aug 20 20:03:14 ansible-demo rq: rv = job.perform()
Aug 20 20:03:14 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 605, in perform
Aug 20 20:03:14 ansible-demo rq: self._result = self._execute()
Aug 20 20:03:14 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 611, in _execute
Aug 20 20:03:14 ansible-demo rq: return self.func(*self.args, **self.kwargs)
Aug 20 20:03:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 70, in sync
Aug 20 20:03:14 ansible-demo rq: d_version.create()
Aug 20 20:03:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py", line 169, in create
Aug 20 20:03:14 ansible-demo rq: loop.run_until_complete(pipeline)
Aug 20 20:03:14 ansible-demo rq: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Aug 20 20:03:14 ansible-demo rq: return future.result()
Aug 20 20:03:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py", line 209, in create_pipeline
Aug 20 20:03:14 ansible-demo rq: await asyncio.gather(*futures)
Aug 20 20:03:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py", line 43, in __call__
Aug 20 20:03:14 ansible-demo rq: await self.run()
Aug 20 20:03:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 205, in run
Aug 20 20:03:14 ansible-demo rq: namespace=metadata["namespace"]["name"],
Aug 20 20:03:14 ansible-demo rq: KeyError: 'namespace'
Aug 20 20:03:14 ansible-demo rq: Traceback (most recent call last):
Aug 20 20:03:14 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 822, in perform_job
Aug 20 20:03:14 ansible-demo rq: rv = job.perform()
Aug 20 20:03:14 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 605, in perform
Aug 20 20:03:14 ansible-demo rq: self._result = self._execute()
Aug 20 20:03:14 ansible-demo rq: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 611, in _execute
Aug 20 20:03:14 ansible-demo rq: return self.func(*self.args, **self.kwargs)
Aug 20 20:03:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 70, in sync
Aug 20 20:03:14 ansible-demo rq: d_version.create()
Aug 20 20:03:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py", line 169, in create
Aug 20 20:03:14 ansible-demo rq: loop.run_until_complete(pipeline)
Aug 20 20:03:14 ansible-demo rq: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Aug 20 20:03:14 ansible-demo rq: return future.result()
Aug 20 20:03:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py", line 209, in create_pipeline
Aug 20 20:03:14 ansible-demo rq: await asyncio.gather(*futures)
Aug 20 20:03:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py", line 43, in __call__
Aug 20 20:03:14 ansible-demo rq: await self.run()
Aug 20 20:03:14 ansible-demo rq: File "/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py", line 205, in run
Aug 20 20:03:14 ansible-demo rq: namespace=metadata["namespace"]["name"],
Aug 20 20:03:14 ansible-demo rq: KeyError: 'namespace'
Aug 20 20:03:15 ansible-demo rq: pulp: rq.worker:INFO: Cleaning registries for queue: reserved-resource-worker-2@ansible-demo.samir.example.com
Aug 20 20:03:15 ansible-demo rq: pulp: rq.worker:INFO: reserved-resource-worker-2@ansible-demo.samir.example.com: pulpcore.tasking.tasks._release_resources('ede8da6b-5f6a-4b83-809f-740c40916dcb') (10cd60e3-ae74-4c74-a136-e355a966bff7)
Aug 20 20:03:15 ansible-demo rq: pulp: rq.worker:INFO: reserved-resource-worker-2@ansible-demo.samir.example.com: Job OK (10cd60e3-ae74-4c74-a136-e355a966bff7)
Actions
Copy link
 #6

Updated by daviddavis over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to daviddavis
Actions
Copy link
 #7

Updated by daviddavis over 5 years ago

  • Tracker changed from Issue to Story
  • Subject changed from Cannot sync repo using pulp's galaxy endpoint as remote to As a user, I can sync a repo using pulp's galaxy endpoint as remote
  • % Done set to 0
Actions
Copy link
 #8

Updated by daviddavis over 5 years ago

  • Subject changed from As a user, I can sync a repo using pulp's galaxy endpoint as remote to As a user, I can sync collections from Pulp
Actions
Copy link
 #9

Updated by daviddavis over 5 years ago

  • Related to Test #5333: As a user, I can sync collections from another Pulp server added
Actions
Copy link
 #10

Updated by daviddavis over 5 years ago

  • Subject changed from As a user, I can sync collections from Pulp to As a user, I can sync collections from another Pulp server

Added by daviddavis over 5 years ago

Revision 80c79b47 | View on GitHub

Enabled pulp-to-pulp syncing of collections

fixes #5288 https://pulp.plan.io/issues/5288

Actions
Copy link
 #11

Updated by daviddavis over 5 years ago

  • Status changed from ASSIGNED to POST
Actions
Copy link
 #12

Updated by rchan over 5 years ago

  • Sprint changed from Sprint 57 to Sprint 58
Actions
Copy link
 #13

Updated by daviddavis over 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions
Copy link
 #14

Updated by daviddavis over 5 years ago

  • Status changed from MODIFIED to ASSIGNED

Added by daviddavis over 5 years ago

Revision 7ee43dfc | View on GitHub

Support the case of syncing all collections pulp-to-pulp

fixes #5288

Actions
Copy link
 #15

Updated by daviddavis over 5 years ago

Additional work was needed to support the case where we can point the remote to http://localhost:24817/pulp_ansible/galaxy/foo/api/v2/collections/ and sync all collections:

https://github.com/pulp/pulp_ansible/pull/194

Actions
Copy link
 #16

Updated by daviddavis over 5 years ago

  • Status changed from ASSIGNED to MODIFIED
Actions
Copy link
 #17

Updated by ggainey over 4 years ago

  • Tags Katello added
  • Tags deleted (Katello-P2)
Actions
Copy link
 #18

Updated by bmbouter about 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions
Send by e-mail Copy link

Also available in: Atom PDF