Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-10-25T15:15:01ZPulp
Planio Ansible Plugin - Issue #9534 (CLOSED - DUPLICATE): Ansible Galaxy Repo failed to synchronize case...https://pulp.plan.io/issues/95342021-10-25T15:15:01Zstosem
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_ansible/747":<a href="https://github.com/pulp/pulp_ansible/issues/747" class="external">https://github.com/pulp/pulp_ansible/issues/747</a></p>
<hr>
<p>The galaxy.ansible.com API uses case-insensitive fields for the owner username and role name. The Ansible Galaxy Hub source code: <a href="https://github.com/ansible/galaxy/blob/e0e4b909171ddc6ca40f0ef2f7d4dce5680777ce/galaxy/api/views/roles.py#L50" class="external">https://github.com/ansible/galaxy/blob/e0e4b909171ddc6ca40f0ef2f7d4dce5680777ce/galaxy/api/views/roles.py#L50</a></p>
<p>But pulp usesnamespace=namespace. It causes the error on case-sensitive repositories (404 not found, or empty set). An example:</p>
<ul>
<li><a href="https://galaxy.ansible.com/api/v1/roles/?owner__username=Oefenweb" class="external">https://galaxy.ansible.com/api/v1/roles/?owner__username=Oefenweb</a></li>
<li><a href="https://galaxy.ansible.com/api/v1/roles/?owner__username=oefenweb" class="external">https://galaxy.ansible.com/api/v1/roles/?owner__username=oefenweb</a></li>
</ul>
<p>Galaxy Ansible returns the same result. But pulp isn't.</p>
<p>So we can't install roles like this using regular <code>ansible-galaxy install oefenweb.postfix</code> command with the Pulp repo.</p> Ansible Plugin - Issue #9220 (CLOSED - CURRENTRELEASE): content_ansible_collection_versions_creat...https://pulp.plan.io/issues/92202021-08-09T17:07:09Zsajha
<p>The <a href="https://docs.pulpproject.org/pulp_ansible/restapi.html#operation/upload_collection" class="external">https://docs.pulpproject.org/pulp_ansible/restapi.html#operation/upload_collection</a> endpoint allows you to upload a Collection tarball. This operation results in an artifact which can then be used to create a new collection content.</p>
<p>However, the endpoint to do so <a href="https://docs.pulpproject.org/pulp_ansible/restapi.html#operation/content_ansible_collection_versions_create" class="external">https://docs.pulpproject.org/pulp_ansible/restapi.html#operation/content_ansible_collection_versions_create</a> seems to have a lot of required fields which are presumably avaialble on the uploaded tarball itself. Is there a way to modify the flow so that associating an artifact from an uploaded collection tarball doesn't need to have all the other information passed.</p>
<p>I am assuming the flow would be to upload tarball, create a collection content using the artifact, add content created to a collection repository.</p> Ansible Plugin - Issue #8957 (CLOSED - CURRENTRELEASE): URL validation error on remote displays r...https://pulp.plan.io/issues/89572021-06-23T18:27:37Zsajha
<p>The url validation is somehow leading to this: ["Invalid URL https://galaxy.ansible.com/. Ensure the URL ends '/'."]} This happens when I have some source specified in the requirements.yml without the trailing "/".. However the error displays the remote URL in the error which happens to have the trailing /..</p>
<p>(02:14:32 PM) rochacbruno: error is on <a href="https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/serializers.py#L166" class="external">https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/serializers.py#L166</a>
(02:14:44 PM) rochacbruno: should be url=url instead of data["url"]</p> Ansible Plugin - Issue #8742 (CLOSED - DUPLICATE): Ansible playbooks fail on database migrationshttps://pulp.plan.io/issues/87422021-05-13T07:09:03Zsli720
<p>I run the ansible playbooks from pulp-installer 3.12.2 (also once with the latest commit of the master branch) on a fresh CentOS 8 as I did last week but suddenly it breaks on database migration with the error:</p>
<pre><code class="text syntaxhl" data-language="text">File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/click/core.py\", line 2108, in __init__", " ) from None", "ValueError: 'default' must be a list when 'multiple' is true."], "stdout": "", "stdout_lines": []}
</code></pre>
<p>Is the click package incompatible? I really don't know now which version of the pulp-installer really works, cause on every rerun of the playbooks another issue appears mostly due to 3rd party incompatibilities (also django upgrade recently broke the installation) . Would it somehow possible to fix all this dependencies somewhere (which file?) cause when I check out 3.12.2 I always expect the same result? Or is there something I do wrong?</p> Ansible Plugin - Issue #8290 (CLOSED - CURRENTRELEASE): Pulp_ansible unable to sync from galaxy e...https://pulp.plan.io/issues/82902021-02-22T18:00:20Zsajha
<p>Steps to reproduce:
Set up a remote with a token and point to a galaxy instance.</p>
<pre><code> {
"auth_url": null,
"ca_cert": null,
"client_cert": null,
"client_key": null,
"download_concurrency": 10,
"name": "hub_1-128797",
"password": null,
"policy": "immediate",
"proxy_url": null,
"pulp_created": "2021-02-19T19:05:47.231796Z",
"pulp_href": "/pulp/api/v3/remotes/ansible/collection/91c20a7d-8477-48d8-86df-46755cdffcae/",
"pulp_last_updated": "2021-02-22T16:53:00.377091Z",
"requirements_file": "---\ncollections:\n- ibm.cloudcollection",
"tls_validation": false,
"token": "xxx-valid-token-xxx",
"url": "https://hostname/api/galaxy/content/community/",
"username": null
},
</code></pre>
<p>When syncing this, I get a 403 from the endpoint with the following logs:</p>
<pre><code>Feb 22 17:13:53 hub-sat1-v4-2-1-jhardy gunicorn[74032]: pulp: django.request:WARNING: Forbidden: /api/galaxy/content/community/
Feb 22 17:13:53 hub-sat1-v4-2-1-jhardy gunicorn[74032]: pulp [-]: - - [22/Feb/2021:17:13:53 +0000] "GET /api/galaxy/content/community/ HTTP/1.0" 403 112 "-" "pulpcore/3.7.3 (cpython 3.6.8-final0, Linux x86_64) (aiohttp 3.7.2)"
Feb 22 17:13:54 hub-sat1-v4-2-1-jhardy gunicorn[74032]: pulp: django.request:WARNING: Forbidden: /api/galaxy/content/community/api/
Feb 22 17:13:54 hub-sat1-v4-2-1-jhardy gunicorn[74032]: pulp [-]: - - [22/Feb/2021:17:13:54 +0000] "GET /api/galaxy/content/community/api/ HTTP/1.0" 403 112 "-" "pulpcore/3.7.3 (cpython 3.6.8-final0, Linux x86_64) (aiohttp 3.7.2)"
</code></pre>
<p>PS: "tls_validation": false is set on the remote.</p>
<p>A ansible-galaxy client with the below cfg file:</p>
<p>[galaxy]
server_list = community_repo</p>
<p>[galaxy_server.community_repo]
url=<a href="https://hostname/api/galaxy/content/community/" class="external">https://hostname/api/galaxy/content/community/</a>
token=xxx-valid-token-xxx</p>
<p>gives the following results:</p>
<p>Secure:</p>
<pre><code># ansible-galaxy collection install -p ./ ibm.cloudcollection
Process install dependency map
ERROR! Unknown error when attempting to call Galaxy at 'https://hub-sat1-v4-2-1-jhardy.c.ansible-tower-engineering.internal/api/galaxy/content/community/api': <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)>
</code></pre>
<p>Ignore SSL certificate validation errors with -c :</p>
<pre><code># ansible-galaxy collection install -c -p ./ ibm.cloudcollection
Process install dependency map
Starting collection install process
Installing 'ibm.cloudcollection:1.21.0' to '/root/ansible_collections/ibm/cloudcollection'
</code></pre>
<p>Expected result:
Pulp_Ansible should be able to ignore cert validations and sync successfully when tls_validation is off.</p> Ansible Plugin - Story #5288 (CLOSED - CURRENTRELEASE): As a user, I can sync collections from an...https://pulp.plan.io/issues/52882019-08-15T20:02:49Zsajha
<p>This is from before the requirements.yaml PR went in..</p>
<p><strong>STEPS TO REPRODUCE:</strong></p>
<pre><code>#Create repo
http POST $BASE_ADDR/pulp/api/v3/repositories/ name=foo
export REPO_HREF=$(http $BASE_ADDR/pulp/api/v3/repositories/ | \
jq -r '.results[] | select(.name == "foo") | ._href')
#Create remote
http POST $BASE_ADDR/pulp/api/v3/remotes/ansible/collection/ name='bar' url='https://galaxy-dev.ansible.com/api/v2/collections/testing/ansible_testing_content'
export REMOTE_HREF=$(http $BASE_ADDR/pulp/api/v3/remotes/ansible/collection/ | jq -r '.results[] | select(.name == "bar") | ._href')
#Sync repo with Remote
export TASK_URL=$(http POST $BASE_ADDR$REMOTE_HREF'sync/' repository=$REPO_HREF \
| jq -r '.task')
export REPOVERSION_HREF=$(http $BASE_ADDR$TASK_URL | jq -r '.created_resources[]')
#Create Distribution for the synced Repo
export TASK_URL=$(http POST $BASE_ADDR/pulp/api/v3/distributions/ansible/ansible/ \
name='baz' \
base_path='foo' \
repository=${REPO_HREF} | jq -r '.task')
export DIST_PATH=$(http $BASE_ADDR$TASK_URL | jq -r '.created_resources[]')
#Form SYNC_URL
export GALAXY_URL=$(http $BASE_ADDR$DIST_PATH | jq -r '.mazer_url')
export SYNC_URL=$GALAXY_URL'/api/v2/collections/testing/ansible_testing_content'
http $SYNC_URL
# Notice Error in /var/log/messages around multiple collection versions being returned.
http $SYNC_URL'/versions'
#-----------------Try to use $SYNC_URL as remote for 2nd repo----------
#Create 2nd repo to sync from the sync url
export REPO_HREF=$(http $BASE_ADDR/pulp/api/v3/repositories/ | \
jq -r '.results[] | select(.name == "foo1") | ._href')
#Create remote with url pointing to pulp's galaxy endpoint of 1st repo
http POST $BASE_ADDR/pulp/api/v3/remotes/ansible/collection/ name='bar1' url=$SYNC_URL
export REMOTE_HREF=$(http $BASE_ADDR/pulp/api/v3/remotes/ansible/collection/ | jq -r '.results[] | select(.name == "bar1") | ._href')
#Notice url in the Remote
http $BASE_HREF$REMOTE_HREF
#Sync repo2 with remote poinitng to 1st repos pulp-galaxy url
export TASK_URL=$(http POST $BASE_ADDR$REMOTE_HREF'sync/' repository=$REPO_HREF \
| jq -r '.task')
#Notice failure in /var/log/messages
http $BASE_HREF$TASK_URL
#http://localhost:24817/pulp_ansible/galaxy/foo/api/v2/collections/testing/ansible_testing_content/versions/4.0.4/ as sync URL fails too
</code></pre>
<p>Tried this with a collection <a href="https://galaxy.ansible.com/api/v2/collections/devoperate/base/versions/" class="external">https://galaxy.ansible.com/api/v2/collections/devoperate/base/versions/</a> having only one version. Got the error:</p>
<pre><code>"error": {
"code": null,
"description": "string indices must be integers",
"traceback": " File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 822, in perform_job\n rv = job.perform()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 605, in perform\n self._result = self._execute()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 611, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py\", line 66, in sync\n d_version.create()\n File \"/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py\", line 169, in create\n loop.run_until_complete(pipeline)\n File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n return future.result()\n File \"/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py\", line 209, in create_pipeline\n await asyncio.gather(*futures)\n File \"/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py\", line 195, in run\n async for metadata in self._fetch_collections():\n File \"/usr/local/lib/pulp/src/pulp-ansible/pulp_ansible/app/tasks/collections.py\", line 242, in _fetch_collections\n \"namespace\": result[\"namespace\"][\"name\"],\n"
},
</code></pre> Ansible Plugin - Story #5251 (CLOSED - DUPLICATE): Dependency Downloading for Ansible Collection ...https://pulp.plan.io/issues/52512019-08-12T16:30:40Zsajha
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_ansible/685":<a href="https://github.com/pulp/pulp_ansible/issues/685" class="external">https://github.com/pulp/pulp_ansible/issues/685</a></p>
<hr>
<a name="Dependency-Downloading-when-syncing-collections"></a>
<h3 >Dependency Downloading when syncing collections:<a href="#Dependency-Downloading-when-syncing-collections" class="wiki-anchor">¶</a></h3>
<p>When syncing a collection, we would like to have the collections listed as dependencies to be synced into the repository as well.<br>
Example: <a href="https://docs.ansible.com/ansible/devel/dev_guide/collections_galaxy_meta.html#examples" class="external">https://docs.ansible.com/ansible/devel/dev_guide/collections_galaxy_meta.html#examples</a></p>
<p>For a collection to be synced, having the below dependencies:</p>
<pre><code>dependencies:
"other_namespace.collection1": ">=1.0.0"
"other_namespace.collection2": ">=2.0.0,<3.0.0"
"anderson55.my_collection": "*" # note: "*" selects the highest version available
</code></pre>
<p>The 3 listed collections should also get synced when syncing the collection.</p> Ansible Plugin - Story #5250 (CLOSED - CURRENTRELEASE): Add requirements.yaml specification suppo...https://pulp.plan.io/issues/52502019-08-12T16:22:23Zsajha
<a name="Requirements-with-requirementsyml"></a>
<h3 >Requirements with requirements.yml:<a href="#Requirements-with-requirementsyml" class="wiki-anchor">¶</a></h3>
<p>We would like to have the ability to use the requirements.yml documented here: <a href="https://docs.ansible.com/ansible/devel/dev_guide/collections_tech_preview.html#install-multiple-collections-with-a-requirements-file" class="external">https://docs.ansible.com/ansible/devel/dev_guide/collections_tech_preview.html#install-multiple-collections-with-a-requirements-file</a></p>
<pre><code>---
collections:
# With just the collection name
- my_namespace.my_collection
# With the collection name, version, and source options
- name: my_namespace.my_other_collection
version: 'version range identifiers (default: ``*``)'
source: 'The Galaxy URL to pull the collection from (default: ``--api-server`` from cmdline)'
</code></pre>
<p>This would sync the collection my_namespace.my_collection and the specified version of my_namespace.my_other_collection. The version specification would be similar to <a href="https://docs.ansible.com/ansible/devel/dev_guide/collections_tech_preview.html#installing-an-older-version-of-a-collection" class="external">https://docs.ansible.com/ansible/devel/dev_guide/collections_tech_preview.html#installing-an-older-version-of-a-collection</a></p>
<a name="Field-Name"></a>
<h3 >Field Name<a href="#Field-Name" class="wiki-anchor">¶</a></h3>
<p>The field on the CollectionRemote should be called <code>requirements_file</code>.</p>
<a name="Validation-of-data-Requirement"></a>
<h3 >Validation of data Requirement<a href="#Validation-of-data-Requirement" class="wiki-anchor">¶</a></h3>
<p>Note that Pulp expects to have all content from a single source and does not expect to have sources defined per content item (as the "source" field allows in this spec). As such, Pulp should not accept a requirements.yaml that contains a definition for "source". If specified when creating/saving a remote it should be rejected as invalid with a message from the user.</p> Ansible Plugin - Issue #5167 (CLOSED - CURRENTRELEASE): Collection Remote sync does not accept mi...https://pulp.plan.io/issues/51672019-07-22T17:33:18Zsajha
<p>The sync operation with a collection remote doesn't honor mirror parameter. Sync succeeds but the contents not present upstream are not removed from the repo version.</p>
<pre><code>Calling API: RemotesCollectionApi.sync ...
HTTP request body param ~BEGIN~
{"repository":"/pulp/api/v3/repositories/88e8ca8c-2da6-4407-81b1-832df39d277c/","mirror":true}
~END~
</code></pre> Ansible Plugin - Story #5023 (CLOSED - CURRENTRELEASE): Ansible Collection sync does not have dat...https://pulp.plan.io/issues/50232019-06-24T19:44:50Zsajha
<p>The progress report in the collection sync task is empty. We should ideally receive data in a structure similar to files. <a href="https://docs.pulpproject.org/en/3.0/nightly/restapi.html#operation/tasks_read" class="external">https://docs.pulpproject.org/en/3.0/nightly/restapi.html#operation/tasks_read</a></p> Ansible Plugin - Story #5018 (CLOSED - CURRENTRELEASE): Add artifact sha details to the content a...https://pulp.plan.io/issues/50182019-06-24T19:39:05Zsajha
<p>It would be helpful to see artifact sha details on the content details/list results to avoid an additional call to artifacts API to fetch sha details.<br>
This endpoint should ideally provide sha details in results: <a href="https://pulp-ansible.readthedocs.io/en/latest/restapi.html#operation/content_ansible_collections_list" class="external">https://pulp-ansible.readthedocs.io/en/latest/restapi.html#operation/content_ansible_collections_list</a></p> Ansible Plugin - Issue #5001 (CLOSED - CURRENTRELEASE): Provide content_ansible_collection_read m...https://pulp.plan.io/issues/50012019-06-20T18:37:10Zsajha
<p>We need the ability to read ansible collection content similar to other content types ex: <a href="https://pulp-ansible.readthedocs.io/en/latest/restapi.html#operation/content_ansible_roles_read" class="external">https://pulp-ansible.readthedocs.io/en/latest/restapi.html#operation/content_ansible_roles_read</a>.</p> Ansible Plugin - Issue #4997 (CLOSED - CURRENTRELEASE): Pulp3 Ansible Collection remote doesn't s...https://pulp.plan.io/issues/49972019-06-20T16:19:01Zsajha
<p>*Steps to Reproduce:*++<br>
Steps based on httpie and jq from documentation: <a href="https://pulp-ansible.readthedocs.io/en/latest/workflows/index.html" class="external">https://pulp-ansible.readthedocs.io/en/latest/workflows/index.html</a></p>
<p>#Create 2 separate repos:<br>
http POST $BASE_ADDR/pulp/api/v3/repositories/ name=test1<br>
http POST $BASE_ADDR/pulp/api/v3/repositories/ name=test2</p>
<p>export REPO_HREF1=$(http $BASE_ADDR/pulp/api/v3/repositories/ | \<br>
jq -r '.results[] | select(.name == "test1") | ._href')</p>
<p>export REPO_HREF2=$(http $BASE_ADDR/pulp/api/v3/repositories/ | \<br>
jq -r '.results[] | select(.name == "test2") | ._href')</p>
<p>#Create collection remote:</p>
<p>http POST $BASE_ADDR/pulp/api/v3/remotes/ansible/collection/ \<br>
name='bar' \<br>
url='<a href="https://galaxy-dev.ansible.com" class="external">https://galaxy-dev.ansible.com</a>' \<br>
whitelist='newswangerd.collection_demo'</p>
<p>export REMOTE_HREF=$(http $BASE_ADDR/pulp/api/v3/remotes/ansible/collection/ | jq -r '.results[] | select(.name == "bar") | ._href')</p>
<p>#Sync both repos with the remote</p>
<p>export TASK_URL1=$(http POST $BASE_ADDR$REMOTE_HREF'sync/' repository=$REPO_HREF1 \<br>
| jq -r '.task')</p>
<p>export TASK_URL2=$(http POST $BASE_ADDR$REMOTE_HREF'sync/' repository=$REPO_HREF2 \<br>
| jq -r '.task')</p>
<p>#Notice that a new repo version is created by 1st task but none by the 2nd.</p>
<p>http $BASE_ADDR$TASK_URL1</p>
<p>http $BASE_ADDR$TASK_URL2</p>
<p>*<strong>*</strong>*********************************************************</p>
<p>I tried with a second remote pointing to the same URL and with the same whitelist and it doesn't seem to create a repo version either.</p> Ansible Plugin - Issue #4996 (CLOSED - CURRENTRELEASE): Ansible collection remote doesn't handle ...https://pulp.plan.io/issues/49962019-06-19T20:06:58Zsajha
<p>Ex:<br>
With a url <a href="https://galaxy.ansible.com" class="external">https://galaxy.ansible.com</a> it works fine.<br>
With a url <a href="https://galaxy.ansible.com/" class="external">https://galaxy.ansible.com/</a> , throws the following error: <a href="https://gist.github.com/sjha4/0afdd783bac7cea56055b706f6b02155" class="external">https://gist.github.com/sjha4/0afdd783bac7cea56055b706f6b02155</a></p> Ansible Plugin - Issue #3898 (CLOSED - CURRENTRELEASE): Filters inherit from a Django's FilterSet...https://pulp.plan.io/issues/38982018-08-02T15:13:08Zvdusek
<p>AnsibleRoleFilter and AnsibleRoleVersionFilter inherit from Django's filterset.FilterSet class not from our BaseFilterSet. I suppose this could cause problems in generating documentation.</p>