Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-11-09T13:48:32ZPulp
Planio Ansible Plugin - Issue #9561 (CLOSED - DUPLICATE): Update galaxy-importer requirementshttps://pulp.plan.io/issues/95612021-11-09T13:48:32Znewswangerd
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_ansible/749":<a href="https://github.com/pulp/pulp_ansible/issues/749" class="external">https://github.com/pulp/pulp_ansible/issues/749</a></p>
<hr>
<p>Pulp ansible currently depends on galaxy-importer>=0.3.1, but relies on a function that's only available in 0.4.1: <a href="https://github.com/pulp/pulp_ansible/commit/db592b44ed367c3c1866456d012a24886d9c55a5#diff-ee4bd390338ab9adff17b9b09f05e8dc56e09a3964633c104402998971723bffR6" class="external">https://github.com/pulp/pulp_ansible/commit/db592b44ed367c3c1866456d012a24886d9c55a5#diff-ee4bd390338ab9adff17b9b09f05e8dc56e09a3964633c104402998971723bffR6</a></p> Ansible Plugin - Issue #9546 (CLOSED - DUPLICATE): When syncing a collection version, all its sig...https://pulp.plan.io/issues/95462021-10-27T18:51:23Zrochacbruno
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_ansible/748":<a href="https://github.com/pulp/pulp_ansible/issues/748" class="external">https://github.com/pulp/pulp_ansible/issues/748</a></p>
<hr>
<p>Related to <a href="https://pulp.plan.io/issues/9543" class="external">https://pulp.plan.io/issues/9543</a></p>
<p>Story:</p>
<ul>
<li>All collections are signed on the host system</li>
<li>When syncing to another instance, together with all the artifacts/versions, signatures will also need to be synced</li>
<li>If for example a signature is corrupted on the source system, user may redo the signature, invalidate the current, do signature rotation and then the clients will need to re-sync to fetch new valid signatures</li>
</ul> Ansible Plugin - Issue #9534 (CLOSED - DUPLICATE): Ansible Galaxy Repo failed to synchronize case...https://pulp.plan.io/issues/95342021-10-25T15:15:01Zstosem
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_ansible/747":<a href="https://github.com/pulp/pulp_ansible/issues/747" class="external">https://github.com/pulp/pulp_ansible/issues/747</a></p>
<hr>
<p>The galaxy.ansible.com API uses case-insensitive fields for the owner username and role name. The Ansible Galaxy Hub source code: <a href="https://github.com/ansible/galaxy/blob/e0e4b909171ddc6ca40f0ef2f7d4dce5680777ce/galaxy/api/views/roles.py#L50" class="external">https://github.com/ansible/galaxy/blob/e0e4b909171ddc6ca40f0ef2f7d4dce5680777ce/galaxy/api/views/roles.py#L50</a></p>
<p>But pulp usesnamespace=namespace. It causes the error on case-sensitive repositories (404 not found, or empty set). An example:</p>
<ul>
<li><a href="https://galaxy.ansible.com/api/v1/roles/?owner__username=Oefenweb" class="external">https://galaxy.ansible.com/api/v1/roles/?owner__username=Oefenweb</a></li>
<li><a href="https://galaxy.ansible.com/api/v1/roles/?owner__username=oefenweb" class="external">https://galaxy.ansible.com/api/v1/roles/?owner__username=oefenweb</a></li>
</ul>
<p>Galaxy Ansible returns the same result. But pulp isn't.</p>
<p>So we can't install roles like this using regular <code>ansible-galaxy install oefenweb.postfix</code> command with the Pulp repo.</p> Ansible Plugin - Issue #9390 (CLOSED - CURRENTRELEASE): Proxy authentication is not working in co...https://pulp.plan.io/issues/93902021-09-13T18:42:17Znewswangerd
<p>We're seeing issues with customers getting <code>aiohttp.client_exceptions.ClientHttpProxyError: 407, message='Proxy Authentication Required'</code> error on Automation Hub 4.2 using pulpcore 3.7.5 and pulp_ansible 0.5.8 during collection sync.</p>
<p>This appears to be happening with Squid.</p>
<p>Squid logs:</p>
<pre><code>1627559170.932 0 xxx.xxx.xxx.xxx TCP_DENIED/407 4125 CONNECT galaxy.ansible.com:443 - HIER_NONE/- text/html
</code></pre> Ansible Plugin - Issue #9053 (CLOSED - DUPLICATE): Implement namespace permission endpointshttps://pulp.plan.io/issues/90532021-07-08T20:11:08Znewswangerd
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_ansible/741":<a href="https://github.com/pulp/pulp_ansible/issues/741" class="external">https://github.com/pulp/pulp_ansible/issues/741</a></p>
<hr>
<p>The current namespaces endpoint embeds permissions as a nested object which is hard to use, error prone and not very scalable.</p>
<p>Rather than embedded updating permissions for objects into the object itself, like the old v3 and <code>_ui</code> apis, object level permissions will be set using <code><object_url>/groups</code> and <code><object_url>/users</code>. These two viewsets will accept PUT operations for setting and updating a list of permissions on the given object and a DELETE operation for removing a list of permissions from a given object.</p>
<p><a href="https://petstore.swagger.io/?url=https://raw.githubusercontent.com/newswangerd/galaxy-api-v4/master/refactored_v3.yaml#/Pulp%20Ansible/get-plugins-ansible-namespaces-name-users" class="external">API Spec</a></p> Ansible Plugin - Issue #8742 (CLOSED - DUPLICATE): Ansible playbooks fail on database migrationshttps://pulp.plan.io/issues/87422021-05-13T07:09:03Zsli720
<p>I run the ansible playbooks from pulp-installer 3.12.2 (also once with the latest commit of the master branch) on a fresh CentOS 8 as I did last week but suddenly it breaks on database migration with the error:</p>
<pre><code class="text syntaxhl" data-language="text">File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/click/core.py\", line 2108, in __init__", " ) from None", "ValueError: 'default' must be a list when 'multiple' is true."], "stdout": "", "stdout_lines": []}
</code></pre>
<p>Is the click package incompatible? I really don't know now which version of the pulp-installer really works, cause on every rerun of the playbooks another issue appears mostly due to 3rd party incompatibilities (also django upgrade recently broke the installation) . Would it somehow possible to fix all this dependencies somewhere (which file?) cause when I check out 3.12.2 I always expect the same result? Or is there something I do wrong?</p> Ansible Plugin - Issue #8290 (CLOSED - CURRENTRELEASE): Pulp_ansible unable to sync from galaxy e...https://pulp.plan.io/issues/82902021-02-22T18:00:20Zsajha
<p>Steps to reproduce:
Set up a remote with a token and point to a galaxy instance.</p>
<pre><code> {
"auth_url": null,
"ca_cert": null,
"client_cert": null,
"client_key": null,
"download_concurrency": 10,
"name": "hub_1-128797",
"password": null,
"policy": "immediate",
"proxy_url": null,
"pulp_created": "2021-02-19T19:05:47.231796Z",
"pulp_href": "/pulp/api/v3/remotes/ansible/collection/91c20a7d-8477-48d8-86df-46755cdffcae/",
"pulp_last_updated": "2021-02-22T16:53:00.377091Z",
"requirements_file": "---\ncollections:\n- ibm.cloudcollection",
"tls_validation": false,
"token": "xxx-valid-token-xxx",
"url": "https://hostname/api/galaxy/content/community/",
"username": null
},
</code></pre>
<p>When syncing this, I get a 403 from the endpoint with the following logs:</p>
<pre><code>Feb 22 17:13:53 hub-sat1-v4-2-1-jhardy gunicorn[74032]: pulp: django.request:WARNING: Forbidden: /api/galaxy/content/community/
Feb 22 17:13:53 hub-sat1-v4-2-1-jhardy gunicorn[74032]: pulp [-]: - - [22/Feb/2021:17:13:53 +0000] "GET /api/galaxy/content/community/ HTTP/1.0" 403 112 "-" "pulpcore/3.7.3 (cpython 3.6.8-final0, Linux x86_64) (aiohttp 3.7.2)"
Feb 22 17:13:54 hub-sat1-v4-2-1-jhardy gunicorn[74032]: pulp: django.request:WARNING: Forbidden: /api/galaxy/content/community/api/
Feb 22 17:13:54 hub-sat1-v4-2-1-jhardy gunicorn[74032]: pulp [-]: - - [22/Feb/2021:17:13:54 +0000] "GET /api/galaxy/content/community/api/ HTTP/1.0" 403 112 "-" "pulpcore/3.7.3 (cpython 3.6.8-final0, Linux x86_64) (aiohttp 3.7.2)"
</code></pre>
<p>PS: "tls_validation": false is set on the remote.</p>
<p>A ansible-galaxy client with the below cfg file:</p>
<p>[galaxy]
server_list = community_repo</p>
<p>[galaxy_server.community_repo]
url=<a href="https://hostname/api/galaxy/content/community/" class="external">https://hostname/api/galaxy/content/community/</a>
token=xxx-valid-token-xxx</p>
<p>gives the following results:</p>
<p>Secure:</p>
<pre><code># ansible-galaxy collection install -p ./ ibm.cloudcollection
Process install dependency map
ERROR! Unknown error when attempting to call Galaxy at 'https://hub-sat1-v4-2-1-jhardy.c.ansible-tower-engineering.internal/api/galaxy/content/community/api': <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)>
</code></pre>
<p>Ignore SSL certificate validation errors with -c :</p>
<pre><code># ansible-galaxy collection install -c -p ./ ibm.cloudcollection
Process install dependency map
Starting collection install process
Installing 'ibm.cloudcollection:1.21.0' to '/root/ansible_collections/ibm/cloudcollection'
</code></pre>
<p>Expected result:
Pulp_Ansible should be able to ignore cert validations and sync successfully when tls_validation is off.</p> Ansible Plugin - Issue #7564 (CLOSED - DUPLICATE): Excess number of suboptimal queries when query...https://pulp.plan.io/issues/75642020-09-24T12:55:44Zosapryki
<p>Request to <code>/pulp/api/v3/content/ansible/collection_versions</code> (see Request) produces <strong>6 + 2N</strong> SQL queries, where N is a number of collection version in response.</p>
<p>This is highly inefficient and may result in a high request latency in production environments. For Automation Hub it results in <strong>1.8-4.2</strong> seconds per request.</p>
<p>Queries list includes 2 queries that runs <code>SELECT DISTINCT</code> over all fields and includes explicit list of UUIDs of all certified collections in the <code>IN</code> clause.</p>
<a name="Request"></a>
<h2 >Request:<a href="#Request" class="wiki-anchor">¶</a></h2>
<p><code>GET /pulp/api/v3/content/ansible/collection_versions/?is_highest=True&offset=0&limit=12&exclude_fields=docs_blob&certification=certified&deprecated=false</code></p>
<a name="Queries"></a>
<h2 >Queries:<a href="#Queries" class="wiki-anchor">¶</a></h2>
<ol>
<li>Queries list:</li>
</ol>
<p><a href="https://gist.github.com/cutwater/70f6adbe68d88cec0df14141d508f5cf" class="external">https://gist.github.com/cutwater/70f6adbe68d88cec0df14141d508f5cf</a></p>
<ol start="2">
<li>Explain analyze of query executed against the production database:</li>
</ol>
<p>To be added later in comments.</p>
<a name="Versions-tested"></a>
<h2 >Versions tested:<a href="#Versions-tested" class="wiki-anchor">¶</a></h2>
<ol>
<li><code>pulpcore==3.5.0 + pulp-ansible==0.2.0b15</code></li>
<li><code>pulpcore==3.7.0 + pulp-ansible==0.4.0</code></li>
</ol>
<a name="Actions-required"></a>
<h2 >Actions required:<a href="#Actions-required" class="wiki-anchor">¶</a></h2>
<ol>
<li>Eliminate 2N queries.</li>
<li>Optimize <code>SELECT DISTINCT</code> or remove.</li>
<li>Remove explicit list of all certified collections in the <code>IN</code> clause. Replace with a subquery if needed.</li>
</ol> Ansible Plugin - Issue #5711 (CLOSED - CURRENTRELEASE): Unnecessary DELETE action for set_certifi...https://pulp.plan.io/issues/57112019-11-14T11:36:24Zromanblanco
<p>The <code>DELETE</code> action has become unused after merging <a href="https://github.com/ansible/galaxy-api/pull/111/files#diff-54ac93fd48d6c179a17e04d0a104efedL146" class="external">https://github.com/ansible/galaxy-api/pull/111</a>,<br>
making it unnecessary to keep in pulp_ansible repository.</p>
<p>Removing the action is done in Pull Request <a href="https://github.com/pulp/pulp_ansible/pull/247" class="external">https://github.com/pulp/pulp_ansible/pull/247</a>.</p> Ansible Plugin - Issue #5674 (CLOSED - DUPLICATE): All import tasks are assigned to the same workerhttps://pulp.plan.io/issues/56742019-11-06T17:33:35Zosapryki
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_ansible/700":<a href="https://github.com/pulp/pulp_ansible/issues/700" class="external">https://github.com/pulp/pulp_ansible/issues/700</a></p>
<hr>
<p>Task `import_collection` now uses resource reservation. It reserves two resources: an artifact and a related repository.<br>
When scheduling a task, the resource manager assigns task that has a resource reservations on a worker that runs a task with any of requested reservations. This causes the following side effect:</p>
<p>Because repository is a global object, all tasks related to that repository are always run sequentially due to being assigned on the same worker.</p> Ansible Plugin - Issue #5365 (CLOSED - CURRENTRELEASE): Collection version API should include "id...https://pulp.plan.io/issues/53652019-08-28T18:51:08Zosapryki
<p>To be able to proxy pulp API endpoints to client it is required to construct valid proxy URLs.<br>
Without "id" field it would require parsing of URLs returned by pulp.</p>
<p>To simplify this workflow "id" field will be added to the CollectionVersionSerializer.</p> Ansible Plugin - Issue #5346 (CLOSED - DUPLICATE): Wrong url resolution when path is the last com...https://pulp.plan.io/issues/53462019-08-26T13:00:55Zosapryki
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_ansible/688":<a href="https://github.com/pulp/pulp_ansible/issues/688" class="external">https://github.com/pulp/pulp_ansible/issues/688</a></p>
<hr>
<p>When <code><path:path></code> component is the last one in <code>GALAXY_API_ROOT</code>, Django redirects all requests having this prefix to the <code>GalaxyVersionView</code>, which is defined in <code>pulp_ansible/app/urls.py</code> as:</p>
<pre><code class="python syntaxhl" data-language="python"><span class="n">path</span><span class="p">(</span><span class="n">GALAXY_API_ROOT</span><span class="p">,</span> <span class="n">GalaxyVersionView</span><span class="p">.</span><span class="n">as_view</span><span class="p">()),</span>
</code></pre> Ansible Plugin - Issue #5310 (CLOSED - DUPLICATE): Full text search trigger is not executed on in...https://pulp.plan.io/issues/53102019-08-20T20:20:12Zosapryki
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_ansible/687":<a href="https://github.com/pulp/pulp_ansible/issues/687" class="external">https://github.com/pulp/pulp_ansible/issues/687</a></p>
<hr>
<p>A trigger that updates full text search vector introduced in migration <code>0004_add_fulltext_search_indexes.py</code> is not executed on INSERT.</p> Ansible Plugin - Story #5251 (CLOSED - DUPLICATE): Dependency Downloading for Ansible Collection ...https://pulp.plan.io/issues/52512019-08-12T16:30:40Zsajha
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_ansible/685":<a href="https://github.com/pulp/pulp_ansible/issues/685" class="external">https://github.com/pulp/pulp_ansible/issues/685</a></p>
<hr>
<a name="Dependency-Downloading-when-syncing-collections"></a>
<h3 >Dependency Downloading when syncing collections:<a href="#Dependency-Downloading-when-syncing-collections" class="wiki-anchor">¶</a></h3>
<p>When syncing a collection, we would like to have the collections listed as dependencies to be synced into the repository as well.<br>
Example: <a href="https://docs.ansible.com/ansible/devel/dev_guide/collections_galaxy_meta.html#examples" class="external">https://docs.ansible.com/ansible/devel/dev_guide/collections_galaxy_meta.html#examples</a></p>
<p>For a collection to be synced, having the below dependencies:</p>
<pre><code>dependencies:
"other_namespace.collection1": ">=1.0.0"
"other_namespace.collection2": ">=2.0.0,<3.0.0"
"anderson55.my_collection": "*" # note: "*" selects the highest version available
</code></pre>
<p>The 3 listed collections should also get synced when syncing the collection.</p> Ansible Plugin - Story #5250 (CLOSED - CURRENTRELEASE): Add requirements.yaml specification suppo...https://pulp.plan.io/issues/52502019-08-12T16:22:23Zsajha
<a name="Requirements-with-requirementsyml"></a>
<h3 >Requirements with requirements.yml:<a href="#Requirements-with-requirementsyml" class="wiki-anchor">¶</a></h3>
<p>We would like to have the ability to use the requirements.yml documented here: <a href="https://docs.ansible.com/ansible/devel/dev_guide/collections_tech_preview.html#install-multiple-collections-with-a-requirements-file" class="external">https://docs.ansible.com/ansible/devel/dev_guide/collections_tech_preview.html#install-multiple-collections-with-a-requirements-file</a></p>
<pre><code>---
collections:
# With just the collection name
- my_namespace.my_collection
# With the collection name, version, and source options
- name: my_namespace.my_other_collection
version: 'version range identifiers (default: ``*``)'
source: 'The Galaxy URL to pull the collection from (default: ``--api-server`` from cmdline)'
</code></pre>
<p>This would sync the collection my_namespace.my_collection and the specified version of my_namespace.my_other_collection. The version specification would be similar to <a href="https://docs.ansible.com/ansible/devel/dev_guide/collections_tech_preview.html#installing-an-older-version-of-a-collection" class="external">https://docs.ansible.com/ansible/devel/dev_guide/collections_tech_preview.html#installing-an-older-version-of-a-collection</a></p>
<a name="Field-Name"></a>
<h3 >Field Name<a href="#Field-Name" class="wiki-anchor">¶</a></h3>
<p>The field on the CollectionRemote should be called <code>requirements_file</code>.</p>
<a name="Validation-of-data-Requirement"></a>
<h3 >Validation of data Requirement<a href="#Validation-of-data-Requirement" class="wiki-anchor">¶</a></h3>
<p>Note that Pulp expects to have all content from a single source and does not expect to have sources defined per content item (as the "source" field allows in this spec). As such, Pulp should not accept a requirements.yaml that contains a definition for "source". If specified when creating/saving a remote it should be rejected as invalid with a message from the user.</p>