Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-10-20T17:27:32ZPulp
Planio Ansible Plugin - Issue #9525 (CLOSED - WORKSFORME): Performing sync in Hub ignores sync toggle an...https://pulp.plan.io/issues/95252021-10-20T17:27:32Zawcrosby
<p>This is seen in galaxy_ng 4.3.3 performing a sync against c.rh.c:</p>
<ul>
<li>galaxy_ng==4.3.3 includes: pulpcore==3.11.2, pulp-ansible==0.7.3</li>
<li>c.rh.com includes: pulpcore==3.11.2 pulp-ansible==0.7.3</li>
<li>FWIW on console.rh.com, editing the sync toggles changes the <code>content_count</code> (527) at <a href="https://console.redhat.com/api/automation-hub/_ui/v1/my-distributions/" class="external">https://console.redhat.com/api/automation-hub/_ui/v1/my-distributions/</a> for <code>5910538-synclist</code> but it does not change the <code>count</code> (104) at <a href="https://console.redhat.com/api/automation-hub/content/5910538-synclist/v3/collections/" class="external">https://console.redhat.com/api/automation-hub/content/5910538-synclist/v3/collections/</a>
</li>
</ul>
<p>This is also seen in galaxy_ng <code>master</code> performing a sync against c.rh.c and c.stage.rh.c</p>
<ul>
<li>galaxy_ng==<code>master</code> includes: pulpcore==3.15.2, pulp-ansible==0.10.1</li>
<li>c.stage.rh.com includes: pulpcore==3.14.6, pulp-ansible==0.9.2</li>
<li>FWIW the count c.stage.rh.com <###>-synclist/v3/collections/ does change after sync toggles</li>
</ul> Ansible Plugin - Story #8486 (CLOSED - CURRENTRELEASE): Provide backend storage url to galaxy-imp...https://pulp.plan.io/issues/84862021-03-31T16:14:31Zawcrosby
<p>Currently galaxy-importer adds a ResponseContentDisposition <a href="https://github.com/ansible/galaxy-importer/blob/master/galaxy_importer/collection.py#L71" class="external">https://github.com/ansible/galaxy-importer/blob/master/galaxy_importer/collection.py#L71</a> to retreive the storage backend url.</p>
<p>This works for backend AWS but not for Azure. It would be preferable if pulp-ansible passed this url to galaxy-importer, calculating using similar logic to pulpcore _serve_content_artifact() <a href="https://github.com/pulp/pulpcore/blob/8a77316d2d73dd31d6f47b964bf148a602da6a21/pulpcore/content/handler.py#L581" class="external">https://github.com/pulp/pulpcore/blob/8a77316d2d73dd31d6f47b964bf148a602da6a21/pulpcore/content/handler.py#L581</a></p> Ansible Plugin - Issue #8238 (CLOSED - NOTABUG): 0030_collectionversion_requires_ansible.py migra...https://pulp.plan.io/issues/82382021-02-10T17:21:38Zalikins
<p>After updating galaxy_ng to use git versions of pulpcore, pulp_ansible, pulp_container I:</p>
<ol>
<li>Ran <code>./compose run api manage handle-artifact-checksums</code>. That seemed to run without errors.</li>
<li>
<code>./compose up</code> which brought up the galaxy_ng servers but showed needed migrations.</li>
<li>In another term, ran <code>./compose run api manage migrate</code>
</li>
</ol>
<p>Got the Error:</p>
<pre><code>Operations to perform:
Apply all migrations: admin, ansible, auth, authtoken, container, contenttypes, core, galaxy, guardian, sessions
Running migrations:
Applying ansible.0027_tag_length... OK
Applying ansible.0028_collectionversion_namespace_length... OK
Applying ansible.0029_manifest_and_files_json_fields... OK
Applying ansible.0030_collectionversion_requires_ansible...Traceback (most recent call last):
File "/venv/bin/django-admin", line 8, in <module>
sys.exit(execute_from_command_line())
File "/venv/lib64/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/venv/lib64/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/venv/lib64/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "/venv/lib64/python3.6/site-packages/django/core/management/base.py", line 364, in execute
output = self.handle(*args, **options)
File "/venv/lib64/python3.6/site-packages/django/core/management/base.py", line 83, in wrapped
res = handle_func(*args, **kwargs)
File "/venv/lib64/python3.6/site-packages/django/core/management/commands/migrate.py", line 234, in handle
fake_initial=fake_initial,
File "/venv/lib64/python3.6/site-packages/django/db/migrations/executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/venv/lib64/python3.6/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/venv/lib64/python3.6/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
state = migration.apply(state, schema_editor)
File "/venv/lib64/python3.6/site-packages/django/db/migrations/migration.py", line 124, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/venv/lib64/python3.6/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
self.code(from_state.apps, schema_editor)
File "/src/pulp_ansible/pulp_ansible/app/migrations/0030_collectionversion_requires_ansible.py", line 19, in set_requires_ansible_and_manifest_and_files_json
with artifact.file.open() as artifact_file, tarfile.open(fileobj=artifact_file, mode="r") as tar:
File "/venv/lib64/python3.6/site-packages/django/db/models/fields/files.py", line 74, in open
self.file = self.storage.open(self.name, mode)
File "/venv/lib64/python3.6/site-packages/django/core/files/storage.py", line 36, in open
return self._open(name, mode)
File "/venv/lib64/python3.6/site-packages/django/core/files/storage.py", line 224, in _open
return File(open(self.path(name), mode))
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/d4/07b75923940bb9a477be4fe019a9ae1ec8b85a998f218aa5913a455488a202'
ERROR: 1
</code></pre> Ansible Plugin - Issue #8078 (CLOSED - CURRENTRELEASE): Make consistent pulp_ansible namespace ma...https://pulp.plan.io/issues/80782021-01-12T16:06:06Zawcrosby
<p>Collection.namespace max_length is 64 [1], while CollectionVersion.namespace max_length is 32 [2]</p>
<p>[1] <a href="https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/models.py#L48" class="external">https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/models.py#L48</a>
[2] <a href="https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/models.py#L140" class="external">https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/models.py#L140</a></p>
<p>I assume CollectionVersion should be changed to match Collection.</p> Ansible Plugin - Issue #7834 (CLOSED - CURRENTRELEASE): Updated deprecation status is not include...https://pulp.plan.io/issues/78342020-11-13T19:23:49Zawcrosby
<p>This was seen on testing with galaxy_ng: In 4.2.0 GA, updated deprecation status is not sync’d from cloud to on-prem hub</p>
<p>Steps to recreate:</p>
<ol>
<li>Ensure the <code>synctest</code> repo in ci.cloud.redhat.com has some collections deprecated, others not</li>
<li>For the first time, sync the repo from cloud to on-prem hub</li>
<li>Confirm deprecation status is as expected on each collection in on-prem hub</li>
<li>Change deprecation status on one of the synced collections on cloud</li>
<li>Sync again</li>
<li>Observe that the collection on-prem does not have the deprecation status changed as expected</li>
</ol> Ansible Plugin - Issue #7788 (CLOSED - CURRENTRELEASE): Allow CollectionUploadViewSet subclass to...https://pulp.plan.io/issues/77882020-11-04T15:28:07ZawcrosbyAnsible Plugin - Issue #7669 (CLOSED - CURRENTRELEASE): pulp_ansible galaxy collection endpoint, ...https://pulp.plan.io/issues/76692020-10-07T18:26:51Zawcrosby
<p>An empty RepositoryVersion shows entries for all CollectionVersions in system, via <code>/pulp_ansible/galaxy/<path>/api/v3/collections/</code></p>
<p>The list should be empty but it is not, it also should be a list of unique collections but in this case collections are shown multiple times with different <code>highest_version</code>s.</p> Ansible Plugin - Issue #7551 (CLOSED - CURRENTRELEASE): Standalone galaxy_ng syncing from cloud.c...https://pulp.plan.io/issues/75512020-09-22T19:40:30Zalikins
<p><a href="https://github.com/ansible/galaxy_ng/issues/441" class="external">https://github.com/ansible/galaxy_ng/issues/441</a> is the origin of this.</p>
<p>It seems like if a standalone galaxy_ng syncs from cloud.ci.redhat.com, the doc blobs for the collections isn't
synced.</p>
<p><a href="https://ci.cloud.redhat.com/api/automation-hub/content/synctest/v3/collections" class="external">https://ci.cloud.redhat.com/api/automation-hub/content/synctest/v3/collections</a> is an example repo set up with
some collections with docs blobs.</p>
<p>At the moment, it's unclear if this is an issue with standalone galaxy_ng, or with the ci.cloud galaxy_ng / pulp_ansible / pulp_core issue. But this is a heads up.</p>
<p>For an example of a collection in that repo:</p>
<p><a href="https://ci.cloud.redhat.com/api/automation-hub/content/synctest/v3/collections/ibm/spectrum_virtualize/" class="external">https://ci.cloud.redhat.com/api/automation-hub/content/synctest/v3/collections/ibm/spectrum_virtualize/</a></p>
<p>And the docs-blob for a particular version:
<a href="https://ci.cloud.redhat.com/api/automation-hub/content/synctest/v3/collections/ibm/spectrum_virtualize/versions/1.0.12-awc16/docs-blob/" class="external">https://ci.cloud.redhat.com/api/automation-hub/content/synctest/v3/collections/ibm/spectrum_virtualize/versions/1.0.12-awc16/docs-blob/</a></p>
<p>The latter shows info available in the docs-blob, but it apperently doesn't sync synced to the standalone galaxy_ng</p>
<p>Versions:</p>
<p>On the standalone side:</p>
<p>galaxy-ng (4.2.0b2)
pulp-ansible (0.3.0)
pulpcore (3.6.0</p> Ansible Plugin - Issue #7032 (CLOSED - DUPLICATE): Provide a way for subclasses of pulp viewset t...https://pulp.plan.io/issues/70322020-06-22T20:58:09Zalikins
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_ansible/710":<a href="https://github.com/pulp/pulp_ansible/issues/710" class="external">https://github.com/pulp/pulp_ansible/issues/710</a></p>
<hr>
<p>This is related to <a href="https://github.com/ansible/galaxy_ng/pull/254" class="external">https://github.com/ansible/galaxy_ng/pull/254</a>
and <a href="https://github.com/ansible/galaxy_ng/issues/247" class="external">https://github.com/ansible/galaxy_ng/issues/247</a></p>
<p>To paraphrase <a href="https://github.com/ansible/galaxy_ng/issues/247#issuecomment-644886557" class="external">https://github.com/ansible/galaxy_ng/issues/247#issuecomment-644886557</a>:</p>
<blockquote>
<p>The serializers used by the pulp collection viewsets (and therefore galaxy_ng collection viewsets) base the values for 'href' on url reverse() lookups, and end up with the pulp paths.</p>
<p>May mean galaxy_ng viewsets need to provide their own serializer.</p>
<p>But may be able to parameterize the reverse()'s to handle both cases. Possibly via providing a ViewSet.get_serializer_context()</p>
</blockquote>
<p><a href="https://github.com/ansible/galaxy_ng/pull/254" class="external">https://github.com/ansible/galaxy_ng/pull/254</a> is a fix / workaround that is based on subclassing the
related pulp collection Serializers and overriding the href related SerializerMethodFields methods.</p>
<p>ie, the 'get_href()' and similar methods.</p>
<p>The changes in #254 are kind of cut&paste and duplicate a chunk of pulp_ansible code (and diverge
from upstream as a result). So if there is a reasonable way to avoid that it's probably worth while.</p>
<p><a href="https://github.com/ansible/galaxy_ng/issues/247#issuecomment-644983783" class="external">https://github.com/ansible/galaxy_ng/issues/247#issuecomment-644983783</a> mentions a few vague ideas on approaches to handle this issue. In particular, the main <a class="issue tracker-3 status-11 priority-7 priority-high2 closed" title="Story: As a user, my rpm sync finishes quickly when upstream metadata hasn't changed (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/2">#2</a> point there.</p>
<p>Gist is providing a way for the serializers get_href methods to use a url namespace (or specify url name explicitly) when the viewset classes have been subclassed (as galaxy_ng.app.api.v3.viewsets.collections.CollectionViewSet does).</p>
<ul>
<li>
<p>There may be existing django or drf support for doing this but it isn't clear to me yet. (The 'content_app' attribute mentioned in <a href="https://github.com/ansible/galaxy_ng/issues/247#issuecomment-644983783" class="external">https://github.com/ansible/galaxy_ng/issues/247#issuecomment-644983783</a> seems like it is intended for exactly this case, but I couldn't get it to work. But that is likely developer error on my part)</p>
</li>
<li>
<p>Maybe this is something drf-nested-routers / rest_framework_nested is meant to support?</p>
</li>
</ul> Ansible Plugin - Task #6909 (CLOSED - WONTFIX): import_collection task to send filepath to galaxy...https://pulp.plan.io/issues/69092020-06-04T18:56:39Zawcrosby
<p>Currently the import_collection task sends the artifact fileobject to galaxy-importer. In galaxy-importer we want move from using the tarfile module to tar in shell.</p>
<p>We would like import_collection task to send the artifact filepath to galaxy-importer.</p>
<p>Also, currently in the fileobject we use the .storage.url property to download the artifact to a container to run "ansible-test sanity". So perhaps we can send the filepath + url instead of the opened fileobject.</p> Ansible Plugin - Issue #6129 (CLOSED - CURRENTRELEASE): sync doesn't do anything because 'downloa...https://pulp.plan.io/issues/61292020-02-11T15:58:41Zalikins
<p>With a fix for <a href="https://pulp.plan.io/issues/6116" class="external">https://pulp.plan.io/issues/6116</a> in place, the next
issue is related to when/where the 'download_url' attribute is available.</p>
<p>Current code in pulp_ansible.app.tasks.collections.CollectionsFirstStage._fetch_collections()
expects 'download_url' to be available but on v3 API, it's ownly on the collection version detail page,
so additional request for that api endpoint will be needed.</p> Ansible Plugin - Issue #6116 (CLOSED - CURRENTRELEASE): Collection sync task fails syncing from a...https://pulp.plan.io/issues/61162020-02-07T21:26:30Zalikins
<p>When attempting to sync a repo to a CollectionRemote that has a 'url' pointing to automation-hub, the sync fails in pulp_ansible.tasks.collections.CollectionSyncFirstStage._fetch_collections().</p>
<p>For automation-hub / v3 collections API, the 'versions_url' field no longer includes a scheme.
Previously a 'versions_url' would be a full url like <a href="https://galaxy.ansible.com/api/v3/collections/some_namespace/some_collection/versions/" class="external">https://galaxy.ansible.com/api/v3/collections/some_namespace/some_collection/versions/</a>, but the v3 api uses '/api/automation-hub/v3/collections/some_namespace/some_collection/versions'.</p>
<p>When the CollectionSync tasks uses 'version_url' (<a href="https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/tasks/collections.py#L349" class="external">https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/tasks/collections.py#L349</a>)
it eventually causes pulpcore DownloadFactory to hit a KeyError looking up the '' scheme (<a href="https://github.com/pulp/pulpcore/blob/master/pulpcore/download/factory.py#L135-L138" class="external">https://github.com/pulp/pulpcore/blob/master/pulpcore/download/factory.py#L135-L138</a>)</p>
<p><a href="https://gist.github.com/alikins/b6349ad49a5ce1b25fa6c9d938a01764#file-gistfile2-txt" class="external">https://gist.github.com/alikins/b6349ad49a5ce1b25fa6c9d938a01764#file-gistfile2-txt</a> has some logs pasted here for reference:</p>
<pre><code>pulp-worker_1 | pulp: rq.worker:ERROR: ValueError: URL: /api/automation-hub/v3/collections/autohubtest2/collection_dep_a_jbkboxzc/versions/ not supported.
pulp-worker_1 | Traceback (most recent call last):
pulp-worker_1 | File "/venv/lib64/python3.6/site-packages/pulpcore/download/factory.py", line 137, in build
pulp-worker_1 | builder = self._handler_map[scheme]
pulp-worker_1 | KeyError: ''
pulp-worker_1 |
pulp-worker_1 | During handling of the above exception, another exception occurred:
pulp-worker_1 |
pulp-worker_1 | Traceback (most recent call last):
pulp-worker_1 | File "/venv/lib64/python3.6/site-packages/rq/worker.py", line 884, in perform_job
pulp-worker_1 | rv = job.perform()
pulp-worker_1 | File "/venv/lib64/python3.6/site-packages/rq/job.py", line 664, in perform
pulp-worker_1 | self._result = self._execute()
pulp-worker_1 | File "/venv/lib64/python3.6/site-packages/rq/job.py", line 670, in _execute
pulp-worker_1 | return self.func(*self.args, **self.kwargs)
pulp-worker_1 | File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 80, in sync
pulp-worker_1 | d_version.create()
pulp-worker_1 | File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
pulp-worker_1 | loop.run_until_complete(pipeline)
pulp-worker_1 | File "/opt/rh/rh-python36/root/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
pulp-worker_1 | return future.result()
pulp-worker_1 | File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 228, in create_pipeline
pulp-worker_1 | await asyncio.gather(*futures)
pulp-worker_1 | File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
pulp-worker_1 | await self.run()
pulp-worker_1 | File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 274, in run
pulp-worker_1 | async for metadata in self._fetch_collections():
pulp-worker_1 | File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 351, in _fetch_collections
pulp-worker_1 | [remote.get_downloader(url=result["versions_url"]).run()]
pulp-worker_1 | File "/venv/lib64/python3.6/site-packages/pulpcore/app/models/repository.py", line 288, in get_downloader
pulp-worker_1 | return self.download_factory.build(url, **kwargs)
pulp-worker_1 | File "/venv/lib64/python3.6/site-packages/pulpcore/download/factory.py", line 140, in build
pulp-worker_1 | raise ValueError(_('URL: {u} not supported.'.format(u=url)))
pulp-worker_1 | ValueError: URL: /api/automation-hub/v3/collections/autohubtest2/collection_dep_a_jbkboxzc/versions/ not supported.
pulp-worker_1 | Traceback (most recent call last):
pulp-worker_1 | File "/venv/lib64/python3.6/site-packages/pulpcore/download/factory.py", line 137, in build
pulp-worker_1 | builder = self._handler_map[scheme]
pulp-worker_1 | KeyError: ''
pulp-worker_1 |
pulp-worker_1 | During handling of the above exception, another exception occurred:
pulp-worker_1 |
pulp-worker_1 | Traceback (most recent call last):
pulp-worker_1 | File "/venv/lib64/python3.6/site-packages/rq/worker.py", line 884, in perform_job
pulp-worker_1 | rv = job.perform()
pulp-worker_1 | File "/venv/lib64/python3.6/site-packages/rq/job.py", line 664, in perform
pulp-worker_1 | self._result = self._execute()
pulp-worker_1 | File "/venv/lib64/python3.6/site-packages/rq/job.py", line 670, in _execute
pulp-worker_1 | return self.func(*self.args, **self.kwargs)
pulp-worker_1 | File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 80, in sync
pulp-worker_1 | d_version.create()
pulp-worker_1 | File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
pulp-worker_1 | loop.run_until_complete(pipeline)
pulp-worker_1 | File "/opt/rh/rh-python36/root/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
pulp-worker_1 | return future.result()
pulp-worker_1 | File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 228, in create_pipeline
pulp-worker_1 | await asyncio.gather(*futures)
pulp-worker_1 | File "/venv/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
pulp-worker_1 | await self.run()
pulp-worker_1 | File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 274, in run
pulp-worker_1 | async for metadata in self._fetch_collections():
pulp-worker_1 | File "/code/pulp_ansible/pulp_ansible/app/tasks/collections.py", line 351, in _fetch_collections
pulp-worker_1 | [remote.get_downloader(url=result["versions_url"]).run()]
pulp-worker_1 | File "/venv/lib64/python3.6/site-packages/pulpcore/app/models/repository.py", line 288, in get_downloader
pulp-worker_1 | return self.download_factory.build(url, **kwargs)
pulp-worker_1 | File "/venv/lib64/python3.6/site-packages/pulpcore/download/factory.py", line 140, in build
pulp-worker_1 | raise ValueError(_('URL: {u} not supported.'.format(u=url)))
pulp-worker_1 | ValueError: URL: /api/automation-hub/v3/collections/autohubtest2/collection_dep_a_jbkboxzc/versions/ not supported.
</code></pre>
<pre><code>
</code></pre> Ansible Plugin - Issue #5583 (CLOSED - DUPLICATE): Add methods/mixins to galaxy.v3.views.Collecti...https://pulp.plan.io/issues/55832019-10-16T19:48:17Zalikins
<p>pulp_ansible.app.galaxy.v3.views.CollectionViewSet needs to support a way to update it's 'deprecated' field.</p>
<p>See also <a href="https://github.com/ansible/galaxy-dev/issues/129" class="external">https://github.com/ansible/galaxy-dev/issues/129</a></p> Ansible Plugin - Task #5368 (CLOSED - CURRENTRELEASE): Write galaxy-import result of contents and...https://pulp.plan.io/issues/53682019-08-28T21:11:22Zawcrosby
<p>The galaxy-importer result includes contents and docs_blob keys. The CollectionVersion model has attributes for these.</p>
<p>This data should be written when the galaxy-importer is called, in both the import_collection and sync workflows.</p> Ansible Plugin - Task #3495 (CLOSED - DUPLICATE): Add sync_mode parameter to endpoint that deploy...https://pulp.plan.io/issues/34952018-03-16T13:21:40Zamacdona@redhat.comaustin@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_ansible/678":<a href="https://github.com/pulp/pulp_ansible/issues/678" class="external">https://github.com/pulp/pulp_ansible/issues/678</a></p>
<hr>
<p>sync_mode will be removed in <a href="https://pulp.plan.io/issues/3492" class="external">https://pulp.plan.io/issues/3492</a>, which will break sync.</p>