Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-12-18T16:19:37ZPulp
Planio RPM Support - Backport #9644 (CLOSED - DUPLICATE): Backport #9636 'FileNotFoundError: [Errno 2] N...https://pulp.plan.io/issues/96442021-12-18T16:19:37Zttereshcttereshc@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2310":<a href="https://github.com/pulp/pulp_rpm/issues/2310" class="external">https://github.com/pulp/pulp_rpm/issues/2310</a></p>
<hr>
<p>Pulp expects custom metadata files to have a specific format for filenames: -.</p>
<p>During sync or pulp-2to3-migration, users get <code>FileNotFoundError: [Errno 2] No such file or directory: ' '</code> because the filename is only a checksum :/, which we remove to determine the filename, so the path we detect becomes an empty string.</p>
<pre><code>Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 478, in migrate_repo_distributor
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: pulp2dist, repo_version)
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 91, in migrate_to_pulp3
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite)
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 344, in publish
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: publication_data.populate()
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 253, in populate
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: self.repomdrecords = self.prepare_metadata_files(main_content)
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 99, in prepare_metadata_files
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: with open(path, "wb") as new_file:
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: FileNotFoundError: [Errno 2] No such file or directory: ''
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: pulp: rq.worker:INFO: 27738@1002a1103081001.xxx.com: c0d58c5a-9ff1-4d40-bbb3-9c24fdf0fdb4
Nov 22 21:31:34 1002a1103081001 pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: 43f43c1a-d09e-46bd-99fe-73b6e2ee397c
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-3: pulp: rq.worker:INFO: 27740@1002a1103081001.xxx.com: Job OK (b25f3fa2-401c-425c-92eb-b49a61415617)
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-2: pulp: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=XXX-Red_Hat_Ansible_Engine_2_7_RPMs_for_Red_Hat_Enterprise_Linux_7_Server_x86_64, version=1
</code></pre> RPM Support - Test #9626 (CLOSED - DUPLICATE): Add tests for SHA repo to test_synchttps://pulp.plan.io/issues/96262021-12-08T21:43:04Zggainey
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2320":<a href="https://github.com/pulp/pulp_rpm/issues/2320" class="external">https://github.com/pulp/pulp_rpm/issues/2320</a></p>
<hr>
<p>We have been bitten a few times now by repos that use 'sha' (instead of 'sha1') checksums. Build a test for same that does <strong>not</strong> require sync'ing all of RHEL6.6...</p> RPM Support - Task #9585 (CLOSED - DUPLICATE): Sub-tree-only sync breaks our model of how syncs a...https://pulp.plan.io/issues/95852021-11-23T21:00:08Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2306":<a href="https://github.com/pulp/pulp_rpm/issues/2306" class="external">https://github.com/pulp/pulp_rpm/issues/2306</a></p>
<hr>
<p>See: <a href="https://pulp.plan.io/issues/9565" class="external">https://pulp.plan.io/issues/9565</a></p>
<p>We haven't considered the possibility of a repository having a .treeinfo but no top-level repodata. We need to investigate whether this breaks anything (for example: autopublish), and potentially add a new test fixture to exercise it.</p> Pulp - Task #9575 (CLOSED - WONTFIX): Move queryset filtering for Tasks to Task.objectshttps://pulp.plan.io/issues/95752021-11-18T21:06:16Zbmbouterbmbouter@redhat.com
<p>Right now the Task queryset scoping is implemented only in DRF. Recently the new task purge endpoint deletes task records in a task. To do that in a way that aligns with the queryset scoping and RBAC the Tasks endpoint already supports we have to have the queryset manager provide the queryset scoping.</p> Container Support - Task #9572 (CLOSED - DUPLICATE): Port the RBAC implementation to the pulpcore...https://pulp.plan.io/issues/95722021-11-16T13:48:53Zmdellweg
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_container/508":<a href="https://github.com/pulp/pulp_container/issues/508" class="external">https://github.com/pulp/pulp_container/issues/508</a></p>
<hr>
<p>Start with a PoC PR to get the pulpcore PR merged first.</p>
<p>Once that is done, we need to write a data migration that will look for the autogenerated groups and translate them into user_object_roles.
To be discussed: Look if we can be clever with global permissions too.</p> Container Support - Story #9509 (CLOSED - DUPLICATE): As a user I can sign container image by pro...https://pulp.plan.io/issues/95092021-10-14T10:14:23Zipanova@redhat.comipanova@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_container/500":<a href="https://github.com/pulp/pulp_container/issues/500" class="external">https://github.com/pulp/pulp_container/issues/500</a></p>
<hr>
<p>As a result signature will be created and saved into the Pulp Container Registry Sigstore</p>
<p>It should be possible to sign whole repo, list of images or a specific image only.</p>
<p>See <a href="https://github.com/containers/skopeo/blob/main/docs/skopeo-standalone-sign.1.md" class="external">https://github.com/containers/skopeo/blob/main/docs/skopeo-standalone-sign.1.md</a> and <a href="https://github.com/containers/image/blob/main/docs/containers-signature.5.md#json-data-format" class="external">https://github.com/containers/image/blob/main/docs/containers-signature.5.md#json-data-format</a></p> Container Support - Story #9507 (CLOSED - DUPLICATE): As a user I can sync container image with i...https://pulp.plan.io/issues/95072021-10-14T10:12:04Zipanova@redhat.comipanova@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_container/498":<a href="https://github.com/pulp/pulp_container/issues/498" class="external">https://github.com/pulp/pulp_container/issues/498</a></p>
<hr>
<p>Synced signatures are not verified. Signature verification is offloaded to the client.</p>
<p>If <code>with-signature=True</code> ( TBD agree on naming) was specified only those images that have signature with be mirrored. Rest of the images will be skipped. <code>Signature server location</code> needs to be specified ( it's the webserver location of signatures, for example <a href="https://registry.redhat.io/containers/sigstore" class="external">https://registry.redhat.io/containers/sigstore</a>).
Otherwise it will be assumed that the signature is stored on the remote registry in a form of manifest or a separate object:</p>
<ul>
<li>If it is a separate object, signature API extensions should be available on the registry and <code>X-Registry-Supports-Signatures</code> header will identify that <a href="https://github.com/containers/image/blob/main/docs/signature-protocols.md#openshift-dockerdistribution-api-extension" class="external">https://github.com/containers/image/blob/main/docs/signature-protocols.md#openshift-dockerdistribution-api-extension</a>
</li>
<li>If it is stored in a form of a manifest then it most likely was signed with cosign <a href="https://github.com/SigStore/cosign#signing-subjects" class="external">https://github.com/SigStore/cosign#signing-subjects</a>. <strong>we will not support this for now</strong>
</li>
</ul>
<p><strong>Q:</strong> does podman/skopeo support verification of cosign signature type? Only atomic type for now <a href="https://github.com/containers/image/blob/main/docs/containers-signature.5.md#criticaltype" class="external">https://github.com/containers/image/blob/main/docs/containers-signature.5.md#criticaltype</a></p>
<p><strong>Q:</strong> how this will work with mirror=True?</p> Container Support - Story #9500 (CLOSED - DUPLICATE): Enable cache in pulp container handlerhttps://pulp.plan.io/issues/95002021-10-11T15:32:22Zipanova@redhat.comipanova@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_container/496":<a href="https://github.com/pulp/pulp_container/issues/496" class="external">https://github.com/pulp/pulp_container/issues/496</a></p>
<hr>
<p><a href="https://github.com/pulp/pulpcore/commit/81684b206e3fd8bceb104e4a77c0d717387667f9#diff-453fdf3a6e81f92ff5391cf65f9c6154852133a656d82082021e3beaa887776fR194" class="external">https://github.com/pulp/pulpcore/commit/81684b206e3fd8bceb104e4a77c0d717387667f9#diff-453fdf3a6e81f92ff5391cf65f9c6154852133a656d82082021e3beaa887776fR194</a></p>
<p>Pulp Container Registry Handler does not use <code>stream_content</code></p> RPM Support - Issue #9406 (CLOSED - NOTABUG): Trivial OOM on sync for a particular Microsoft repohttps://pulp.plan.io/issues/94062021-09-16T03:33:52Zdalleydalley@redhat.com
<p>Syncing <a href="https://packages.microsoft.com/rhel/8/prod/" class="external">https://packages.microsoft.com/rhel/8/prod/</a> results in an OOM even on a box with ~10gb of available memory</p> RPM Support - Issue #9399 (CLOSED - NOTABUG): sync error: invalid memory alloc request sizehttps://pulp.plan.io/issues/93992021-09-15T09:20:51Zkeilr
<p>pulp version: container image docker.io/pulp/pulp:3.15</p>
<p>remote repository definition:</p>
<pre><code class="text syntaxhl" data-language="text">$ pulp --base-url http://localhost:8080 --username admin --password secret rpm remote show --name "packages-microsoft-com-prod-rhel8"
{
"pulp_href": "/pulp/api/v3/remotes/rpm/rpm/c455d98d-4c8c-446d-aa4d-dffe568675d6/",
"pulp_created": "2021-09-14T14:51:03.146140Z",
"name": "packages-microsoft-com-prod-rhel8",
"url": "https://packages.microsoft.com/rhel/8/prod/",
"ca_cert": null,
"client_cert": null,
"tls_validation": true,
"proxy_url": "http://proxy.example.com:8080",
"pulp_labels": {},
"pulp_last_updated": "2021-09-14T14:51:03.146171Z",
"download_concurrency": null,
"max_retries": null,
"policy": "immediate",
"total_timeout": null,
"connect_timeout": null,
"sock_connect_timeout": null,
"sock_read_timeout": null,
"headers": null,
"rate_limit": null,
"sles_auth_token": null
}
</code></pre>
<p>error:</p>
<pre><code class="text syntaxhl" data-language="text">$ podman logs --follow pulp
pulp [505c8f64043741a7b8f09eac46fa8331]: pulpcore.tasking.pulpcore_worker:INFO: Task a7a26b28-9251-4da1-82ef-19272b6779f0 failed (invalid memory alloc request size 1073741824
)
pulp [505c8f64043741a7b8f09eac46fa8331]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/local/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 323, in _perform_task
result = func(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 471, in synchronize
version = dv.create()
File "/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
loop.run_until_complete(pipeline)
File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
await asyncio.gather(*futures)
File "/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
await self.run()
File "/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/content_stages.py", line 174, in run
await sync_to_async(process_batch)()
File "/usr/local/lib/python3.8/site-packages/asgiref/sync.py", line 444, in __call__
ret = await asyncio.wait_for(future, timeout=None)
File "/usr/lib64/python3.8/asyncio/tasks.py", line 455, in wait_for
return await fut
File "/usr/lib64/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.8/site-packages/asgiref/sync.py", line 486, in thread_handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/content_stages.py", line 122, in process_batch
d_content.content.save()
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/base.py", line 149, in save
return super().save(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/django_lifecycle/mixins.py", line 134, in save
save(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 726, in save
self.save_base(using=using, force_insert=force_insert,
File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 763, in save_base
updated = self._save_table(
File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 868, in _save_table
results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 906, in _do_insert
return manager._insert(
File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1270, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
</code></pre>
<p>All other repo sync jobs work without problems. (e.g. official rhel(7|8) repos, epel, ...)</p>
<p>Is anyone able to reproduce this MS repo issue?</p> RPM Support - Backport #9313 (CLOSED - DUPLICATE): Backport #9309 "Add support for new memory-eff...https://pulp.plan.io/issues/93132021-08-30T14:28:02Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2297":<a href="https://github.com/pulp/pulp_rpm/issues/2297" class="external">https://github.com/pulp/pulp_rpm/issues/2297</a></p> RPM Support - Refactor #9309 (CLOSED - DUPLICATE): Add support for new memory-efficient createrep...https://pulp.plan.io/issues/93092021-08-30T03:50:56Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2296":<a href="https://github.com/pulp/pulp_rpm/issues/2296" class="external">https://github.com/pulp/pulp_rpm/issues/2296</a></p> RPM Support - Task #9259 (CLOSED - DUPLICATE): workflow-docs should use pulp-cli instead of httpiehttps://pulp.plan.io/issues/92592021-08-19T14:09:05Zggainey
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2314":<a href="https://github.com/pulp/pulp_rpm/issues/2314" class="external">https://github.com/pulp/pulp_rpm/issues/2314</a></p>
<hr>
<p>Probably shouldn't do this unless/until <strong>all</strong> workflows are supported by pulp-cli commands - mixing pulp-cli and httpie is Very Confusing to the user.</p>
<p>Putting this up as a placeholder for docs-day</p> RPM Support - Test #8809 (CLOSED - DUPLICATE): Better tests for metadata mirroringhttps://pulp.plan.io/issues/88092021-05-24T19:54:30Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2277":<a href="https://github.com/pulp/pulp_rpm/issues/2277" class="external">https://github.com/pulp/pulp_rpm/issues/2277</a></p>
<hr>
<p>We need a fixture repository with some of the extra files, such as repomd.xml.asc (metadata signature), extra_files.json, .treeinfo, possibly licenses, multiple package directories / package locations, extra repomd entries that Pulp doesn't natively care about, etc. And then we need to test that mirroring works properly with such repos.</p> RPM Support - Story #8673 (CLOSED - DUPLICATE): Auto-publishing should be more fault-toleranthttps://pulp.plan.io/issues/86732021-04-30T14:56:08Zsskracic@redhat.comsskracic@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2273":<a href="https://github.com/pulp/pulp_rpm/issues/2273" class="external">https://github.com/pulp/pulp_rpm/issues/2273</a></p>
<hr>
<p>I admit the title is a bit vague.</p>
<p>During auto-publishing sync of a very large repository (rhel-7-server-rpms), the <code>rq</code> process got killed by oom-killer sometime in the middle of the publishing step. So the new repository version (1) got created, but accompanying
publication did not.</p>
<p>On the subsequent sync runs, the repository did not get published, as no new content was available to sync, hence a new repository version was not created, which in turn should trigger publication and distribution update.</p>
<p>Of course, the repo can still be published and distributed using the non-autopublishing REST API, but I wonder whether the behavior was intended.</p>