Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-12-18T16:19:37ZPulp
Planio RPM Support - Backport #9644 (CLOSED - DUPLICATE): Backport #9636 'FileNotFoundError: [Errno 2] N...https://pulp.plan.io/issues/96442021-12-18T16:19:37Zttereshcttereshc@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2310":<a href="https://github.com/pulp/pulp_rpm/issues/2310" class="external">https://github.com/pulp/pulp_rpm/issues/2310</a></p>
<hr>
<p>Pulp expects custom metadata files to have a specific format for filenames: -.</p>
<p>During sync or pulp-2to3-migration, users get <code>FileNotFoundError: [Errno 2] No such file or directory: ' '</code> because the filename is only a checksum :/, which we remove to determine the filename, so the path we detect becomes an empty string.</p>
<pre><code>Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 478, in migrate_repo_distributor
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: pulp2dist, repo_version)
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 91, in migrate_to_pulp3
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite)
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 344, in publish
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: publication_data.populate()
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 253, in populate
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: self.repomdrecords = self.prepare_metadata_files(main_content)
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 99, in prepare_metadata_files
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: with open(path, "wb") as new_file:
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: FileNotFoundError: [Errno 2] No such file or directory: ''
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: pulp: rq.worker:INFO: 27738@1002a1103081001.xxx.com: c0d58c5a-9ff1-4d40-bbb3-9c24fdf0fdb4
Nov 22 21:31:34 1002a1103081001 pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: 43f43c1a-d09e-46bd-99fe-73b6e2ee397c
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-3: pulp: rq.worker:INFO: 27740@1002a1103081001.xxx.com: Job OK (b25f3fa2-401c-425c-92eb-b49a61415617)
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-2: pulp: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=XXX-Red_Hat_Ansible_Engine_2_7_RPMs_for_Red_Hat_Enterprise_Linux_7_Server_x86_64, version=1
</code></pre> RPM Support - Test #9626 (CLOSED - DUPLICATE): Add tests for SHA repo to test_synchttps://pulp.plan.io/issues/96262021-12-08T21:43:04Zggainey
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2320":<a href="https://github.com/pulp/pulp_rpm/issues/2320" class="external">https://github.com/pulp/pulp_rpm/issues/2320</a></p>
<hr>
<p>We have been bitten a few times now by repos that use 'sha' (instead of 'sha1') checksums. Build a test for same that does <strong>not</strong> require sync'ing all of RHEL6.6...</p> RPM Support - Issue #9619 (CLOSED - DUPLICATE): OpenAPI schema for ModulemdDefault is incorrecthttps://pulp.plan.io/issues/96192021-12-08T16:59:02Zdkliban@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2307":<a href="https://github.com/pulp/pulp_rpm/issues/2307" class="external">https://github.com/pulp/pulp_rpm/issues/2307</a></p>
<hr>
<p>The open api schema for ModulemdDefaults is incorrect. When trying to generate a client with openapi-generator-cli 5.3.0, the following exception is emitted:</p>
<pre><code>Exception in thread "main" java.lang.RuntimeException: Could not generate api file for 'ContentModulemdDefaults'
</code></pre> RPM Support - Task #9585 (CLOSED - DUPLICATE): Sub-tree-only sync breaks our model of how syncs a...https://pulp.plan.io/issues/95852021-11-23T21:00:08Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2306":<a href="https://github.com/pulp/pulp_rpm/issues/2306" class="external">https://github.com/pulp/pulp_rpm/issues/2306</a></p>
<hr>
<p>See: <a href="https://pulp.plan.io/issues/9565" class="external">https://pulp.plan.io/issues/9565</a></p>
<p>We haven't considered the possibility of a repository having a .treeinfo but no top-level repodata. We need to investigate whether this breaks anything (for example: autopublish), and potentially add a new test fixture to exercise it.</p> RPM Support - Issue #9583 (CLOSED - DUPLICATE): Distribution tree uniqueness constraint is not en...https://pulp.plan.io/issues/95832021-11-23T17:43:48Zttereshcttereshc@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2305":<a href="https://github.com/pulp/pulp_rpm/issues/2305" class="external">https://github.com/pulp/pulp_rpm/issues/2305</a></p>
<hr>
<p>Currently <a href="https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models/distribution.py#L133" class="external">Pulp requires</a> the combination of the following fields to be unique:</p>
<pre><code> unique_together = (
"header_version",
"release_name",
"release_short",
"release_version",
"arch",
"build_timestamp",
)
</code></pre>
<p>In some cases, it doesn't seem enough.
For some reason multiple repositories might have all those fields exactly the same and they differ in variants definition only. Such are not proper distribution trees, the majority do not have any images associated.</p>
<p>Examples brought by always helpful @gdve from <a href="https://pulp.plan.io/issues/8566#note-33" class="external">https://pulp.plan.io/issues/8566#note-33</a>:</p>
<pre><code>CentOS/8-stream/AppStream/x86_64/os/.treeinfo:build_timestamp = 1625615144
CentOS/8-stream/BaseOS/x86_64/os/.treeinfo:build_timestamp = 1625615155
CentOS/8-stream/HighAvailability/x86_64/os/.treeinfo:build_timestamp = 1625026406
CentOS/8-stream/PowerTools/x86_64/os/.treeinfo:build_timestamp = 1625026406
CentOS/8-stream/RT/x86_64/os/.treeinfo:build_timestamp = 1625026406
</code></pre>
<pre><code>AlmaLinux/8.4/AppStream/x86_64/kickstart/.treeinfo:build_timestamp = 1622014553
AlmaLinux/8.4/AppStream/x86_64/os/.treeinfo:build_timestamp = 1622014553
AlmaLinux/8.4/BaseOS/x86_64/kickstart/.treeinfo:build_timestamp = 1622014553
AlmaLinux/8.4/BaseOS/x86_64/os/.treeinfo:build_timestamp = 1622014553
AlmaLinux/8.4/HighAvailability/x86_64/kickstart/.treeinfo:build_timestamp = 1622014558
AlmaLinux/8.4/HighAvailability/x86_64/os/.treeinfo:build_timestamp = 1622014558
AlmaLinux/8.4/PowerTools/x86_64/kickstart/.treeinfo:build_timestamp = 1622014558
AlmaLinux/8.4/PowerTools/x86_64/os/.treeinfo:build_timestamp = 1622014558
AlmaLinux/8.4/extras/x86_64/kickstart/.treeinfo:build_timestamp = 1622014558
AlmaLinux/8.4/extras/x86_64/os/.treeinfo:build_timestamp = 1622014558
</code></pre>
<p>Pulp 2to3 migration fails with <code>No declared artifact with relative path ".treeinfo" for content "<DistributionTree: pk=64f44866-0207-4005-9c06-0f45e52cbdd1>"</code>.
I would expect sync to behave the similarly, needs testing though.</p> Pulp - Task #9575 (CLOSED - WONTFIX): Move queryset filtering for Tasks to Task.objectshttps://pulp.plan.io/issues/95752021-11-18T21:06:16Zbmbouterbmbouter@redhat.com
<p>Right now the Task queryset scoping is implemented only in DRF. Recently the new task purge endpoint deletes task records in a task. To do that in a way that aligns with the queryset scoping and RBAC the Tasks endpoint already supports we have to have the queryset manager provide the queryset scoping.</p> Container Support - Task #9572 (CLOSED - DUPLICATE): Port the RBAC implementation to the pulpcore...https://pulp.plan.io/issues/95722021-11-16T13:48:53Zmdellweg
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_container/508":<a href="https://github.com/pulp/pulp_container/issues/508" class="external">https://github.com/pulp/pulp_container/issues/508</a></p>
<hr>
<p>Start with a PoC PR to get the pulpcore PR merged first.</p>
<p>Once that is done, we need to write a data migration that will look for the autogenerated groups and translate them into user_object_roles.
To be discussed: Look if we can be clever with global permissions too.</p> Container Support - Story #9509 (CLOSED - DUPLICATE): As a user I can sign container image by pro...https://pulp.plan.io/issues/95092021-10-14T10:14:23Zipanova@redhat.comipanova@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_container/500":<a href="https://github.com/pulp/pulp_container/issues/500" class="external">https://github.com/pulp/pulp_container/issues/500</a></p>
<hr>
<p>As a result signature will be created and saved into the Pulp Container Registry Sigstore</p>
<p>It should be possible to sign whole repo, list of images or a specific image only.</p>
<p>See <a href="https://github.com/containers/skopeo/blob/main/docs/skopeo-standalone-sign.1.md" class="external">https://github.com/containers/skopeo/blob/main/docs/skopeo-standalone-sign.1.md</a> and <a href="https://github.com/containers/image/blob/main/docs/containers-signature.5.md#json-data-format" class="external">https://github.com/containers/image/blob/main/docs/containers-signature.5.md#json-data-format</a></p> Container Support - Story #9507 (CLOSED - DUPLICATE): As a user I can sync container image with i...https://pulp.plan.io/issues/95072021-10-14T10:12:04Zipanova@redhat.comipanova@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_container/498":<a href="https://github.com/pulp/pulp_container/issues/498" class="external">https://github.com/pulp/pulp_container/issues/498</a></p>
<hr>
<p>Synced signatures are not verified. Signature verification is offloaded to the client.</p>
<p>If <code>with-signature=True</code> ( TBD agree on naming) was specified only those images that have signature with be mirrored. Rest of the images will be skipped. <code>Signature server location</code> needs to be specified ( it's the webserver location of signatures, for example <a href="https://registry.redhat.io/containers/sigstore" class="external">https://registry.redhat.io/containers/sigstore</a>).
Otherwise it will be assumed that the signature is stored on the remote registry in a form of manifest or a separate object:</p>
<ul>
<li>If it is a separate object, signature API extensions should be available on the registry and <code>X-Registry-Supports-Signatures</code> header will identify that <a href="https://github.com/containers/image/blob/main/docs/signature-protocols.md#openshift-dockerdistribution-api-extension" class="external">https://github.com/containers/image/blob/main/docs/signature-protocols.md#openshift-dockerdistribution-api-extension</a>
</li>
<li>If it is stored in a form of a manifest then it most likely was signed with cosign <a href="https://github.com/SigStore/cosign#signing-subjects" class="external">https://github.com/SigStore/cosign#signing-subjects</a>. <strong>we will not support this for now</strong>
</li>
</ul>
<p><strong>Q:</strong> does podman/skopeo support verification of cosign signature type? Only atomic type for now <a href="https://github.com/containers/image/blob/main/docs/containers-signature.5.md#criticaltype" class="external">https://github.com/containers/image/blob/main/docs/containers-signature.5.md#criticaltype</a></p>
<p><strong>Q:</strong> how this will work with mirror=True?</p> RPM Support - Issue #9406 (CLOSED - NOTABUG): Trivial OOM on sync for a particular Microsoft repohttps://pulp.plan.io/issues/94062021-09-16T03:33:52Zdalleydalley@redhat.com
<p>Syncing <a href="https://packages.microsoft.com/rhel/8/prod/" class="external">https://packages.microsoft.com/rhel/8/prod/</a> results in an OOM even on a box with ~10gb of available memory</p> RPM Support - Issue #9399 (CLOSED - NOTABUG): sync error: invalid memory alloc request sizehttps://pulp.plan.io/issues/93992021-09-15T09:20:51Zkeilr
<p>pulp version: container image docker.io/pulp/pulp:3.15</p>
<p>remote repository definition:</p>
<pre><code class="text syntaxhl" data-language="text">$ pulp --base-url http://localhost:8080 --username admin --password secret rpm remote show --name "packages-microsoft-com-prod-rhel8"
{
"pulp_href": "/pulp/api/v3/remotes/rpm/rpm/c455d98d-4c8c-446d-aa4d-dffe568675d6/",
"pulp_created": "2021-09-14T14:51:03.146140Z",
"name": "packages-microsoft-com-prod-rhel8",
"url": "https://packages.microsoft.com/rhel/8/prod/",
"ca_cert": null,
"client_cert": null,
"tls_validation": true,
"proxy_url": "http://proxy.example.com:8080",
"pulp_labels": {},
"pulp_last_updated": "2021-09-14T14:51:03.146171Z",
"download_concurrency": null,
"max_retries": null,
"policy": "immediate",
"total_timeout": null,
"connect_timeout": null,
"sock_connect_timeout": null,
"sock_read_timeout": null,
"headers": null,
"rate_limit": null,
"sles_auth_token": null
}
</code></pre>
<p>error:</p>
<pre><code class="text syntaxhl" data-language="text">$ podman logs --follow pulp
pulp [505c8f64043741a7b8f09eac46fa8331]: pulpcore.tasking.pulpcore_worker:INFO: Task a7a26b28-9251-4da1-82ef-19272b6779f0 failed (invalid memory alloc request size 1073741824
)
pulp [505c8f64043741a7b8f09eac46fa8331]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/local/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 323, in _perform_task
result = func(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 471, in synchronize
version = dv.create()
File "/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
loop.run_until_complete(pipeline)
File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
await asyncio.gather(*futures)
File "/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
await self.run()
File "/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/content_stages.py", line 174, in run
await sync_to_async(process_batch)()
File "/usr/local/lib/python3.8/site-packages/asgiref/sync.py", line 444, in __call__
ret = await asyncio.wait_for(future, timeout=None)
File "/usr/lib64/python3.8/asyncio/tasks.py", line 455, in wait_for
return await fut
File "/usr/lib64/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.8/site-packages/asgiref/sync.py", line 486, in thread_handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/content_stages.py", line 122, in process_batch
d_content.content.save()
File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/base.py", line 149, in save
return super().save(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/django_lifecycle/mixins.py", line 134, in save
save(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 726, in save
self.save_base(using=using, force_insert=force_insert,
File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 763, in save_base
updated = self._save_table(
File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 868, in _save_table
results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 906, in _do_insert
return manager._insert(
File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1270, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
</code></pre>
<p>All other repo sync jobs work without problems. (e.g. official rhel(7|8) repos, epel, ...)</p>
<p>Is anyone able to reproduce this MS repo issue?</p> RPM Support - Issue #9337 (CLOSED - DUPLICATE): Dependency solving does not pull all stream depen...https://pulp.plan.io/issues/93372021-09-02T15:00:08Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2299":<a href="https://github.com/pulp/pulp_rpm/issues/2299" class="external">https://github.com/pulp/pulp_rpm/issues/2299</a></p> RPM Support - Issue #9331 (CLOSED - DUPLICATE): Dependency solver takes an extremely long time to...https://pulp.plan.io/issues/93312021-09-02T03:01:50Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2298":<a href="https://github.com/pulp/pulp_rpm/issues/2298" class="external">https://github.com/pulp/pulp_rpm/issues/2298</a></p> RPM Support - Refactor #9309 (CLOSED - DUPLICATE): Add support for new memory-efficient createrep...https://pulp.plan.io/issues/93092021-08-30T03:50:56Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2296":<a href="https://github.com/pulp/pulp_rpm/issues/2296" class="external">https://github.com/pulp/pulp_rpm/issues/2296</a></p> RPM Support - Issue #8967 (CLOSED - DUPLICATE): "duplicate key value violates unique constraint" ...https://pulp.plan.io/issues/89672021-06-24T13:23:18Zwilful
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2278":<a href="https://github.com/pulp/pulp_rpm/issues/2278" class="external">https://github.com/pulp/pulp_rpm/issues/2278</a></p>
<hr>
<p>The original issue is difficult to reproduce any longer, but there are similar issues which can be. see <a href="https://pulp.plan.io/issues/8967#note-16" class="external">https://pulp.plan.io/issues/8967#note-16</a></p>
<p>========================</p>
<p>Hi for all!</p>
<p>Me need added for pulp server two repositories:</p>
<p><a href="http://downloads.linux.hpe.com/SDR/repo/spp/redhat/7/x86_64/current/" class="external">http://downloads.linux.hpe.com/SDR/repo/spp/redhat/7/x86_64/current/</a></p>
<p><a href="http://downloads.linux.hpe.com/SDR/repo/mcp/CentOS/7/x86_64/current/" class="external">http://downloads.linux.hpe.com/SDR/repo/mcp/CentOS/7/x86_64/current/</a></p>
<p>But i can't do it, becouse:</p>
<pre><code class="text syntaxhl" data-language="text"> "description": "duplicate key value violates unique constraint \"rpm_package_pkgId_key\"\nDETAIL: Key (\"pkgId\")=(ebf96fb31b880280a25d07c596bde204df50d140) already exists.\
n"
</code></pre>
<p>How can I find out in which repository this package is?</p>