Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-12-21T17:23:07ZPulp
Planio RPM Support - Issue #9655 (CLOSED - DUPLICATE): requesting lots of packages with a subset of fiel...https://pulp.plan.io/issues/96552021-12-21T17:23:07Zjsherril@redhat.comjsherril@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2312":<a href="https://github.com/pulp/pulp_rpm/issues/2312" class="external">https://github.com/pulp/pulp_rpm/issues/2312</a></p>
<hr>
<p>When we make a request like this:</p>
<pre><code>/pulp/api/v3/content/rpm/packages/?fields=pulp_href%2Cname%2Cversion%2Crelease%2Carch%2Cepoch%2Csummary%2Cis_modular%2Crpm_sourcerpm%2Clocation_href%2CpkgId&limit=1&offset=1000&repository_version=%2Fpulp%2Fapi%2Fv3%2Frepositories%2Frpm%2Frpm%2Fc6b93206-22bb-4c58-ba07-828c32326330%2Fversions%2F1%2F
</code></pre>
<p>where some subset of fields is requested, we're still seeing all of the fields being loaded from the database. The result is that the query takes much much longer than it should</p>
<p>Testing on the rhel7 repo, it took about ~100 seconds on my machine to fetch all 32K packages specifying the fields above. If I comment out the filelists, changelogs, provides/requires from the model and serializers, that drops down to ~33 seconds. This is a huge improvement.</p>
<p>Reading this: <a href="https://stackoverflow.com/questions/53319787/how-can-i-select-specific-fields-in-django-rest-framework" class="external">https://stackoverflow.com/questions/53319787/how-can-i-select-specific-fields-in-django-rest-framework</a></p>
<p>it looks like its may be possible to exclude certain fields from the query itself fairly easily.</p> RPM Support - Issue #9651 (CLOSED - DUPLICATE): Sync creates publication but no new repository ve...https://pulp.plan.io/issues/96512021-12-21T09:37:32Zmgoddard
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2311":<a href="https://github.com/pulp/pulp_rpm/issues/2311" class="external">https://github.com/pulp/pulp_rpm/issues/2311</a></p>
<hr>
<p>I have a nightly job that uses Ansible Squeezer modules to synchronise, publish and distribute some repositories. Every few days I hit an error like this:</p>
<pre><code>Found multiple matches for publication ({'repository_version': '/pulp/api/v3/repositories/rpm/rpm/b9eb112b-28c3-46bc-a40e-594531b54925/versions/4/
</code></pre>
<p>I have verified that this is the case. There is one publication created after the last successful sync, and another created today for the same version.</p>
<pre><code> pulp rpm publication list --repository-version /pulp/api/v3/repositories/rpm/rpm/b9eb112b-28c3-46bc-a40e-594531b54925/versions/4/
[
{
"pulp_href": "/pulp/api/v3/publications/rpm/rpm/21680308-1fc4-4bea-a5fc-1e3c609533f1/",
"pulp_created": "2021-12-21T02:31:44.243390Z",
"repository_version": "/pulp/api/v3/repositories/rpm/rpm/b9eb112b-28c3-46bc-a40e-594531b54925/versions/4/",
"repository": "/pulp/api/v3/repositories/rpm/rpm/b9eb112b-28c3-46bc-a40e-594531b54925/",
"metadata_checksum_type": "unknown",
"package_checksum_type": "unknown",
"gpgcheck": 0,
"repo_gpgcheck": 1,
"sqlite_metadata": true
},
{
"pulp_href": "/pulp/api/v3/publications/rpm/rpm/4a0c6a75-bb86-4b96-bb6c-6d0f08763847/",
"pulp_created": "2021-12-15T02:31:07.260722Z",
"repository_version": "/pulp/api/v3/repositories/rpm/rpm/b9eb112b-28c3-46bc-a40e-594531b54925/versions/4/",
"repository": "/pulp/api/v3/repositories/rpm/rpm/b9eb112b-28c3-46bc-a40e-594531b54925/",
"metadata_checksum_type": "unknown",
"package_checksum_type": "unknown",
"gpgcheck": 0,
"repo_gpgcheck": 1,
"sqlite_metadata": true
}
]
</code></pre>
<p>I checked the sync task from today, and it completed successfully. However, it lists the new publication as a created resource, but no new repo version.</p>
<pre><code> {
"pulp_href": "/pulp/api/v3/tasks/5e132510-89cb-4224-9966-d1f22d49a4e1/",
"pulp_created": "2021-12-21T02:30:53.723988Z",
"state": "completed",
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"logging_cid": "0a4dc729907842aaa5ba9605e418cdd4",
"started_at": "2021-12-21T02:30:53.801940Z",
"finished_at": "2021-12-21T02:31:44.897653Z",
"error": null,
"worker": "/pulp/api/v3/workers/605f92b7-9b71-4039-a3de-0af017d86651/",
"parent_task": null,
"child_tasks": [],
"task_group": null,
"progress_reports": [
{
"message": "Downloading Metadata Files",
"code": "sync.downloading.metadata",
"state": "completed",
"total": null,
"done": 8,
"suffix": null
},
{
"message": "Downloading Artifacts",
"code": "sync.downloading.artifacts",
"state": "completed",
"total": null,
"done": 297,
"suffix": null
},
{
"message": "Associating Content",
"code": "associating.content",
"state": "completed",
"total": null,
"done": 0,
"suffix": null
},
{
"message": "Parsed Packages",
"code": "sync.parsing.packages",
"state": "completed",
"total": null,
"done": 299,
"suffix": null
},
{
"message": "Un-Associating Content",
"code": "unassociating.content",
"state": "completed",
"total": null,
"done": 0,
"suffix": null
}
],
"created_resources": [
"/pulp/api/v3/publications/rpm/rpm/21680308-1fc4-4bea-a5fc-1e3c609533f1/"
],
"reserved_resources_record": [
"/pulp/api/v3/repositories/rpm/rpm/b9eb112b-28c3-46bc-a40e-594531b54925/",
"shared:/pulp/api/v3/remotes/rpm/rpm/7b6bc03e-787e-4266-ba33-425c4f9e540b/"
]
},
</code></pre>
<p>Comparing with another sync task, I see a repository version listed in the created_resources instead.</p>
<p>Here is one of the affected repos:</p>
<pre><code>{
"pulp_href": "/pulp/api/v3/repositories/rpm/rpm/b9eb112b-28c3-46bc-a40e-594531b54925/",
"pulp_created": "2021-11-19T13:21:20.971989Z",
"versions_href": "/pulp/api/v3/repositories/rpm/rpm/b9eb112b-28c3-46bc-a40e-594531b54925/versions/",
"pulp_labels": {},
"latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/b9eb112b-28c3-46bc-a40e-594531b54925/versions/4/",
"name": "CentOS Stream 8 - NFV OpenvSwitch",
"description": null,
"retain_repo_versions": null,
"remote": null,
"autopublish": false,
"metadata_signing_service": null,
"retain_package_versions": 0,
"metadata_checksum_type": null,
"package_checksum_type": null,
"gpgcheck": 0,
"repo_gpgcheck": 0,
"sqlite_metadata": false
}
</code></pre>
<p>And the corresponding remote:</p>
<pre><code> {
"pulp_href": "/pulp/api/v3/remotes/rpm/rpm/7b6bc03e-787e-4266-ba33-425c4f9e540b/",
"pulp_created": "2021-11-19T13:21:41.147452Z",
"name": "CentOS Stream 8 - NFV OpenvSwitch-remote",
"url": "http://mirrorlist.centos.org/?release=8-stream&arch=x86_64&repo=nfv-openvswitch-2",
"ca_cert": null,
"client_cert": null,
"tls_validation": true,
"proxy_url": null,
"pulp_labels": {},
"pulp_last_updated": "2021-11-19T13:21:41.147492Z",
"download_concurrency": null,
"max_retries": null,
"policy": "immediate",
"total_timeout": null,
"connect_timeout": null,
"sock_connect_timeout": null,
"sock_read_timeout": null,
"headers": null,
"rate_limit": null,
"sles_auth_token": null
},
</code></pre>
<p>I'm using <code>policy: immediate</code> and <code>sync_policy: mirror_complete</code> when syncing.</p>
<p>Versions:</p>
<pre><code> {
"component": "core",
"version": "3.16.0"
},
{
"component": "rpm",
"version": "3.16.1"
},
{
"component": "file",
"version": "1.10.1"
},
{
"component": "deb",
"version": "2.16.0"
},
{
"component": "container",
"version": "2.9.0"
},
{
"component": "certguard",
"version": "1.5.1"
}
</code></pre> RPM Support - Issue #9647 (CLOSED - DUPLICATE): ULN-remotes HTTP-Proxy use brokenhttps://pulp.plan.io/issues/96472021-12-20T09:33:07Zmbucher
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2322":<a href="https://github.com/pulp/pulp_rpm/issues/2322" class="external">https://github.com/pulp/pulp_rpm/issues/2322</a></p>
<hr>
<p>Using ULN-Remotes, using HTTP-proxy does not work.</p>
<pre><code>Cannot connect to host linux-update.oracle.com:443 ssl:default [Connect call failed ('138.1.51.46', 443)
</code></pre> RPM Support - Issue #9636 (CLOSED - DUPLICATE): FileNotFoundError: [Errno 2] No such file or dire...https://pulp.plan.io/issues/96362021-12-13T17:05:32Zttereshcttereshc@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2321":<a href="https://github.com/pulp/pulp_rpm/issues/2321" class="external">https://github.com/pulp/pulp_rpm/issues/2321</a></p>
<hr>
<p>Pulp expects custom metadata files to have a specific format for filenames: -.</p>
<p>During pulp-2to3-migration, users get <code>FileNotFoundError: [Errno 2] No such file or directory: ' '</code> because the filename is only a checksum :/, which we remove to determine the filename, so the path we detect becomes an empty string.</p>
<pre><code>Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 478, in migrate_repo_distributor
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: pulp2dist, repo_version)
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 91, in migrate_to_pulp3
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite)
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 344, in publish
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: publication_data.populate()
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 253, in populate
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: self.repomdrecords = self.prepare_metadata_files(main_content)
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 99, in prepare_metadata_files
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: with open(path, "wb") as new_file:
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: FileNotFoundError: [Errno 2] No such file or directory: ''
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-1: pulp: rq.worker:INFO: 27738@1002a1103081001.xxx.com: c0d58c5a-9ff1-4d40-bbb3-9c24fdf0fdb4
Nov 22 21:31:34 1002a1103081001 pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: 43f43c1a-d09e-46bd-99fe-73b6e2ee397c
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-3: pulp: rq.worker:INFO: 27740@1002a1103081001.xxx.com: Job OK (b25f3fa2-401c-425c-92eb-b49a61415617)
Nov 22 21:31:34 1002a1103081001 pulpcore-worker-2: pulp: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=XXX-Red_Hat_Ansible_Engine_2_7_RPMs_for_Red_Hat_Enterprise_Linux_7_Server_x86_64, version=1
</code></pre>
<p>Some repositories like <a href="https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/repodata/repomd.xml" class="external">https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/repodata/repomd.xml</a> have filenames in the form of UUID.</p>
<p>Observed with pulp2 to pulp3 migration only so far, need to test sync.</p> RPM Support - Task #9633 (CLOSED - DUPLICATE): Use repo priorities in the dependency solverhttps://pulp.plan.io/issues/96332021-12-12T06:09:47Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2309":<a href="https://github.com/pulp/pulp_rpm/issues/2309" class="external">https://github.com/pulp/pulp_rpm/issues/2309</a></p>
<hr>
<p>We ought to be setting repo priorities such that for every set of copies, any matching RPMs present in the same repository are prioritzed over ones in other repos.</p> RPM Support - Issue #9631 (CLOSED - DUPLICATE): aiohttp.client_exceptions.ClientPayloadErrorhttps://pulp.plan.io/issues/96312021-12-10T13:17:21Zjanr7
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2308":<a href="https://github.com/pulp/pulp_rpm/issues/2308" class="external">https://github.com/pulp/pulp_rpm/issues/2308</a></p>
<hr>
<p>Hi All</p>
<p>Pulp 3.16.1 on docker.</p>
<p>The Sync from remote never completes.
If you could please assist.</p>
<p>We are seeing Ansible error:</p>
<pre><code>An exception occurred during task execution. To see the full traceback, use -vvv. The error was: Exception: Task failed to complete. (failed; A file located at the url https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/src/kernel-source-4.12.14-122.98.1.src.rpm failed validation due to checksum.)
failed: [localhost] (item={'name': 'SLES12-SP5-Updates', 'backend': 'https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/', 'base': 'SLE/SLES12/SP5/updates'}) => {"ansible_index_var": "s_idx", "ansible_loop_var": "item", "changed": false, "item": {"backend": "https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/", "base": "SLE/SLES12/SP5/updates", "name": "SLES12-SP5-Updates"}, "msg": "Task failed to complete. (failed; A file located at the url https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/src/kernel-source-4.12.14-122.98.1.src.rpm failed validation due to checksum.)", "s_idx": 3}
</code></pre>
<p>On remote: rmt.abcgroup.net</p>
<a name="sysctl-agrep-aio"></a>
<h1 >sysctl -a|grep aio<a href="#sysctl-agrep-aio" class="wiki-anchor">¶</a></h1>
<p>fs.aio-max-nr = 65536
fs.aio-nr = 2080</p>
<p>On pulp server</p>
<a name="sysctl-agrep-aio-2"></a>
<h1 >sysctl -a|grep aio<a href="#sysctl-agrep-aio-2" class="wiki-anchor">¶</a></h1>
<p>fs.aio-max-nr = 65536
fs.aio-nr = 2</p>
<p>================
Attached is the Docker pulp log.
docker logs pulp</p>
<p>aio reported errors for various syncs(tries) of the same repo.</p>
<p>The sha256 for the RPMs are correct on the remote server's repodata.</p>
<pre><code>Backing off download_wrapper(...) for 0.7s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [6efc07543b0c4821922029cf5cda3e90]: backoff:INFO: Backing off download_wrapper(...) for 0.7s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 3.5s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [6efc07543b0c4821922029cf5cda3e90]: backoff:INFO: Backing off download_wrapper(...) for 3.5s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 0.4s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 0.4s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 0.3s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 0.3s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 1.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 1.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 0.9s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 0.9s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 3.1s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 3.1s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 1.1s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 1.1s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 1.6s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 1.6s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 2.5s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 2.5s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 0.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 0.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 0.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 0.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 0.9s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 0.9s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 0.4s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 0.4s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 3.8s (aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected)
pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 3.8s (aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected)
Backing off download_wrapper(...) for 1.2s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 1.2s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 6.6s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 6.6s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Backing off download_wrapper(...) for 6.3s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 6.3s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
Giving up download_wrapper(...) after 5 tries (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:ERROR: Giving up download_wrapper(...) after 5 tries (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed)
</code></pre>
<hr>
<p>The failures reported soon after the reported aio errors: (see attached log file)</p>
<pre><code>pulp [2fa84b10f57c41f39635eddffc89ee0a]: pulpcore.tasking.pulpcore_worker:INFO: Task 086ebba6-1f81-471c-9f60-47eda47969a6 failed ()
pulp [2fa84b10f57c41f39635eddffc89ee0a]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/local/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 370, in _perform_task
--
pulp [6efc07543b0c4821922029cf5cda3e90]: pulpcore.tasking.pulpcore_worker:INFO: Task c375f81f-7b64-4356-983b-b8729048094f failed (A file located at the url https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/noarch/kernel-source-azure-4.12.14-16.76.2.noarch.rpm failed validation due to checksum.)
pulp [6efc07543b0c4821922029cf5cda3e90]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/local/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 370, in _perform_task
</code></pre>
<a name="--"></a>
<h2 >--<a href="#--" class="wiki-anchor">¶</a></h2>
<p>Installation information:</p>
<pre><code>pulp rpm remote show --name "remote_SLES12-SP5-Updates"
"download_concurrency": 2, (tried 1, 4, 10 - same result)
</code></pre>
<p>pulp --version
pulp3 command line interface, version 0.12.0</p>
<p>pip freeze
appdirs==1.4.3
asn1crypto==0.24.0
Babel==2.8.0
beautifulsoup4==4.10.0
bs4==0.0.1
certifi==2021.5.30
cffi==1.13.2
chardet==4.0.0
click==8.0.1
colorama==0.4.4
commonmark==0.9.1
contextlib2==21.6.0
cryptography==2.8
dataclasses==0.8
httpie==1.0.3
idna==2.10
importlib-metadata==4.6.1
iotop==0.6
isc==2.0
Jinja2==2.10.1
MarkupSafe==1.0
packaging==20.3
pip-search==0.0.7
ply==3.10
pulp-cli==0.12.0
py==1.8.1
pyasn1==0.4.2
pycparser==2.17
Pygments==2.6.1
pyOpenSSL==19.0.0
pyparsing==2.4.7
pytz==2020.5
PyYAML==5.4.1
requests==2.25.1
rich==10.15.2
schema==0.7.4
six==1.14.0
soupsieve==2.3.1
toml==0.10.2
typing-extensions==3.10.0.0
urllib3==1.26.6
zipp==3.5.0</p>
<p>pulp status
"versions": [
{
"component": "core",
"version": "3.16.1"
},
{
"component": "rpm",
"version": "3.16.1"
},
{
"component": "python",
"version": "3.5.2"
},
{
"component": "file",
"version": "1.10.1"
},
{
"component": "deb",
"version": "2.16.0"
},
{
"component": "container",
"version": "2.9.1"
},
{
"component": "certguard",
"version": "1.5.1"
},
{
"component": "ansible",
"version": "0.10.1"
}</p>
<p>Docker server</p>
<pre><code>cat /etc/os-release
NAME="SLES"
VERSION="15-SP3"
VERSION_ID="15.3"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3"
</code></pre>
<p>Thanks so much for the support.
Jan</p> RPM Support - Issue #9627 (MODIFIED): publish fails on MD5-checksummed repos, on FIPShttps://pulp.plan.io/issues/96272021-12-09T18:59:35Zggainey
<p>See associated BZ for details, reproducer</p> RPM Support - Test #9626 (CLOSED - DUPLICATE): Add tests for SHA repo to test_synchttps://pulp.plan.io/issues/96262021-12-08T21:43:04Zggainey
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2320":<a href="https://github.com/pulp/pulp_rpm/issues/2320" class="external">https://github.com/pulp/pulp_rpm/issues/2320</a></p>
<hr>
<p>We have been bitten a few times now by repos that use 'sha' (instead of 'sha1') checksums. Build a test for same that does <strong>not</strong> require sync'ing all of RHEL6.6...</p> RPM Support - Test #9622 (MODIFIED): Add a repo signed using 'sha' as alias for 'sha1'https://pulp.plan.io/issues/96222021-12-08T19:00:00Zggainey
<p>'sha' support exists in the wild, is the same as 'sha1', and has broken us several times now, Let's make it possible to write tests for it.</p> RPM Support - Issue #9619 (CLOSED - DUPLICATE): OpenAPI schema for ModulemdDefault is incorrecthttps://pulp.plan.io/issues/96192021-12-08T16:59:02Zdkliban@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2307":<a href="https://github.com/pulp/pulp_rpm/issues/2307" class="external">https://github.com/pulp/pulp_rpm/issues/2307</a></p>
<hr>
<p>The open api schema for ModulemdDefaults is incorrect. When trying to generate a client with openapi-generator-cli 5.3.0, the following exception is emitted:</p>
<pre><code>Exception in thread "main" java.lang.RuntimeException: Could not generate api file for 'ContentModulemdDefaults'
</code></pre> RPM Support - Issue #9605 (CLOSED - DUPLICATE): regression in pulp-rpm 3.14.6https://pulp.plan.io/issues/96052021-12-06T14:54:59Zvchepkov
<p>The original fix for issue <a href="https://pulp.plan.io/issues/8875" class="external">https://pulp.plan.io/issues/8875</a> fixed the problem, but recent changes broke functionality again</p>
<pre><code>Dec 6 05:05:50 foreman901 pulpcore-worker-6: pulp [19375194-2146-4216-8e9d-9c53e3188190]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('http://amazonlinux.us-east-1.amazonaws.com/blobstore/0f3098b2c2be0b1b3a4421f1187386bf35e1325faf11a3efff568caf06b1cd2e/java-11-amazon-corretto-headless-11.0.13+8-1.amzn2.x86_64.rpm'))
Dec 6 05:05:51 foreman901 pulpcore-worker-6: pulp [19375194-2146-4216-8e9d-9c53e3188190]: pulpcore.tasking.pulpcore_worker:INFO: Task 786abd82-c009-4ddc-b5ba-f10ecf3e0d97 failed (403, message='Forbidden', url=URL('http://amazonlinux.us-east-1.amazonaws.com/blobstore/0f3098b2c2be0b1b3a4421f1187386bf35e1325faf11a3efff568caf06b1cd2e/java-11-amazon-corretto-headless-11.0.13+8-1.amzn2.x86_64.rpm'))
</code></pre>
<p>downgrading from python3-pulp-rpm-3.14.6-2.el7.noarch to python3-pulp-rpm-3.14.3-1.el7.noarch restored successful synchronization</p> RPM Support - Issue #9587 (CLOSED - NOTABUG): manually upload rpm packageshttps://pulp.plan.io/issues/95872021-11-25T16:23:51ZTiagodCC
<p>Hello All</p>
<p>I am currently developing an Ansible playbook to upload custom RPM packages to specific pulp repositories. It uses the pulp.squeezer collection and some API calls.<br>
For this I have used the following documentations: <a href="https://docs.pulpproject.org/pulp_rpm/workflows/upload.html" class="external">https://docs.pulpproject.org/pulp_rpm/workflows/upload.html</a> & <a href="https://docs.pulpproject.org/pulp_rpm/restapi.html#operation/" class="external">https://docs.pulpproject.org/pulp_rpm/restapi.html#operation/</a></p>
<p>Following the upload of the file, the creation of rpm content and the association of the content with the desired repository, the change to the repository is published and distributed.<br>
However, when Pulp is invoked via the browser, it becomes apparent that the package is not displayed with the actual package name, but with its "location_href" from the metadata.</p>
<p>Below you can find a snippet from the metadata of one of the affected packages:</p>
<pre><code><package type="rpm">
<name>amsd</name>
<arch>x86_64</arch>
<version epoch="0" ver="2.5.0" rel="1675.24.rhel8"/>
<checksum type="sha256" pkgid="YES">554b4534999d9e0b778c8c1825f1ae098fe858b36daeaa5ac7c38b81804ffb4a</checksum>
<summary>Agentless Management Service</summary>
<description>This package contains the helper daemon that provides information for iLO5
embedded health and alerting.</description>
<packager>Hewlett-Packard Enterprise</packager>
<url>http://www.hpe.com/go/proliantlinux</url>
<time file="1637837726" build="1630381441"/>
<size package="3026011" installed="8620992" archive="8632796"/>
<location href="Packages/f/fcbdc174-f1f4-4277-8913-3e2ce843e028"/>
<format>
<rpm:license>MIT and BSD</rpm:license>
<rpm:vendor>Hewlett-Packard Enterprise</rpm:vendor>
<rpm:group>System Environment/Daemons</rpm:group>
<rpm:buildhost>ablrh7ex6416.sde.rdlabs.hpecorp.net</rpm:buildhost>
<rpm:sourcerpm>amsd-2.5.0-1675.24.rhel8.src.rpm</rpm:sourcerpm>
<rpm:header-range start="872" end="55032"/>
<rpm:provides>
<rpm:entry name="config(amsd)" flags="EQ" epoch="0" ver="2.5.0" rel="1675.24.rhel8"/>
<rpm:entry name="amsd(x86-64)" flags="EQ" epoch="0" ver="2.5.0" rel="1675.24.rhel8"/>
<rpm:entry name="amsd" flags="EQ" epoch="0" ver="2.5.0" rel="1675.24.rhel8"/>
</rpm:provides>
<rpm:requires>
<rpm:entry name="libc.so.6(GLIBC_2.17)(64bit)"/>
<rpm:entry name="/bin/sh"/>
<rpm:entry name="/bin/sh" pre="1"/>
<rpm:entry name="systemd"/>
<rpm:entry name="rtld(GNU_HASH)"/>
<rpm:entry name="pciutils"/>
<rpm:entry name="lsof"/>
<rpm:entry name="libsystemd.so.0(LIBSYSTEMD_209)(64bit)"/>
<rpm:entry name="libsystemd.so.0()(64bit)"/>
<rpm:entry name="libstdc++.so.6(GLIBCXX_3.4.21)(64bit)"/>
<rpm:entry name="libstdc++.so.6(GLIBCXX_3.4.20)(64bit)"/>
<rpm:entry name="libstdc++.so.6(GLIBCXX_3.4)(64bit)"/>
<rpm:entry name="libstdc++.so.6(CXXABI_1.3)(64bit)"/>
<rpm:entry name="libstdc++.so.6()(64bit)"/>
<rpm:entry name="librt.so.1(GLIBC_2.2.5)(64bit)"/>
<rpm:entry name="librt.so.1()(64bit)"/>
<rpm:entry name="librpmio.so.8()(64bit)"/>
<rpm:entry name="librpm.so.8()(64bit)"/>
<rpm:entry name="libpthread.so.0(GLIBC_2.2.5)(64bit)"/>
<rpm:entry name="libpthread.so.0()(64bit)"/>
<rpm:entry name="libm.so.6(GLIBC_2.2.5)(64bit)"/>
<rpm:entry name="libm.so.6()(64bit)"/>
<rpm:entry name="libjson-c.so.4()(64bit)"/>
<rpm:entry name="libgcc_s.so.1(GCC_3.0)(64bit)"/>
<rpm:entry name="libgcc_s.so.1()(64bit)"/>
<rpm:entry name="libdl.so.2(GLIBC_2.2.5)(64bit)"/>
<rpm:entry name="libdl.so.2()(64bit)"/>
<rpm:entry name="coreutils"/>
<rpm:entry name="bash"/>
<rpm:entry name="/bin/sh"/>
</rpm:requires>
<rpm:conflicts>
<rpm:entry name="hp-ams"/>
</rpm:conflicts>
<file>/sbin/smad_rev</file>
<file>/sbin/smad</file>
<file>/sbin/mr_cpqScsi_rev</file>
<file>/sbin/mr_cpqScsi</file>
<file>/sbin/cpqiScsi</file>
<file>/sbin/cpqScsi_rev</file>
<file>/sbin/cpqScsi</file>
<file>/sbin/cpqIde_rev</file>
<file>/sbin/cpqIde</file>
<file>/sbin/cpqFca_rev</file>
<file>/sbin/cpqFca</file>
<file>/sbin/amsd_rev</file>
<file>/sbin/amsd</file>
<file>/sbin/ahslog</file>
<file>/etc/sysconfig/smad_rev</file>
<file>/etc/sysconfig/smad</file>
<file>/etc/sysconfig/mr_cpqScsi_rev</file>
<file>/etc/sysconfig/mr_cpqScsi</file>
<file>/etc/sysconfig/cpqiScsi</file>
<file>/etc/sysconfig/cpqScsi_rev</file>
<file>/etc/sysconfig/cpqScsi</file>
<file>/etc/sysconfig/cpqIde_rev</file>
<file>/etc/sysconfig/cpqIde</file>
<file>/etc/sysconfig/cpqFca_rev</file>
<file>/etc/sysconfig/cpqFca</file>
<file>/etc/sysconfig/amsd_rev</file>
<file>/etc/sysconfig/amsd</file>
<file>/etc/sysconfig/ahslog</file>
</format>
</package>
</code></pre>
<p>The package is located at (as mentioned in the metadata). The desired place would look something like this: </p>
<p>The package can be found and installed via clients, but this makes it more difficult to understand.<br>
I unfortunately didn't find anything in the documentation about this.</p>
<p>I therefore wanted to ask what the reason for this is and how to fix it.</p>
<p>Thanks in advance for the answer.</p> RPM Support - Task #9585 (CLOSED - DUPLICATE): Sub-tree-only sync breaks our model of how syncs a...https://pulp.plan.io/issues/95852021-11-23T21:00:08Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2306":<a href="https://github.com/pulp/pulp_rpm/issues/2306" class="external">https://github.com/pulp/pulp_rpm/issues/2306</a></p>
<hr>
<p>See: <a href="https://pulp.plan.io/issues/9565" class="external">https://pulp.plan.io/issues/9565</a></p>
<p>We haven't considered the possibility of a repository having a .treeinfo but no top-level repodata. We need to investigate whether this breaks anything (for example: autopublish), and potentially add a new test fixture to exercise it.</p> RPM Support - Issue #9583 (CLOSED - DUPLICATE): Distribution tree uniqueness constraint is not en...https://pulp.plan.io/issues/95832021-11-23T17:43:48Zttereshcttereshc@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2305":<a href="https://github.com/pulp/pulp_rpm/issues/2305" class="external">https://github.com/pulp/pulp_rpm/issues/2305</a></p>
<hr>
<p>Currently <a href="https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models/distribution.py#L133" class="external">Pulp requires</a> the combination of the following fields to be unique:</p>
<pre><code> unique_together = (
"header_version",
"release_name",
"release_short",
"release_version",
"arch",
"build_timestamp",
)
</code></pre>
<p>In some cases, it doesn't seem enough.
For some reason multiple repositories might have all those fields exactly the same and they differ in variants definition only. Such are not proper distribution trees, the majority do not have any images associated.</p>
<p>Examples brought by always helpful @gdve from <a href="https://pulp.plan.io/issues/8566#note-33" class="external">https://pulp.plan.io/issues/8566#note-33</a>:</p>
<pre><code>CentOS/8-stream/AppStream/x86_64/os/.treeinfo:build_timestamp = 1625615144
CentOS/8-stream/BaseOS/x86_64/os/.treeinfo:build_timestamp = 1625615155
CentOS/8-stream/HighAvailability/x86_64/os/.treeinfo:build_timestamp = 1625026406
CentOS/8-stream/PowerTools/x86_64/os/.treeinfo:build_timestamp = 1625026406
CentOS/8-stream/RT/x86_64/os/.treeinfo:build_timestamp = 1625026406
</code></pre>
<pre><code>AlmaLinux/8.4/AppStream/x86_64/kickstart/.treeinfo:build_timestamp = 1622014553
AlmaLinux/8.4/AppStream/x86_64/os/.treeinfo:build_timestamp = 1622014553
AlmaLinux/8.4/BaseOS/x86_64/kickstart/.treeinfo:build_timestamp = 1622014553
AlmaLinux/8.4/BaseOS/x86_64/os/.treeinfo:build_timestamp = 1622014553
AlmaLinux/8.4/HighAvailability/x86_64/kickstart/.treeinfo:build_timestamp = 1622014558
AlmaLinux/8.4/HighAvailability/x86_64/os/.treeinfo:build_timestamp = 1622014558
AlmaLinux/8.4/PowerTools/x86_64/kickstart/.treeinfo:build_timestamp = 1622014558
AlmaLinux/8.4/PowerTools/x86_64/os/.treeinfo:build_timestamp = 1622014558
AlmaLinux/8.4/extras/x86_64/kickstart/.treeinfo:build_timestamp = 1622014558
AlmaLinux/8.4/extras/x86_64/os/.treeinfo:build_timestamp = 1622014558
</code></pre>
<p>Pulp 2to3 migration fails with <code>No declared artifact with relative path ".treeinfo" for content "<DistributionTree: pk=64f44866-0207-4005-9c06-0f45e52cbdd1>"</code>.
I would expect sync to behave the similarly, needs testing though.</p> RPM Support - Issue #9580 (CLOSED - DUPLICATE): Artifact() got an unexpected keyword argument 'sha'https://pulp.plan.io/issues/95802021-11-22T15:01:34Zttereshcttereshc@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2319":<a href="https://github.com/pulp/pulp_rpm/issues/2319" class="external">https://github.com/pulp/pulp_rpm/issues/2319</a></p>
<hr>
<p>From <a href="https://projects.theforeman.org/issues/33216" class="external">https://projects.theforeman.org/issues/33216</a>:</p>
<p>I recently ran into an issue while trying to sync a rhel 6.2 repo to my foreman server:</p>
<p>Artifact() got an unexpected keyword argument 'sha'</p>
<p>I saw a similar post here: <a href="https://projects.theforeman.org/issues/31559" class="external">https://projects.theforeman.org/issues/31559</a></p>
<p>The issue is marked as resolved for katello 3.18, yet I'm still having it with katello 4.1 and foreman 2.5.2</p>
<p>The error is as follow:</p>
<pre><code>error"=>
{"traceback"=>
" File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 266, in _perform_task\n" +
" result = func(*args, **kwargs)\n" +
" File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 422, in synchronize\n" +
" version = dv.create()\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 151, in create\n" +
" loop.run_until_complete(pipeline)\n" +
" File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n" +
" return future.result()\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n" +
" await asyncio.gather(*futures)\n" +
" File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n" +
" await self.run()\n" +
" File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 705, in run\n" +
" await self.parse_repository_metadata(repomd, repomd_files, file_extension)\n" +
" File \"/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 800, in parse_repository_metadata\n" +
" artifact=Artifact(**file_data),\n" +
" File \"/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py\", line 25, in __init__\n" +
" super().__init__(*args, **kwargs)\n" +
" File \"/usr/lib/python3.6/site-packages/django/db/models/base.py\", line 501, in __init__\n" +
" raise TypeError(\"%s() got an unexpected keyword argument '%s'\" % (cls.__name__, kwarg))\n",
"description"=>"Artifact() got an unexpected keyword argument 'sha'"},
</code></pre>