Issue #7190
closedPulp 3 - pulp-deb : APT client installing same patches again and again (field Multi-Arch)
Added by swisscom over 4 years ago. Updated over 3 years ago.
Description
Dear support,
We have the same issue than reported in https://pulp.plan.io/issues/6982 but for field Multi-Arch (was fixed for field Installed-Size in https://github.com/pulp/pulp_deb/pull/184)
Would it be possible to fix missing Multi-Arch field ? And in addition if possible not put SHA1 field in Packages file when SHA1 value is empty...
Pulp components version :
(pulp) -bash-4.2$ pip list | grep pulp
pulp-deb 2.6.0b1.dev0 /var/lib/pulp/pulp_deb
pulp-rpm 3.4.2
pulpcore 3.5.0
Thanks in advance
Related issues
Updated by quba42 over 4 years ago
Thanks for opening these bug reports.
Rather than playing wack-a-mole with these issues, I think we need to systematically go through all well defined control file fields, and check whether they are handled correctly by the pulp_deb plugin.
Here, "handled correctly" includes making exactly those fields "mandatory" that officially are so, and of course ensuring that fields that are present in an upstream repository will also be present in our pulp_deb publications!
I would like to solve the whole class of issues, rather than just this individual case.
Updated by swisscom over 4 years ago
Thanks quba42 for your updatre
My preferred approach is to integrate all possible fields (except ones not present for the concerned package in upstream repository) in order to be able to publish an identical Packages content for the concerned packages. Most of the fields are described here : https://www.debian.org/doc/debian-policy/ch-controlfields.html However, Multi-Arch field is not present in this specification. More details abour this field here : https://wiki.ubuntu.com/MultiarchSpec
I understand that Debian documentation is inconsitent about content of Packages files. I can try to open a case to them if you want
Hint i you'd like to check by yourself if requrired fields are missing :
- Setup a repo, sync it with remote, create distribution and publication
- Connect your apt client to this newly created publication and patch your system using it
- You will see that same packages apply forever
- Play around with files in /var/lib/apt/lists/_Packages by adding content you find on the upstream repo Packages file
- You will see that when you run apt-get upgrade, the packages for which you modified attributes in /var/lib/apt/lists/.... based on upstream repository content will not be proposed anymore for upgrade as apt will be able to identify this package is already up-to-date
Let me know if you'd like to have more details :)
Thanks a lot for your support
Updated by swisscom over 4 years ago
I think it could be interesting to add to your test fixtures the following scenario :
- After publishing a new distribution, patch your debian or ubuntu client
- After patching, apt client must say there are no more applicable updates
Updated by pulpbot over 4 years ago
- Status changed from NEW to POST
Updated by swisscom over 4 years ago
Hi, I applied the fix on my Pulp server. Now the repo sync fails with the following error :
"description": "value too long for type character varying(255)\n",
"traceback": " File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 883, in perform_job\n rv = job.perform()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 657, in perform\n self._result = self._execute()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 663, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/var/lib/pulp/pulp_deb/pulp_deb/app/tasks/synchronizing.py\", line 106, in synchronize\n DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 148, in create\n loop.run_until_complete(pipeline)\n File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n return future.result()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n await asyncio.gather(*futures)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py\", line 102, in run\n d_content.content.save()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/base.py\", line 110, in save\n return super().save(*args, **kwargs)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py\", line 741, in save\n force_update=force_update, update_fields=update_fields)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py\", line 779, in save_base\n force_update, using, update_fields,\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py\", line 870, in _save_table\n result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py\", line 908, in _do_insert\n using=using, raw=raw)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/manager.py\", line 82, in manager_method\n return getattr(self.get_queryset(), name)(*args, **kwargs)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/query.py\", line 1186, in _insert\n return query.get_compiler(using=using).execute_sql(return_id)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/sql/compiler.py\", line 1375, in execute_sql\n cursor.execute(sql, params)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 67, in execute\n return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 76, in _execute_with_wrappers\n return executor(sql, params, many, context)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 84, in _execute\n return self.cursor.execute(sql, params)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/utils.py\", line 89, in __exit__\n raise dj_exc_value.with_traceback(traceback) from exc_value\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 84, in _execute\n return self.cursor.execute(sql, params)\n"
Details about the remote used for syncing :
{
"architectures": "amd64",
"ca_cert": null,
"client_cert": null,
"client_key": null,
"components": "main contrib",
"distributions": "buster bullseye buster-updates bullseye-updates",
"download_concurrency": 20,
"gpgkey": null,
"name": "debian",
"password": null,
"policy": "on_demand",
"proxy_url": "http://10.2.5.69:3128",
"pulp_created": "2020-07-24T09:23:25.241744Z",
"pulp_href": "/pulp/api/v3/remotes/deb/apt/a3036ec8-8c0c-408d-a493-007668ee4ad9/",
"pulp_last_updated": "2020-07-24T09:25:29.131085Z",
"sync_installer": false,
"sync_sources": false,
"sync_udebs": false,
"tls_validation": true,
"url": "http://ftp.ch.debian.org/debian/",
"username": null
}
Added by msinghal over 4 years ago
Updated by quba42 over 4 years ago
@swisscom We had the same issue, and fixed it under a separate issue: https://pulp.plan.io/issues/7257
I expect both fixes will be merged into master today.
However, there is one more related/similar issue (https://pulp.plan.io/issues/7236). The remaining problem is that the pulp_deb plugin is currently declaring some fields as mandatory even though Debian says they are only "recommended". This causes failed syncs when there are packages that do not provide those fields.
Once all three of these issues are fixed, it is my hope that we will have fixed all (or at least the vast majority) of "There is some package control field we are not handling correctly" type issues...
Updated by quba42 over 4 years ago
- Related to Issue #7266: Add tests to ensure all control file fields present in an upstream package index will be present in pulp_deb added
Updated by quba42 over 4 years ago
- Related to Issue #7257: Many pulp_deb control file fields are limited to 255 characters but should not be added
Updated by quba42 over 4 years ago
- Related to Issue #7236: Sync fails if packages do not have a 'section' field added
Updated by msinghal over 4 years ago
- Status changed from POST to MODIFIED
Applied in changeset bcc790c8d6a2e1801b00dee0d787c9b2451bcb5e.
Updated by quba42 over 4 years ago
- Sprint/Milestone changed from Katello to 2.6.0b1
Updated by quba42 over 3 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Add missing fields in Package file
fixes #7190 https://pulp.plan.io/issues/7190