Project

Profile

Help

Issue #7190

Pulp 3 - pulp-deb : APT client installing same patches again and again (field Multi-Arch)

Added by swisscom 3 months ago. Updated about 2 months ago.

Status:
MODIFIED
Priority:
High
Assignee:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
4. Urgent
Version - Debian:
master
Platform Release:
Target Release - Debian:
OS:
CentOS 7
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Dear support,

We have the same issue than reported in https://pulp.plan.io/issues/6982 but for field Multi-Arch (was fixed for field Installed-Size in https://github.com/pulp/pulp_deb/pull/184)

Would it be possible to fix missing Multi-Arch field ? And in addition if possible not put SHA1 field in Packages file when SHA1 value is empty...

Pulp components version :

(pulp) -bash-4.2$ pip list | grep pulp
pulp-deb                        2.6.0b1.dev0 /var/lib/pulp/pulp_deb
pulp-rpm                        3.4.2
pulpcore                        3.5.0

Thanks in advance


Related issues

Related to Debian Support - Issue #7266: Add tests to ensure all control file fields present in an upstream package index will be present in pulp_debPOST<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to Debian Support - Issue #7257: Many pulp_deb control file fields are limited to 255 characters but should not beMODIFIED<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to Debian Support - Issue #7236: Sync fails if packages do not have a 'section' fieldMODIFIED<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision bcc790c8 View on GitHub
Added by msinghal 3 months ago

Add missing fields in Package file

fixes #7190 https://pulp.plan.io/issues/7190

History

#1 Updated by quba42 3 months ago

  • Checklist item Ensure exactly those control file fields are mandatory, that officially are so. added
  • Checklist item Ensure any control file fields present in an upstream repo will also be present in pulp_deb publications! added
  • Checklist item Add relevant assertions to our publication tests. added
  • Checklist item If needed expand our test fixtures with example packages that use a maximum set of control file fields. added

Thanks for opening these bug reports.

Rather than playing wack-a-mole with these issues, I think we need to systematically go through all well defined control file fields, and check whether they are handled correctly by the pulp_deb plugin.

Here, "handled correctly" includes making exactly those fields "mandatory" that officially are so, and of course ensuring that fields that are present in an upstream repository will also be present in our pulp_deb publications!

I would like to solve the whole class of issues, rather than just this individual case.

#2 Updated by swisscom 3 months ago

Thanks quba42 for your updatre

My preferred approach is to integrate all possible fields (except ones not present for the concerned package in upstream repository) in order to be able to publish an identical Packages content for the concerned packages. Most of the fields are described here : https://www.debian.org/doc/debian-policy/ch-controlfields.html However, Multi-Arch field is not present in this specification. More details abour this field here : https://wiki.ubuntu.com/MultiarchSpec

I understand that Debian documentation is inconsitent about content of Packages files. I can try to open a case to them if you want

Hint i you'd like to check by yourself if requrired fields are missing :

  1. Setup a repo, sync it with remote, create distribution and publication
  2. Connect your apt client to this newly created publication and patch your system using it
  3. You will see that same packages apply forever
  4. Play around with files in /var/lib/apt/lists/_Packages by adding content you find on the upstream repo Packages file
  5. You will see that when you run apt-get upgrade, the packages for which you modified attributes in /var/lib/apt/lists/.... based on upstream repository content will not be proposed anymore for upgrade as apt will be able to identify this package is already up-to-date

Let me know if you'd like to have more details :)

Thanks a lot for your support

#3 Updated by swisscom 3 months ago

I think it could be interesting to add to your test fixtures the following scenario :

  • After publishing a new distribution, patch your debian or ubuntu client
  • After patching, apt client must say there are no more applicable updates

#4 Updated by pulpbot 3 months ago

  • Status changed from NEW to POST

#5 Updated by quba42 3 months ago

  • Sprint/Milestone set to Katello

#6 Updated by swisscom 3 months ago

Hi, I applied the fix on my Pulp server. Now the repo sync fails with the following error :

 "description": "value too long for type character varying(255)\n",
                "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 883, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 657, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 663, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/var/lib/pulp/pulp_deb/pulp_deb/app/tasks/synchronizing.py\", line 106, in synchronize\n    DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 148, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py\", line 102, in run\n    d_content.content.save()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/base.py\", line 110, in save\n    return super().save(*args, **kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py\", line 741, in save\n    force_update=force_update, update_fields=update_fields)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py\", line 779, in save_base\n    force_update, using, update_fields,\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py\", line 870, in _save_table\n    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py\", line 908, in _do_insert\n    using=using, raw=raw)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/manager.py\", line 82, in manager_method\n    return getattr(self.get_queryset(), name)(*args, **kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/query.py\", line 1186, in _insert\n    return query.get_compiler(using=using).execute_sql(return_id)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/sql/compiler.py\", line 1375, in execute_sql\n    cursor.execute(sql, params)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 67, in execute\n    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 76, in _execute_with_wrappers\n    return executor(sql, params, many, context)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 84, in _execute\n    return self.cursor.execute(sql, params)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/utils.py\", line 89, in __exit__\n    raise dj_exc_value.with_traceback(traceback) from exc_value\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 84, in _execute\n    return self.cursor.execute(sql, params)\n"

Details about the remote used for syncing :

        {
            "architectures": "amd64",
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "components": "main contrib",
            "distributions": "buster bullseye buster-updates bullseye-updates",
            "download_concurrency": 20,
            "gpgkey": null,
            "name": "debian",
            "password": null,
            "policy": "on_demand",
            "proxy_url": "http://10.2.5.69:3128",
            "pulp_created": "2020-07-24T09:23:25.241744Z",
            "pulp_href": "/pulp/api/v3/remotes/deb/apt/a3036ec8-8c0c-408d-a493-007668ee4ad9/",
            "pulp_last_updated": "2020-07-24T09:25:29.131085Z",
            "sync_installer": false,
            "sync_sources": false,
            "sync_udebs": false,
            "tls_validation": true,
            "url": "http://ftp.ch.debian.org/debian/",
            "username": null
        }

#7 Updated by quba42 3 months ago

  • Priority changed from Normal to High

#8 Updated by quba42 3 months ago

@swisscom We had the same issue, and fixed it under a separate issue: https://pulp.plan.io/issues/7257

I expect both fixes will be merged into master today.

However, there is one more related/similar issue (https://pulp.plan.io/issues/7236). The remaining problem is that the pulp_deb plugin is currently declaring some fields as mandatory even though Debian says they are only "recommended". This causes failed syncs when there are packages that do not provide those fields.

Once all three of these issues are fixed, it is my hope that we will have fixed all (or at least the vast majority) of "There is some package control field we are not handling correctly" type issues...

#9 Updated by quba42 3 months ago

  • Checklist item deleted (Ensure exactly those control file fields are mandatory, that officially are so.)
  • Checklist item deleted (Ensure any control file fields present in an upstream repo will also be present in pulp_deb publications!)
  • Checklist item deleted (Add relevant assertions to our publication tests.)
  • Checklist item deleted (If needed expand our test fixtures with example packages that use a maximum set of control file fields.)

#10 Updated by quba42 3 months ago

  • Related to Issue #7266: Add tests to ensure all control file fields present in an upstream package index will be present in pulp_deb added

#11 Updated by quba42 3 months ago

  • Related to Issue #7257: Many pulp_deb control file fields are limited to 255 characters but should not be added

#12 Updated by quba42 3 months ago

  • Related to Issue #7236: Sync fails if packages do not have a 'section' field added

#13 Updated by msinghal 3 months ago

  • Status changed from POST to MODIFIED

#14 Updated by quba42 about 2 months ago

  • Sprint/Milestone changed from Katello to 2.6.0b1

Please register to edit this issue

Also available in: Atom PDF