Project

Profile

Help

Issue #7190

closed

Pulp 3 - pulp-deb : APT client installing same patches again and again (field Multi-Arch)

Added by swisscom over 3 years ago. Updated almost 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
4. Urgent
Version - Debian:
master
Platform Release:
Target Release - Debian:
OS:
CentOS 7
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Dear support,

We have the same issue than reported in https://pulp.plan.io/issues/6982 but for field Multi-Arch (was fixed for field Installed-Size in https://github.com/pulp/pulp_deb/pull/184)

Would it be possible to fix missing Multi-Arch field ? And in addition if possible not put SHA1 field in Packages file when SHA1 value is empty...

Pulp components version :

(pulp) -bash-4.2$ pip list | grep pulp
pulp-deb                        2.6.0b1.dev0 /var/lib/pulp/pulp_deb
pulp-rpm                        3.4.2
pulpcore                        3.5.0

Thanks in advance


Related issues

Related to Debian Support - Issue #7266: Add tests to ensure all control file fields present in an upstream package index will be present in pulp_debCLOSED - DUPLICATEActions
Related to Debian Support - Issue #7257: Many pulp_deb control file fields are limited to 255 characters but should not beCLOSED - CURRENTRELEASEActions
Related to Debian Support - Issue #7236: Sync fails if packages do not have a 'section' fieldCLOSED - CURRENTRELEASEActions
Actions #1

Updated by quba42 over 3 years ago

Thanks for opening these bug reports.

Rather than playing wack-a-mole with these issues, I think we need to systematically go through all well defined control file fields, and check whether they are handled correctly by the pulp_deb plugin.

Here, "handled correctly" includes making exactly those fields "mandatory" that officially are so, and of course ensuring that fields that are present in an upstream repository will also be present in our pulp_deb publications!

I would like to solve the whole class of issues, rather than just this individual case.

Actions #2

Updated by swisscom over 3 years ago

Thanks quba42 for your updatre

My preferred approach is to integrate all possible fields (except ones not present for the concerned package in upstream repository) in order to be able to publish an identical Packages content for the concerned packages. Most of the fields are described here : https://www.debian.org/doc/debian-policy/ch-controlfields.html However, Multi-Arch field is not present in this specification. More details abour this field here : https://wiki.ubuntu.com/MultiarchSpec

I understand that Debian documentation is inconsitent about content of Packages files. I can try to open a case to them if you want

Hint i you'd like to check by yourself if requrired fields are missing :

  1. Setup a repo, sync it with remote, create distribution and publication
  2. Connect your apt client to this newly created publication and patch your system using it
  3. You will see that same packages apply forever
  4. Play around with files in /var/lib/apt/lists/_Packages by adding content you find on the upstream repo Packages file
  5. You will see that when you run apt-get upgrade, the packages for which you modified attributes in /var/lib/apt/lists/.... based on upstream repository content will not be proposed anymore for upgrade as apt will be able to identify this package is already up-to-date

Let me know if you'd like to have more details :)

Thanks a lot for your support

Actions #3

Updated by swisscom over 3 years ago

I think it could be interesting to add to your test fixtures the following scenario :

  • After publishing a new distribution, patch your debian or ubuntu client
  • After patching, apt client must say there are no more applicable updates
Actions #4

Updated by pulpbot over 3 years ago

  • Status changed from NEW to POST
Actions #5

Updated by quba42 over 3 years ago

  • Sprint/Milestone set to Katello
Actions #6

Updated by swisscom over 3 years ago

Hi, I applied the fix on my Pulp server. Now the repo sync fails with the following error :

 "description": "value too long for type character varying(255)\n",
                "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 883, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 657, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 663, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/var/lib/pulp/pulp_deb/pulp_deb/app/tasks/synchronizing.py\", line 106, in synchronize\n    DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 148, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py\", line 102, in run\n    d_content.content.save()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/base.py\", line 110, in save\n    return super().save(*args, **kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py\", line 741, in save\n    force_update=force_update, update_fields=update_fields)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py\", line 779, in save_base\n    force_update, using, update_fields,\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py\", line 870, in _save_table\n    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/base.py\", line 908, in _do_insert\n    using=using, raw=raw)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/manager.py\", line 82, in manager_method\n    return getattr(self.get_queryset(), name)(*args, **kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/query.py\", line 1186, in _insert\n    return query.get_compiler(using=using).execute_sql(return_id)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/sql/compiler.py\", line 1375, in execute_sql\n    cursor.execute(sql, params)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 67, in execute\n    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 76, in _execute_with_wrappers\n    return executor(sql, params, many, context)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 84, in _execute\n    return self.cursor.execute(sql, params)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/utils.py\", line 89, in __exit__\n    raise dj_exc_value.with_traceback(traceback) from exc_value\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py\", line 84, in _execute\n    return self.cursor.execute(sql, params)\n"

Details about the remote used for syncing :

        {
            "architectures": "amd64",
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "components": "main contrib",
            "distributions": "buster bullseye buster-updates bullseye-updates",
            "download_concurrency": 20,
            "gpgkey": null,
            "name": "debian",
            "password": null,
            "policy": "on_demand",
            "proxy_url": "http://10.2.5.69:3128",
            "pulp_created": "2020-07-24T09:23:25.241744Z",
            "pulp_href": "/pulp/api/v3/remotes/deb/apt/a3036ec8-8c0c-408d-a493-007668ee4ad9/",
            "pulp_last_updated": "2020-07-24T09:25:29.131085Z",
            "sync_installer": false,
            "sync_sources": false,
            "sync_udebs": false,
            "tls_validation": true,
            "url": "http://ftp.ch.debian.org/debian/",
            "username": null
        }

Actions #7

Updated by quba42 over 3 years ago

  • Priority changed from Normal to High

Added by msinghal over 3 years ago

Revision bcc790c8 | View on GitHub

Add missing fields in Package file

fixes #7190 https://pulp.plan.io/issues/7190

Actions #8

Updated by quba42 over 3 years ago

@swisscom We had the same issue, and fixed it under a separate issue: https://pulp.plan.io/issues/7257

I expect both fixes will be merged into master today.

However, there is one more related/similar issue (https://pulp.plan.io/issues/7236). The remaining problem is that the pulp_deb plugin is currently declaring some fields as mandatory even though Debian says they are only "recommended". This causes failed syncs when there are packages that do not provide those fields.

Once all three of these issues are fixed, it is my hope that we will have fixed all (or at least the vast majority) of "There is some package control field we are not handling correctly" type issues...

Actions #9

Updated by quba42 over 3 years ago

Actions #10

Updated by quba42 over 3 years ago

  • Related to Issue #7266: Add tests to ensure all control file fields present in an upstream package index will be present in pulp_deb added
Actions #11

Updated by quba42 over 3 years ago

  • Related to Issue #7257: Many pulp_deb control file fields are limited to 255 characters but should not be added
Actions #12

Updated by quba42 over 3 years ago

  • Related to Issue #7236: Sync fails if packages do not have a 'section' field added
Actions #13

Updated by msinghal over 3 years ago

  • Status changed from POST to MODIFIED
Actions #14

Updated by quba42 over 3 years ago

  • Sprint/Milestone changed from Katello to 2.6.0b1
Actions #15

Updated by quba42 almost 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF