Project

Profile

Help

Issue #6920

Pulp 3 - pulp-deb : Issue synchronizing bullseye-security repo

Added by swisscom 4 months ago. Updated about 1 month ago.

Status:
NEW
Priority:
Normal
Assignee:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version - Debian:
master
Platform Release:
Target Release - Debian:
OS:
CentOS 7
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Quarter:

Description

Dear support team,

I just discovered that the sync task fails when bullseye-security distribution is included in a remote, like the following :

        {
            "architectures": "amd64", 
            "ca_cert": null, 
            "client_cert": null, 
            "client_key": null, 
            "components": null, 
            "distributions": "bullseye-security", 
            "download_concurrency": 20, 
            "gpgkey": null, 
            "name": "debian-security", 
            "policy": "on_demand", 
            "proxy_url": null, 
            "pulp_created": "2020-06-08T07:33:26.302753Z", 
            "pulp_href": "/pulp/api/v3/remotes/deb/apt/d394132f-413f-416b-b5aa-64e5e6883179/", 
            "pulp_last_updated": "2020-06-08T07:56:41.103430Z", 
            "sync_installer": false, 
            "sync_sources": false, 
            "sync_udebs": false, 
            "tls_validation": true, 
            "url": "http://security.debian.org/debian-security/"
        }, 

The error is the following :

File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 886, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 664, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 670, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py\", line 106, in synchronize\n    DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 149, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 228, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py\", line 120, in run\n    ContentArtifact.objects.bulk_get_or_create(content_artifact_bulk)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/content.py\", line 55, in bulk_get_or_create\n    objs[i] = objs[i].__class__.objects.get(objs[i].q())\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/manager.py\", line 82, in manager_method\n    return getattr(self.get_queryset(), name)(*args, **kwargs)\n  File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/query.py\", line 408, in get\n    self.model._meta.object_name\n

If I patch the remote and include some components like "main contrib", the sync task do not fail anymore but the resulting publication's (dists/bullseye-security) contain only the Release file, not the "main" and "contrib" directories.

So I guess Pulp has an issue to discover the components of bullseye-security remote

Any idea if the issue resides on Pulp side or on the remote itself ?


Related issues

Related to Debian Support - Test #6051: Add tests for known cases where the distribution is not the codename or suiteMODIFIED<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

History

#1 Updated by quba42 4 months ago

  • Related to Test #6051: Add tests for known cases where the distribution is not the codename or suite added

#2 Updated by quba42 4 months ago

The remote looks fine to me.

It is also important to note, that Debian appears to have given up on their policy of publishing their security repos using an "unusual" structure starting with "bullseye".

This does NOT appear to be a codename != distribution issue!

I will need to investigate.

May I ask what version of pulp_deb you are using?

#3 Updated by swisscom 4 months ago

Hi, I am using 2.3.0b1 and manually applied the fixes from following issues : 6907 6873 6876 6787

I have no clue if all these fixes have been published in a version installed with pip or not yet. If you'd like me to upgrade pulp_deb or reinstal pulp server, let me know

#4 Updated by quba42 3 months ago

  • Sprint/Milestone set to Katello

#5 Updated by quba42 2 months ago

As far as I can tell the main trouble with the bullseye-security repo is that it does not yet contain any packages.

(There is a possible second issue, in that this repository includes a weird circular symlink for backwards compatibility with the legacy dist folder structure).

The plugin is currently completely unable to handle valid repositories that do not contain any packages.

Syncing such repositories is a valid use case that needs to be fixed.

I have opend a draft PR for adding relevant test fixtures here: https://github.com/pulp/pulp-fixtures/pull/187

#6 Updated by quba42 about 1 month ago

Update: Syncing empty (no package) repositories works and has test coverage, but bullseye-security is still broken.

My only remaining theory is that the circular symlink is the root cause. However, it is still far from clear to me, why it should matter.

#7 Updated by quba42 about 1 month ago

  • Tags Katello added

Please register to edit this issue

Also available in: Atom PDF