Issue #4138
closed
Handling publish paths correctly
Description
Most Debian repositories use one of the following paths to the Release file:
<base_path>/dists/<codename>/Release
<base_path>/dists/<suite>/Release
The current Pulp 2 behaviour is to always publish using <codename>.
However, this is not actually how these paths are specified by Debian.
(See https://wiki.debian.org/DebianRepository/Format)
The specification uses:
<base_path>/dists/<distribution>/Release
Where <distribution> may contain an arbitrary number of folder levels and may be different to both <codename> and <suite>. (A prominent example is the debian-security repository which uses <distribution> = "<codename>/updates" or <distribution> = "<suite>/updates").
Not using the same publish paths as the upstream repositories we are syncing (this is the case whenever <distribution> != <codename> for the upstream repo) leads to a host of problems.
One example is: https://pulp.plan.io/issues/3464
Currently the <distribution> is only known to the original repo_importer.
Since publishers should be independent of the repo_importer, the only clean way I can think of to solve this and all related bugs is to add a distribution field to the units_deb_release collection and then use it accordingly.
Related issues
Updated by mdellweg about 6 years ago
- Related to Issue #3464: Unable to sync multiple releases in single deb repo added
Updated by quba42 about 6 years ago
Apologies for the messed up formatting, here is what I meant to write:
Most Debian repositories use one of the following paths to the Release file:
<base_path>/dists/<codename>/Release
<base_path>/dists/<suite>/Release
The current Pulp 2 behaviour is to always publish using <codename>.
However, this is not actually how these paths are specified by Debian.
(See https://wiki.debian.org/DebianRepository/Format)
The specification uses:
<base_path>/dists/<distribution>/Release
Where <distribution> may contain an arbitrary number of folder levels and may be different to both <codename> and <suite>.
(A prominent example is the debian-security repository which uses <distribution> = "<codename>/updates" or <distribution> = "<suite>/updates").
Not using the same publish paths as the upstream repositories we are syncing (this is the case whenever <distribution> != <codename> for the upstream repo) leads to a host of problems.
Examples include:
https://pulp.plan.io/issues/4008
https://pulp.plan.io/issues/3464
Currently the <distribution> is only known to the original repo_importer.
Since publishers should be independent of the repo_importer, the only clean way I can think of to solve this and all related bugs is to add a distribution field to the units_deb_release collection and then use it accordingly.
Updated by mdellweg about 6 years ago
- Related to Issue #4008: python-debpkgr does not structure Repositories correcly (in some case) added
Updated by bmbouter over 5 years ago
- Status changed from NEW to CLOSED - WONTFIX
Updated by bmbouter over 5 years ago
- Status changed from CLOSED - WONTFIX to NEW
Was closed as part of Pulp2 mass-fix, but reopening so the Debian plugin devs can determine what they want to do.
Updated by mdellweg over 5 years ago
- Has duplicate Issue #4705: Pulp creates Release files in the outer folder (stretch/updates) added
Updated by mdellweg over 5 years ago
- Has duplicate Story #4707: Folder structure of the deb repos does not match upstreams added
Updated by quba42 over 5 years ago
I now have a PR that fixes this issue:
https://github.com/pulp/pulp_deb/pull/83
Added by quba42 over 5 years ago
Added by quba42 over 5 years ago
Revision 97b75fd2 | View on GitHub
Add the distribution field to the DB models
Within a standard Debian repository structure, the term "distribution" refers to the unique string given by the path segment between the "dists/" folder, and some "Release" file (without the trailing slash).
Since each "Release" file in the directory structure is associated with exactly one unique distribution string, the terms "distribution" and "release" can be (and often are) used interchangably.
The distribution string is most commonly (but not always) given by either the "codename" or the "suite". The pulp_deb implementation prior to this commit, has assumed that the distribution string is always equal to codename, and has therefore imposed a uniqueness constraint on the codename for all releases/distributions within a single repository.
Since upstream repository sources make no such assumption and are not necessarily structured using the codename, this has lead to a plathora of unpredictable and buggy behaviour when synchronizing upstream repositories with 'codename != distribution'.
This change fixes these problems by introducing and using a "distribution" field for both the units_deb_release and units_deb_component collections.
revealed #4871 (depends on the fix for this issue) https://pulp.plan.io/issues/4871
ref #3464, #4055 https://pulp.plan.io/issues/3464 https://pulp.plan.io/issues/4055
fixes #4138, #4705, #4707 https://pulp.plan.io/issues/4138 https://pulp.plan.io/issues/4705 https://pulp.plan.io/issues/4707
Updated by quba42 over 5 years ago
- Status changed from NEW to MODIFIED
Applied in changeset 97b75fd286959549df706fffcc5a0ef423320e5e.
Updated by quba42 over 5 years ago
- Related to Issue #4884: Pulp 2 Nightly Regression: Pulp debian migrations fail on pulp DB initialization added
Updated by quba42 over 5 years ago
- Related to Issue #4924: KeyError during migration 0004 added
Updated by quba42 over 5 years ago
- Related to Issue #4951: Existence of /usr/lib/pulp/plugins/types/deb.json will prevent migration 0004 from working. added
Added by quba42 over 5 years ago
Revision f6f09378 | View on GitHub
Add support for Katello to migration 0004
With this addition, the best effort part of migration 0004 will work correctly for any Katello content view versions and lifecycle environments.
Note that the changes will only take effect once any effected repositories are next republished.
To facillitate such a republish, a list of altered repositories (by repo_id) is logged and stored in the following file:
/var/lib/pulp/0004_deb_repo_republish_candidates.txt
Also added additional debug logging.
Updated by ttereshc over 5 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
Updated by quba42 over 5 years ago
- Has duplicate Issue #4955: Deb repo with Releases=bionic/mongodb-org/4.1 can't be published added
.
Add an optional pre index part for DB migrations
This is needed to provide a way to prepare the data base for an index change BEFORE that index is applied. The new mechanism is subject to certain risks and limitations. It should only be used where absolutely necessary.
ref #4055, #4138 https://pulp.plan.io/issues/4055 https://pulp.plan.io/issues/4138
closes #4871 https://pulp.plan.io/issues/4871