Project

Profile

Help

Issue #4138

Handling publish paths correctly

Added by quba42 about 1 year ago. Updated 6 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Severity:
3. High
Version - Debian:
Platform Release:
2.20.0
Blocks Release:
Target Release - Debian:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
Yes
Verification Required:
No
Sprint:

Description

Most Debian repositories use one of the following paths to the Release file:

<base_path>/dists//Release <base_path>/dists/<suite>/Release The current Pulp 2 behaviour is to always publish using <codename>. However, this is not actually how these paths are specified by Debian. (See https://wiki.debian.org/DebianRepository/Format) The specification uses: <base_path>/dists/<distribution>/Release Where <distribution> may contain an arbitrary number of folder levels and may be different to both <codename> and <suite>. (A prominent example is the debian-security repository which uses <distribution> = "<codename>/updates" or <distribution> = "<suite>/updates"). Not using the same publish paths as the upstream repositories we are syncing (this is the case whenever <distribution> != <codename> for the upstream repo) leads to a host of problems. One example is: https://pulp.plan.io/issues/3464 Currently the <distribution> is only known to the original repo_importer. Since publishers should be independent of the repo_importer, the only clean way I can think of to solve this and all related bugs is to add a distribution field to the units_deb_release collection and then use it accordingly.


Related issues

Related to Debian Support - Issue #3464: Unable to sync multiple releases in single deb repo NEW Actions
Related to Debian Support - Issue #4008: python-debpkgr does not structure Repositories correcly (in some case) CLOSED - CURRENTRELEASE Actions
Related to Debian Support - Issue #4884: Pulp 2 Nightly Regression: Pulp debian migrations fail on pulp DB initialization CLOSED - CURRENTRELEASE Actions
Related to Debian Support - Issue #4924: KeyError during migration 0004 CLOSED - CURRENTRELEASE Actions
Related to Debian Support - Issue #4951: Existence of /usr/lib/pulp/plugins/types/deb.json will prevent migration 0004 from working. CLOSED - CURRENTRELEASE Actions
Duplicated by Debian Support - Issue #4705: Pulp creates Release files in the outer folder (stretch/updates) CLOSED - CURRENTRELEASE Actions
Duplicated by Debian Support - Story #4707: Folder structure of the deb repos does not match upstreams CLOSED - CURRENTRELEASE Actions
Duplicated by Debian Support - Issue #4955: Deb repo with Releases=bionic/mongodb-org/4.1 can't be published CLOSED - DUPLICATE Actions

Associated revisions

Revision 2ebcb7bf View on GitHub
Added by quba42 8 months ago

Add an optional pre index part for DB migrations

This is needed to provide a way to prepare the data base for an index
change BEFORE that index is applied. The new mechanism is subject to
certain risks and limitations. It should only be used where absolutely
necessary.

ref #4055, #4138
https://pulp.plan.io/issues/4055
https://pulp.plan.io/issues/4138

closes #4871
https://pulp.plan.io/issues/4871

Revision 97b75fd2 View on GitHub
Added by quba42 8 months ago

Add the distribution field to the DB models

Within a standard Debian repository structure, the term "distribution"
refers to the unique string given by the path segment between the
"dists/" folder, and some "Release" file (without the trailing slash).

Since each "Release" file in the directory structure is associated with
exactly one unique distribution string, the terms "distribution" and
"release" can be (and often are) used interchangably.

The distribution string is most commonly (but not always) given by
either the "codename" or the "suite". The pulp_deb implementation prior
to this commit, has assumed that the distribution string is always equal
to codename, and has therefore imposed a uniqueness constraint on the
codename for all releases/distributions within a single repository.

Since upstream repository sources make no such assumption and are not
necessarily structured using the codename, this has lead to a plathora
of unpredictable and buggy behaviour when synchronizing upstream
repositories with 'codename != distribution'.

This change fixes these problems by introducing and using a
"distribution" field for both the units_deb_release and
units_deb_component collections.

revealed #4871 (depends on the fix for this issue)
https://pulp.plan.io/issues/4871

ref #3464, #4055
https://pulp.plan.io/issues/3464
https://pulp.plan.io/issues/4055

fixes #4138, #4705, #4707
https://pulp.plan.io/issues/4138
https://pulp.plan.io/issues/4705
https://pulp.plan.io/issues/4707

Revision f6f09378 View on GitHub
Added by quba42 7 months ago

Add support for Katello to migration 0004

With this addition, the best effort part of migration 0004 will work
correctly for any Katello content view versions and lifecycle
environments.

Note that the changes will only take effect once any effected
repositories are next republished.

To facillitate such a republish, a list of altered repositories (by
repo_id) is logged and stored in the following file:

/var/lib/pulp/0004_deb_repo_republish_candidates.txt

Also added additional debug logging.

History

#1 Updated by mdellweg about 1 year ago

  • Related to Issue #3464: Unable to sync multiple releases in single deb repo added

#2 Updated by quba42 about 1 year ago

Apologies for the messed up formatting, here is what I meant to write:

Most Debian repositories use one of the following paths to the Release file:

<base_path>/dists/<codename>/Release
<base_path>/dists/<suite>/Release

The current Pulp 2 behaviour is to always publish using <codename>.

However, this is not actually how these paths are specified by Debian.
(See https://wiki.debian.org/DebianRepository/Format)

The specification uses:

<base_path>/dists/<distribution>/Release

Where <distribution> may contain an arbitrary number of folder levels and may be different to both <codename> and <suite>.
(A prominent example is the debian-security repository which uses <distribution> = "<codename>/updates" or <distribution> = "<suite>/updates").

Not using the same publish paths as the upstream repositories we are syncing (this is the case whenever <distribution> != <codename> for the upstream repo) leads to a host of problems.

Examples include:
https://pulp.plan.io/issues/4008
https://pulp.plan.io/issues/3464

Currently the <distribution> is only known to the original repo_importer.
Since publishers should be independent of the repo_importer, the only clean way I can think of to solve this and all related bugs is to add a distribution field to the units_deb_release collection and then use it accordingly.

#3 Updated by mdellweg about 1 year ago

  • Related to Issue #4008: python-debpkgr does not structure Repositories correcly (in some case) added

#4 Updated by bmbouter 9 months ago

  • Status changed from NEW to CLOSED - WONTFIX

#5 Updated by bmbouter 9 months ago

  • Status changed from CLOSED - WONTFIX to NEW

Was closed as part of Pulp2 mass-fix, but reopening so the Debian plugin devs can determine what they want to do.

#6 Updated by bmbouter 9 months ago

  • Tags Pulp 2 added

#7 Updated by mdellweg 9 months ago

  • Assignee set to quba42

#8 Updated by mdellweg 9 months ago

  • Duplicated by Issue #4705: Pulp creates Release files in the outer folder (stretch/updates) added

#9 Updated by mdellweg 9 months ago

  • Duplicated by Story #4707: Folder structure of the deb repos does not match upstreams added

#10 Updated by quba42 9 months ago

I now have a PR that fixes this issue:
https://github.com/pulp/pulp_deb/pull/83

#11 Updated by quba42 8 months ago

  • Status changed from NEW to MODIFIED

#12 Updated by quba42 7 months ago

  • Related to Issue #4884: Pulp 2 Nightly Regression: Pulp debian migrations fail on pulp DB initialization added

#13 Updated by quba42 7 months ago

  • Related to Issue #4924: KeyError during migration 0004 added

#14 Updated by quba42 7 months ago

  • Related to Issue #4951: Existence of /usr/lib/pulp/plugins/types/deb.json will prevent migration 0004 from working. added

#15 Updated by mdellweg 7 months ago

  • Triaged changed from No to Yes

#16 Updated by ttereshc 7 months ago

  • Platform Release set to 2.20.0

#17 Updated by ttereshc 7 months ago

  • Sprint/Milestone set to 2.20.0

#18 Updated by ttereshc 7 months ago

  • Status changed from MODIFIED to ON_QA

#19 Updated by ttereshc 6 months ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#20 Updated by quba42 6 months ago

  • Duplicated by Issue #4955: Deb repo with Releases=bionic/mongodb-org/4.1 can't be published added

Please register to edit this issue

Also available in: Atom PDF