Project

Profile

Help

Issue #4138

Handling publish paths correctly

Added by quba42 almost 2 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
3. High
Version - Debian:
Platform Release:
2.20.0
Target Release - Debian:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Most Debian repositories use one of the following paths to the Release file:

<base_path>/dists/<codename>/Release
<base_path>/dists/<suite>/Release

The current Pulp 2 behaviour is to always publish using <codename>.

However, this is not actually how these paths are specified by Debian.
(See https://wiki.debian.org/DebianRepository/Format)

The specification uses:

<base_path>/dists/<distribution>/Release

Where <distribution> may contain an arbitrary number of folder levels and may be different to both <codename> and <suite>. (A prominent example is the debian-security repository which uses <distribution> = "<codename>/updates" or <distribution> = "<suite>/updates").

Not using the same publish paths as the upstream repositories we are syncing (this is the case whenever <distribution> != <codename> for the upstream repo) leads to a host of problems.

One example is: https://pulp.plan.io/issues/3464

Currently the <distribution> is only known to the original repo_importer.
Since publishers should be independent of the repo_importer, the only clean way I can think of to solve this and all related bugs is to add a distribution field to the units_deb_release collection and then use it accordingly.


Related issues

Related to Debian Support - Issue #3464: Unable to sync multiple releases in single deb repoCLOSED - WONTFIX<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to Debian Support - Issue #4008: python-debpkgr does not structure Repositories correcly (in some case)CLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to Debian Support - Issue #4884: Pulp 2 Nightly Regression: Pulp debian migrations fail on pulp DB initializationCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to Debian Support - Issue #4924: KeyError during migration 0004CLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to Debian Support - Issue #4951: Existence of /usr/lib/pulp/plugins/types/deb.json will prevent migration 0004 from working.CLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Has duplicate Debian Support - Issue #4705: Pulp creates Release files in the outer folder (stretch/updates)CLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Has duplicate Debian Support - Story #4707: Folder structure of the deb repos does not match upstreamsCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Has duplicate Debian Support - Issue #4955: Deb repo with Releases=bionic/mongodb-org/4.1 can't be publishedCLOSED - DUPLICATE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 2ebcb7bf View on GitHub
Added by quba42 over 1 year ago

Add an optional pre index part for DB migrations

This is needed to provide a way to prepare the data base for an index change BEFORE that index is applied. The new mechanism is subject to certain risks and limitations. It should only be used where absolutely necessary.

ref #4055, #4138 https://pulp.plan.io/issues/4055 https://pulp.plan.io/issues/4138

closes #4871 https://pulp.plan.io/issues/4871

Revision 97b75fd2 View on GitHub
Added by quba42 over 1 year ago

Add the distribution field to the DB models

Within a standard Debian repository structure, the term "distribution" refers to the unique string given by the path segment between the "dists/" folder, and some "Release" file (without the trailing slash).

Since each "Release" file in the directory structure is associated with exactly one unique distribution string, the terms "distribution" and "release" can be (and often are) used interchangably.

The distribution string is most commonly (but not always) given by either the "codename" or the "suite". The pulp_deb implementation prior to this commit, has assumed that the distribution string is always equal to codename, and has therefore imposed a uniqueness constraint on the codename for all releases/distributions within a single repository.

Since upstream repository sources make no such assumption and are not necessarily structured using the codename, this has lead to a plathora of unpredictable and buggy behaviour when synchronizing upstream repositories with 'codename != distribution'.

This change fixes these problems by introducing and using a "distribution" field for both the units_deb_release and units_deb_component collections.

revealed #4871 (depends on the fix for this issue) https://pulp.plan.io/issues/4871

ref #3464, #4055 https://pulp.plan.io/issues/3464 https://pulp.plan.io/issues/4055

fixes #4138, #4705, #4707 https://pulp.plan.io/issues/4138 https://pulp.plan.io/issues/4705 https://pulp.plan.io/issues/4707

Revision f6f09378 View on GitHub
Added by quba42 over 1 year ago

Add support for Katello to migration 0004

With this addition, the best effort part of migration 0004 will work correctly for any Katello content view versions and lifecycle environments.

Note that the changes will only take effect once any effected repositories are next republished.

To facillitate such a republish, a list of altered repositories (by repo_id) is logged and stored in the following file:

/var/lib/pulp/0004_deb_repo_republish_candidates.txt

Also added additional debug logging.

History

#1 Updated by mdellweg almost 2 years ago

  • Related to Issue #3464: Unable to sync multiple releases in single deb repo added

#2 Updated by quba42 almost 2 years ago

Apologies for the messed up formatting, here is what I meant to write:

Most Debian repositories use one of the following paths to the Release file:

<base_path>/dists/<codename>/Release
<base_path>/dists/<suite>/Release

The current Pulp 2 behaviour is to always publish using <codename>.

However, this is not actually how these paths are specified by Debian.
(See https://wiki.debian.org/DebianRepository/Format)

The specification uses:

<base_path>/dists/<distribution>/Release

Where <distribution> may contain an arbitrary number of folder levels and may be different to both <codename> and <suite>.
(A prominent example is the debian-security repository which uses <distribution> = "<codename>/updates" or <distribution> = "<suite>/updates").

Not using the same publish paths as the upstream repositories we are syncing (this is the case whenever <distribution> != <codename> for the upstream repo) leads to a host of problems.

Examples include:
https://pulp.plan.io/issues/4008
https://pulp.plan.io/issues/3464

Currently the <distribution> is only known to the original repo_importer.
Since publishers should be independent of the repo_importer, the only clean way I can think of to solve this and all related bugs is to add a distribution field to the units_deb_release collection and then use it accordingly.

#3 Updated by mdellweg almost 2 years ago

  • Related to Issue #4008: python-debpkgr does not structure Repositories correcly (in some case) added

#4 Updated by bmbouter over 1 year ago

  • Status changed from NEW to CLOSED - WONTFIX

#5 Updated by bmbouter over 1 year ago

  • Status changed from CLOSED - WONTFIX to NEW

Was closed as part of Pulp2 mass-fix, but reopening so the Debian plugin devs can determine what they want to do.

#6 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

#7 Updated by mdellweg over 1 year ago

  • Assignee set to quba42

#8 Updated by mdellweg over 1 year ago

  • Has duplicate Issue #4705: Pulp creates Release files in the outer folder (stretch/updates) added

#9 Updated by mdellweg over 1 year ago

  • Has duplicate Story #4707: Folder structure of the deb repos does not match upstreams added

#10 Updated by quba42 over 1 year ago

I now have a PR that fixes this issue:
https://github.com/pulp/pulp_deb/pull/83

#11 Updated by quba42 over 1 year ago

  • Status changed from NEW to MODIFIED

#12 Updated by quba42 over 1 year ago

  • Related to Issue #4884: Pulp 2 Nightly Regression: Pulp debian migrations fail on pulp DB initialization added

#13 Updated by quba42 over 1 year ago

  • Related to Issue #4924: KeyError during migration 0004 added

#14 Updated by quba42 over 1 year ago

  • Related to Issue #4951: Existence of /usr/lib/pulp/plugins/types/deb.json will prevent migration 0004 from working. added

#15 Updated by mdellweg over 1 year ago

  • Triaged changed from No to Yes

#16 Updated by ttereshc over 1 year ago

  • Platform Release set to 2.20.0

#17 Updated by ttereshc over 1 year ago

  • Sprint/Milestone set to 2.20.0

#18 Updated by ttereshc over 1 year ago

  • Status changed from MODIFIED to 5

#19 Updated by ttereshc over 1 year ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#20 Updated by quba42 over 1 year ago

  • Has duplicate Issue #4955: Deb repo with Releases=bionic/mongodb-org/4.1 can't be published added

Please register to edit this issue

Also available in: Atom PDF