Project

Profile

Help

Issue #3464

Unable to sync multiple releases in single deb repo

Added by taylorcw over 2 years ago. Updated 28 days ago.

Status:
NEW
Priority:
Normal
Assignee:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version - Debian:
Platform Release:
Target Release - Debian:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:

Description

In order to keep the same relative-url to match upstream mirrors, trying to sync the releases: xenial, xenial-updates, xenial-security

CLI: pulp-admin deb repo create --repo-id 'xenial' --relative-url '/ubuntu/' --feed 'http://mirrors.us.kernel.org/ubuntu/' --releases 'xenial,xenial-updates,xenial-security' --components 'main' --architectures 'amd64'

This will allow a mirror like: http://mirrors.us.kernel.org/ubuntu/dists/ (see xenial*)

CLI: pulp-admin deb repo sync run --repo-id 'xenial'

An exception then occurs: "Exception: Checksum did not match"

pulp-server.log.txt (8.54 KB) pulp-server.log.txt taylorcw, 03/13/2018 06:52 PM

Related issues

Related to Debian Support - Issue #4138: Handling publish paths correctlyCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 97b75fd2 View on GitHub
Added by quba42 about 1 year ago

Add the distribution field to the DB models

Within a standard Debian repository structure, the term "distribution" refers to the unique string given by the path segment between the "dists/" folder, and some "Release" file (without the trailing slash).

Since each "Release" file in the directory structure is associated with exactly one unique distribution string, the terms "distribution" and "release" can be (and often are) used interchangably.

The distribution string is most commonly (but not always) given by either the "codename" or the "suite". The pulp_deb implementation prior to this commit, has assumed that the distribution string is always equal to codename, and has therefore imposed a uniqueness constraint on the codename for all releases/distributions within a single repository.

Since upstream repository sources make no such assumption and are not necessarily structured using the codename, this has lead to a plathora of unpredictable and buggy behaviour when synchronizing upstream repositories with 'codename != distribution'.

This change fixes these problems by introducing and using a "distribution" field for both the units_deb_release and units_deb_component collections.

revealed #4871 (depends on the fix for this issue) https://pulp.plan.io/issues/4871

ref #3464, #4055 https://pulp.plan.io/issues/3464 https://pulp.plan.io/issues/4055

fixes #4138, #4705, #4707 https://pulp.plan.io/issues/4138 https://pulp.plan.io/issues/4705 https://pulp.plan.io/issues/4707

History

#1 Updated by taylorcw over 2 years ago

I found part of the issue is with Ubuntu's release file has Suite is the sub-dist folder and Codename as the overall dist.

For example:
xenial-security = https://mirrors.edge.kernel.org/ubuntu/dists/xenial-security/Release
xenial-updates = https://mirrors.edge.kernel.org/ubuntu/dists/xenial-updates/Release
xenial = https://mirrors.edge.kernel.org/ubuntu/dists/xenial/Release

The following at least corrects the relative-url path. It doesn't fix the checksum issue as I think importers/sync.py is using the release.codename to compare vs release.suite. (Haven't had a change to confirm yet)

/usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/distributor.py

<
            repometa = aptrepo.AptRepoMeta(
                codename=codename,
                components=[comp.name for comp in rel_components],
                architectures=list(architectures),
            )

>
            repometa = aptrepo.AptRepoMeta(
                codename=release_unit.suite,
                components=[comp.name for comp in rel_components],
                architectures=list(architectures),
            )

Would it be possible to use the release tag as the repo vs relaying on the "Release" file inside the repo?

#2 Updated by mihai.ibanescu@gmail.com over 2 years ago

I am not sure what you mean by "release tag". Can you please elaborate?

#3 Updated by taylorcw over 2 years ago

Hi, sure thing. If a repo is created like this:

pulp-admin deb repo create --repo-id 'update-channels' --relative-url '/ubuntu/' --feed 'http://mirrors.us.kernel.org/ubuntu/' --components 'main,universe,multiverse,restricted' --architectures 'amd64' --releases "xenial-updates,trusty-updates"

Could the distributor publish with "/ubuntu/dists/xenial-update" & "/ubuntu/dists/trusty-update" which is what the user provides as the release name. Today, it's using "/ubuntu/dists/" + codename

#4 Updated by taylorcw almost 2 years ago

Any thoughts on getting this fixed? Thanks

#5 Updated by quba42 over 1 year ago

I have been working on the relevant code for unrelated reasons so I feel qualified to add my two cents.

The problem I see, is that there is no clean way of fixing this problem that does not risk creating new problems for other cases. Using the user supplied release name in the `MetadataStep` of `distributor.py` would currently make this step dependent on information stored in the `repo_importer` data base collection. This is not desirable since distributors and importers are meant to exist independently of one another.

A better solution might be to add an additional `distribution` field to the `units_deb_release` collection to store the needed information where it belongs. This would require a data base change and by extension a data base migration. Not a trivial fix.

For now the workaround is to sync these ubuntu releases in separate repositories.

#6 Updated by mdellweg over 1 year ago

  • Related to Issue #4138: Handling publish paths correctly added

#7 Updated by chriswilson over 1 year ago

I get the error message: "Exception: Checksum did not match"
spanish dictionary

#8 Updated by bmbouter about 1 year ago

  • Status changed from NEW to CLOSED - WONTFIX

#9 Updated by bmbouter about 1 year ago

  • Status changed from CLOSED - WONTFIX to NEW

Was closed as part of Pulp2 mass-fix, but reopening so the Debian plugin devs can determine what they want to do.

#10 Updated by bmbouter about 1 year ago

  • Tags Pulp 2 added

#11 Updated by quba42 about 1 year ago

I initially expected this PR (https://github.com/pulp/pulp_deb/pull/83) to solve this issue.

However, it appears that this is actually an issue within "python-debpkgr".
I suspect debpkgr is tripping up for a similar reason. (That is, it is using "codename" as an internal dict key, even though codename is not always unique within a single repository.)

My PR fixes this kind of issue within pulp_deb but not within debpkgr.

Until someone volunteers a fix for python-debpkgr, the only solution/work around is to keep the various "xenial" releases (i.e "xenial", "xenial-updates", "xenial-security", etc.) in separate repositories.

#12 Updated by mdellweg about 1 year ago

  • Triaged changed from No to Yes

#13 Updated by Milesmsksth 6 months ago

Thank you so much for this. I was into this issue and tired to tinker around to check if its possible but couldnt get it done. Now that i have seen the way you did it, thanks guys
with
regards
https://www.ucbrowser.vip/ https://shareit.onl/ https://mxplayer.pro/

#14 Updated by jayarambro 5 months ago

Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well.
I wanted to thank you for this websites! Thanks for sharing. Great websites!

https://shareits.xyz

#15 Updated by jayarambro 5 months ago

#16 Updated by Micheal786 4 months ago

Agree With Quba

The problem I see is that there is no clean way of fixing this problem that does not risk creating new problems for other cases. Using the user-supplied release name in the MetadataStep of distributor.py would currently make this step dependent on information stored in the repo_importer database collection. This is not desirable since distributors and importers are meant to exist independently of one another.

https://geometrydashapk.co/

#17 Updated by dean123 4 months ago

Correct

However, it appears that this is actually an issue within "python-debpkgr". I suspect debpkgr is tripping up for a similar reason. (That is, it is using "codename" as an internal dict key, even though codename is not always unique within a single repository.)

https://avgcleanerproapk.com/sd-maid-pro-apk/

#20 Updated by Anonymous 2 months ago

https://lotterysambadd.in/ nice web for lottery result.

#22 Updated by marry25 about 2 months ago

I was more than happy to uncover this great site. I need to to thank you for your time due to this fantastic read!! I definitely enjoyed every bit of it and I have you bookmarked to see new information on your blog. https://textfontsgenerator.blogspot.com/

#23 Updated by pinknemo13 about 2 months ago

Brilliant Blog! I might want to thank for the endeavors you have made recorded as a hard copy this post. I am trusting a similar best work from you later on too. I needed to thank you for this sites! A debt of gratitude is in order for sharing. Extraordinary sites! https://get-9apps.com https://get-9apps.com/download/ https://get-cartoonhd.com

#24 Updated by RoBerti about 2 months ago

The article has a very good idea, I found it useful information from this article. I will continue to monitor the article. https://skribblio.online

#25 Updated by sunnywilsonstore@gmail.com about 1 month ago

I read this article! I hope you will continue to have such articles to share with everyone! thank you!

#26 Updated by blackmartapk 28 days ago

Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites! [[https://blackmartalpha.co/]] https://blackmartalpha.co/

Please register to edit this issue

Also available in: Atom PDF