Project

Profile

Help

Story #7502

As a user I want to sync upstream repositories using "Flat Repository Format"

Added by quba42 about 1 year ago. Updated 8 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Target Release - Debian:
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Quarter:

Description

It appears some APT repositories do not store their release files somewhere beneath a dists/ folder, using the repo root (or some arbitrary path) instead.

Example: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/

Apparently this repository can be included in an /etc/apt/sources.list file as follows (at least on Ubuntu):

deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /

There is some Ubuntu documentation hinting at specifying an "exact path" here: http://manpages.ubuntu.com/manpages/xenial/man5/sources.list.5.html (Also see the examples).


Related issues

Related to Debian Support - Issue #7673: Support synchronization of upstream repos with minimal Release file fields including for flat reposMODIFIED<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 80ac6cc3 View on GitHub
Added by ThikaXer 10 months ago

Support flat repository format

fixes #7502 https://pulp.plan.io/issues/7502

History

#1 Updated by quba42 about 1 year ago

Some initial thoughts:

  • We need to make sure the solution does not break any existing use cases.
  • If there is no unambiguous way to handle these new cases using the existing distribution field, then we need a new field/option in our remotes. (use this as a last resort)
  • We need to make sure we consider every instance of hard coded dists path segments in our solution.
  • The exception is the simple publisher which generally ignores upstream repo structure in favour of its own.
  • How should the structured publisher deal with so structured repos? (It probably needs to stay true to its word and retain upstream Release and Packages file placement.)
  • We will need to think about fixtures and tests for this.
  • I consider this to be a new feature, not a bugfix.

#2 Updated by quba42 about 1 year ago

  • Tracker changed from Task to Story
  • Subject changed from Support upstream repositories that do not store Release files in a dists/ folder to As a user I want to sync upstream repositories that do not store Release files within /dists/

#3 Updated by quba42 about 1 year ago

  • Tags Katello added

#4 Updated by quba42 about 1 year ago

It appears Debian has a specification for this as well: https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format

They refer to this as a "Flat Repository Format".

#5 Updated by quba42 about 1 year ago

  • Subject changed from As a user I want to sync upstream repositories that do not store Release files within /dists/ to As a user I want to sync upstream repositories using "Flat Repository Format"

Needs further discussion:

We could add a separate FlatAptRemote, or try to use the existing AptRemote. If we do go for two remotes, a common parent class would probably make sense.

#6 Updated by quba42 12 months ago

  • Related to Issue #7673: Support synchronization of upstream repos with minimal Release file fields including for flat repos added

#7 Updated by pulpbot 12 months ago

  • Status changed from NEW to POST

#8 Updated by quba42 10 months ago

Note:

Thought this issue is nearing completion, the example repository in the ticket description won't become synchronizable with this change!

That repository has multiple issues that are independent of it being in flat repository format:

  • No Codename field in the Release file.
  • No Architectures field in the Release file. (They did add an Architecture field that does not contain a valid Debian architecture string...)
  • No Components field in the Release file.
  • The metadata references packages that don't actually exist. (This can be worked around by using policy="on_demand" in the remote).

Since APT is nevertheless willing to consume this repository, we might look into opening separate issues to support these corner cases.

Example repo:

deb http://developer.download.nvidia.com/compute/cuda/repos/debian10/x86_64/ /

Worst repository ever! (For good measure the public key needed to verify the repository is published right there in the repository...)

If we can get this repository to synchronize, we can synchronize anything!

#9 Updated by ThikaXer 10 months ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#10 Updated by quba42 10 months ago

  • Sprint/Milestone changed from Katello to 2.8.0

#11 Updated by pulpbot 10 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

#12 Updated by juliya11 8 months ago

  • File cash-for-cars-toowoomba.jpg added

#13 Updated by quba42 8 months ago

  • File deleted (cash-for-cars-toowoomba.jpg)

Please register to edit this issue

Also available in: Atom PDF