Project

Profile

Help

Story #7502

closed

As a user I want to sync upstream repositories using "Flat Repository Format"

Added by quba42 over 3 years ago. Updated about 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Target Release - Debian:
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Quarter:

Description

It appears some APT repositories do not store their release files somewhere beneath a dists/ folder, using the repo root (or some arbitrary path) instead.

Example: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/

Apparently this repository can be included in an /etc/apt/sources.list file as follows (at least on Ubuntu):

deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /

There is some Ubuntu documentation hinting at specifying an "exact path" here: http://manpages.ubuntu.com/manpages/xenial/man5/sources.list.5.html (Also see the examples).


Related issues

Related to Debian Support - Issue #7673: Support synchronization of upstream repos with minimal Release file fields including for flat reposCLOSED - CURRENTRELEASEActions
Actions #1

Updated by quba42 over 3 years ago

Some initial thoughts:

  • We need to make sure the solution does not break any existing use cases.
  • If there is no unambiguous way to handle these new cases using the existing distribution field, then we need a new field/option in our remotes. (use this as a last resort)
  • We need to make sure we consider every instance of hard coded dists path segments in our solution.
  • The exception is the simple publisher which generally ignores upstream repo structure in favour of its own.
  • How should the structured publisher deal with so structured repos? (It probably needs to stay true to its word and retain upstream Release and Packages file placement.)
  • We will need to think about fixtures and tests for this.
  • I consider this to be a new feature, not a bugfix.
Actions #2

Updated by quba42 over 3 years ago

  • Tracker changed from Task to Story
  • Subject changed from Support upstream repositories that do not store Release files in a dists/ folder to As a user I want to sync upstream repositories that do not store Release files within /dists/
Actions #3

Updated by quba42 over 3 years ago

  • Tags Katello added
Actions #4

Updated by quba42 over 3 years ago

It appears Debian has a specification for this as well: https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format

They refer to this as a "Flat Repository Format".

Actions #5

Updated by quba42 over 3 years ago

  • Subject changed from As a user I want to sync upstream repositories that do not store Release files within /dists/ to As a user I want to sync upstream repositories using "Flat Repository Format"

Needs further discussion:

We could add a separate FlatAptRemote, or try to use the existing AptRemote. If we do go for two remotes, a common parent class would probably make sense.

Actions #6

Updated by quba42 over 3 years ago

  • Related to Issue #7673: Support synchronization of upstream repos with minimal Release file fields including for flat repos added
Actions #7

Updated by pulpbot over 3 years ago

  • Status changed from NEW to POST

Added by ThikaXer over 3 years ago

Revision 80ac6cc3 | View on GitHub

Support flat repository format

fixes #7502 https://pulp.plan.io/issues/7502

Actions #8

Updated by quba42 over 3 years ago

Note:

Thought this issue is nearing completion, the example repository in the ticket description won't become synchronizable with this change!

That repository has multiple issues that are independent of it being in flat repository format:

  • No Codename field in the Release file.
  • No Architectures field in the Release file. (They did add an Architecture field that does not contain a valid Debian architecture string...)
  • No Components field in the Release file.
  • The metadata references packages that don't actually exist. (This can be worked around by using policy="on_demand" in the remote).

Since APT is nevertheless willing to consume this repository, we might look into opening separate issues to support these corner cases.

Example repo:

deb http://developer.download.nvidia.com/compute/cuda/repos/debian10/x86_64/ /

Worst repository ever! (For good measure the public key needed to verify the repository is published right there in the repository...)

If we can get this repository to synchronize, we can synchronize anything!

Actions #9

Updated by ThikaXer over 3 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #10

Updated by quba42 over 3 years ago

  • Sprint/Milestone changed from Katello to 2.8.0
Actions #11

Updated by pulpbot over 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions #12

Updated by juliya11 about 3 years ago

  • File cash-for-cars-toowoomba.jpg added
Actions #13

Updated by quba42 about 3 years ago

  • File deleted (cash-for-cars-toowoomba.jpg)

Also available in: Atom PDF