Project

Profile

Help

Story #1769

As a user, I would like to be able to sync distribution trees

Added by jcline@redhat.com over 4 years ago. Updated over 1 year ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Overview

Our RPM plugin claims that it syncs "distributions", but much of the upstream content is not mirrored. We are failing to handle what I am going to call "addon repositories" correctly for at least RHEL6 and RHEL7 (I have not confirmed others, but I expect this is across the board). The problem is that there are additional repositories within a kickstart-able repository and we fail to sync those repositories.

To witness this:

  1. Sync https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/kickstart/ (or a some mirror thereof)
  2. Inspect the published repository. Note that /content/dist/rhel/server/6/6Server/x86_64/kickstart/{LoadBalancer,HighAvailability,ResilientStorage,ScalableFileSystem}/ contain nothing except upstream version of the repomd.xml (which we shouldn't be syncing in any case).
  3. Inspect the database for one or more of the packages that are in the upstream repository's version of those directories. Note they are not present.
  4. To see the impact of this, try to kickstart from the Pulp repository and when it comes time to select additional repositories, try to select one of the missing ones (Load Balancer, for example). It will fail.

This story is about getting our story straight when it comes to distribution trees. The first and most important task for this story is making sure we completely understand distribution trees and how to work with them.

Distribution trees seem to (these days) be generated by a tool called pungi[0] (which uses tools like lorax[1]). Red Hat's release engineering team (RCM) has extensive documentation[2] on treeinfo files (which seem to describe a distribution tree) so it would be wise to talk to them to make sure we understand what a distribution tree is and how it will vary from release to release (both in the past and any upcoming changes). It's also worth noting both lorax and RCM's tooling is written in Python and creates/parses the treeinfo files.

Once we understand the role of the treeinfo file and what content falls into the distribution category, we should come up with a solution to handle past and present formats of distribution trees. One solution that has already been attempted is the PULP_DISTRIBUTION.xml, which was a way to have us sync content that's not referenced by a treeinfo file or the repodata. We need to determine whether having such a file is something we require (and it may be required to support older versions of distribution trees only) and if that is the case, we need to model the metadata file and those files it references. If we do choose to have such a file, we should spend some time designing it to make sure it fits our needs (contains validation data, has a well defined schema, has a well defined purpose, etc). We should also make sure it is not a burden on the user (currently RCM has to generate the PULP_DISTRIBUTION.xml manually).

The current version of the treeinfo file may well provide us with everything we need to sync a distribution tree. If that is the case, we should use it.

Files Missing from treeinfo/repodata

This is a list of files I've noticed in various repositories that do not appear to be referenced by either the treeinfo file or the repodata files.

  • License, EULA, and README text files
  • GPG signing keys
  • Release notes (either in a directory or in the root of the repository)
  • `isolinux` and `EFI` directories that appear to contain boot images (in addition to the `images` and `LiveOS` directories that are mentioned in the treeinfo file).
  • A few images in the images/ directory

[0] https://pagure.io/pungi
[1] https://github.com/rhinstaller/lorax/
[2] https://release-engineering.github.io/productmd/


Checklist


Related issues

Related to RPM Support - Issue #1661: Syncing a repository with a Packages directory results many duplicate symlinksCLOSED - WONTFIX<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to RPM Support - Issue #1768: Unable to sync RHEL 5 repositories with a distributionCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to Pulp - Story #1997: RFE: Need to get distribution info from cdn atomic treeCLOSED - WONTFIX

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Blocks Pulp - Story #764: As a user, I can use the Pulp API to upload distribution content unitsCLOSED - WONTFIX

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

History

#1 Updated by rbarlow over 4 years ago

The Lorax speaks for the trees.

--
Sent from your iPhone.

#2 Updated by jcline@redhat.com over 4 years ago

  • Related to Issue #1661: Syncing a repository with a Packages directory results many duplicate symlinks added

#3 Updated by jcline@redhat.com over 4 years ago

  • Blocks Story #764: As a user, I can use the Pulp API to upload distribution content units added

#4 Updated by jcline@redhat.com over 4 years ago

  • Description updated (diff)

#5 Updated by jcline@redhat.com over 4 years ago

  • Related to Issue #1768: Unable to sync RHEL 5 repositories with a distribution added

#6 Updated by jcline@redhat.com over 4 years ago

  • Description updated (diff)

#7 Updated by mhrivnak over 4 years ago

  • Sprint/Milestone set to 20
  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

#8 Updated by jcline@redhat.com over 4 years ago

  • Description updated (diff)

#9 Updated by jcline@redhat.com over 4 years ago

  • Description updated (diff)

Documentation PR with my findings thus far: https://github.com/pulp/pulp_rpm/pull/867

#11 Updated by jcline@redhat.com over 4 years ago

I've filed an issue with Pungi as well: https://pagure.io/pungi/issue/295

#12 Updated by mhrivnak over 4 years ago

  • Sprint/Milestone changed from 20 to 21

#13 Updated by jcline@redhat.com over 4 years ago

  • Related to Story #1997: RFE: Need to get distribution info from cdn atomic tree added

#14 Updated by mhrivnak over 4 years ago

  • Sprint/Milestone changed from 21 to 22

#15 Updated by mhrivnak about 4 years ago

  • Sprint/Milestone deleted (22)

#16 Updated by rchan over 2 years ago

  • Sprint Candidate deleted (Yes)

#17 Updated by bmbouter over 1 year ago

  • Status changed from NEW to CLOSED - WONTFIX
  • Sprint Candidate set to No

#18 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF