Project

Profile

Help

Task #927

Test with openSUSE

Added by mhrivnak over 5 years ago. Updated over 1 year ago.

Status:
CLOSED - NOTABUG
Priority:
High
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Quarter:

Description

Test basic yum repo management for opensuse.

Deliverables

  • sync and publish opensuse repos
  • if possible, install and provision from a pulp-published repo
  • install software and updates from a pulp-published repo
  • file issues for any problems or gaps that are discovered
create_package_descr (21.2 KB) create_package_descr semyers, 10/26/2015 04:45 PM

Related issues

Related to RPM Support - Story #240: YUM repositories using deltainfo don't get delta RPMs publishedCLOSED - NOTABUG

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to RPM Support - Story #1391: As a user, I can distribute the *contents* of an isoCLOSED - WONTFIX

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Copied to RPM Support - Task #928: Test with SLESCLOSED - NOTABUG

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

History

#1 Updated by mhrivnak over 5 years ago

#2 Updated by mhrivnak over 5 years ago

  • Description updated (diff)

#3 Updated by mhrivnak over 5 years ago

  • Tags Groomed added

#4 Updated by bmbouter over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bmbouter
  • Sprint/Milestone set to 15

#5 Updated by mhrivnak over 5 years ago

  • Related to Story #240: YUM repositories using deltainfo don't get delta RPMs published added

#6 Updated by bmbouter over 5 years ago

  • Sprint/Milestone changed from 15 to 16
  • Groomed set to No
  • Sprint Candidate set to No

#7 Updated by bmbouter over 5 years ago

  • Groomed changed from No to Yes
  • Tags deleted (Groomed)

#8 Updated by bmbouter over 5 years ago

  • Sprint Candidate changed from No to Yes
  • Tags deleted (Sprint Candidate)

#9 Updated by mhrivnak over 5 years ago

  • Priority changed from Normal to High

#10 Updated by dkliban@redhat.com over 5 years ago

  • Assignee changed from bmbouter to dkliban@redhat.com

#11 Updated by dkliban@redhat.com over 5 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (dkliban@redhat.com)

#12 Updated by mhrivnak over 5 years ago

  • Sprint/Milestone deleted (16)

#13 Updated by dkliban@redhat.com over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com

#14 Updated by mhrivnak over 5 years ago

  • Priority changed from High to Normal

#15 Updated by dkliban@redhat.com over 5 years ago

  • Status changed from ASSIGNED to NEW

#16 Updated by mhrivnak about 5 years ago

  • Assignee deleted (dkliban@redhat.com)

#17 Updated by mhrivnak about 5 years ago

  • Priority changed from Normal to High

#18 Updated by semyers about 5 years ago

  • Status changed from NEW to ASSIGNED

I'm seeing this on the consumer side when pulling from the OpenSUSE 13.2 main repo: https://pulp.plan.io/issues/1322

I'm going to get off of master for the time being, and test this on 2.6.

#19 Updated by semyers about 5 years ago

  • Status changed from ASSIGNED to NEW

#20 Updated by semyers about 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to semyers

#21 Updated by semyers about 5 years ago

Syncing an openSUSE repo with the RPM plugin worked well enough, but attempting to do an openSUSE netinstall aimed at the published repo made the installer sad.

For a quick summary of what's wrong, here's what the "main" openSUSE yast repo looks like:
http://download.opensuse.org/distribution/13.2/repo/oss/

And here's what a normal yum repo looks like (fedora 22):
http://ftp.linux.ncsu.edu/pub/fedora/linux/releases/22/Everything/x86_64/os/

Inside the yast repo, you can peek into the "suse" directory and see the expected "repodata" folder, and this is enough for pulp to sync with the rpm plugin. However, watching pulp's httpd logs when trying to use the published repo from the openSUSE network install reveals that openSUSE is very interested in files in the repo's base dir, /oss, specifically /oss/content. To me, this means that for pulp to properly sync and publish a yast repo, we'll need to either pull more metadata than we currently are and/or present the metadata we have in a way that makes the published repo look like a proper yast repo.

Given the difference in the location of the "repodata" dir from the root of the repository, and the fact that its location is configurable in a YaST repo[0], my thinking at this point is that pulp_rpm probably isn't the tool to be used when syncing yast repos. That said, I don't know enough about pulp to have any good reason to think that, so I'll be asking the team about this tomorrow.

But hey, at least they publish their repo standards! :)

[0]: https://en.opensuse.org/openSUSE:Standards_YaST2_Repository_Metadata

#22 Updated by semyers about 5 years ago

What is the minimum amount of work required to distribute a yum repo as a YaST repo?

To answer that question, I've found this rather helpful document: https://en.opensuse.org/SDB:Creating_YaST_installation_sources

As far as I can tell, these are the major differences between a yum repo and a YaST repo:

  • yum repos are organized somewhat arbitrarily, as a directory containing rpm packages and a repodata dir that lets clients make sense of it all
  • YaST repos are more structured, with a top-level directory containing a few metadata files, including one that says which directory contains all the packages (organized by architecture) and the repodata dir

metadata files

These files are based at the root of the repository, and are both required.

The media descriptor file

media.1/media can be created at distribution time. The content of this file is as simple as

Pulp
YYYYMMDDhhmmss
1

Regarding the first field, "Pulp", the docs for this file state that this field "...should contain an identifier to quickly verify the media set the media belongs to. This identifier is not shown to the user but should contain human-readable data for debugging purposes".

The content descriptor file

content can also be created a distribution time. It has several fields available for use, documented here, but I'll outline what I believe are the fields we should (or must?) include, based on what I've seen in "real" openSUSE repos:
CONTENTSTYLE, DATADIR, DESCRDIR, DISTRO, LINGUAS, VENDOR

CONTENTSTYLE must be first in the file, and must be set to 11 (identifies this as a Code11 YaST repo, which we're impersonating)
DATADIR is the directory (specifically: a subdirectory in the same directory as the content file) where the packages are contained. The DATADIRs that I've observed contain packages sorted into dirs by arch, as well as #repodata@, which we already know how to make
DESCRDIR is a little bit tricker. It's created by an awesome perl script (redundant, I know) that you can find linked on the "Creating_YaST_installation_sources" page in the openSUSE repo, or you can look at the one that I've conveniently attached to this ticket. :) I expect we'd reimplement this in python, if this hasn't already been done (I haven't looked).
DISTRO appears to be an undocumented abbreviation of DISTRIBUTION, which (acording to the docs) is "Some string denoting the distribution. The same string is most probably used in the .rpms to denote the distribution. Usually a composition of name version and flavor." Based on that, I expect we can (probably?) snag it from the packages. Given the requirement that it's "some string", I'm not sure how important it is that this contains a useful value.
LINGUAS (not LANGUAGES, apparently) is a space-delimited list of languages supported by this repo. I'm not quite sure how to populate this, but it's related to what we generate at in the DESCRDIR step with the create_package_descr script, and is possibly derivable from package metadata.
VENDOR is, predictably, the name of the repository vendor. In our case, likely "Pulp".

LABEL is the nicely-formatted human-readable string that will be shown by packaging tools using this repo. It is not required, but seems like a nice thing to do. :)

In addition, there are META, HASH, and KEY tags, which all contain the expected hashes of non-package files that may be downloaded from this repo (e.g. the packages files found in the DESCRDIR)

Conflicting information

The wiki page for the content file include a section (at the time I write this, at least) with a little bit of Q&A, specifically someone asking what fields are required:
https://en.opensuse.org/openSUSE:Standards_YaST2_Repository_Metadata_content#Not_documented.2Funclear

It says that NAME, VERSION, BASEARCHS, and VENDOR are required. But, based on the content file for the main opensuse repo, that can't be true. Only VENDOR can be found in this file, despite it clearly stating CONTENTSTYLE 11:
http://download.opensuse.org/distribution/13.2/repo/oss/content

When conflicts like this appear, it seems preferable to defer to the actual repo that openSUSE users connect to, rather than relying on potentially outdated wiki docs. It would probably also be good to correct their wiki where possible.

Cryptography

gpg

The content file may be signed, by including content.asc alongside the content file. Additionally, the public part of the key used to generate that signature can be included in content.key.

https

Nope!

In the iso installer, I wasn't able to get openSUSE to even do https without booting into the advanced install env. Even when I did, my pulp install was using a self-signed cert so I was never able to establish a connection from the installer due to a lack of trust. The repositories published by openSUSE itself are not secured by HTTPS, instead appearing to completely rely on the GPG-signed content file.

#23 Updated by semyers almost 5 years ago

Story so far:

  • sync and publish opensuse repos

This works. The layout of the openSUSE repository is mangled when republished by pulp, but it doesn't seem to matter. openSUSE indexes the published repo and installs packages just fine.

  • if possible, install and provision from a pulp-published repo

This might be possible, if pulp were to publish a YaST repo as described in earlier comments. Satellite 5 currently does this by exposing the openSUSE install iso via HTTP with some manual work, and using SUSE's version of kickstarting (autoyast) to install from the exposed iso. You can, incidentally, add a pulp repo as a content source in the autoyast file at this point if desired, but it's not required for the install.

There are community plugins to help foreman out with making the autoyast kickstart XML file. What remains, it seems, is talking with Sat 6 folks about what I've learned here, and replicating what I've found so far in SLES.

  • install software and updates from a pulp-published repo

This completely works. After publishing a normal yum repo via pulp, you can add the repo to SUSE with zypper:
zypper addrepository -f http://hostname/pulp/path/to/repo reponame

zypper search -s <package_name> is an easy way to verify what repo a package can come from.

#24 Updated by semyers almost 5 years ago

Just quick update, I'm still working with the sat6 team to identify gaps between what sat5 provides and what sat6 provides based on my conclusions in the previous comment. Once they're identified and any outstanding RFEs are created, this should be done.

#25 Updated by semyers almost 5 years ago

  • Related to Story #1391: As a user, I can distribute the *contents* of an iso added

#26 Updated by semyers almost 5 years ago

  • Status changed from ASSIGNED to CLOSED - NOTABUG

openSUSE provisioning is something that is out of scope for pulp, and should be handled by foreman/satellite. The satellite team is still identifying the steps that would need to be taken to get feature parity in satellite 6 (via foreman) to what we have in satellite 5.

Pulp can help foreman out a little bit by exposing the contents of isos, which I've opened as story #1391.

closing with NOTABUG, since this wasn't a bug. :)

#27 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF