RPM Support

Notes¶

Conversation with `beav` about the original problem¶

[08:16:18] <bherring Good day, beav! I am Brian Herring from Pulp QE. I had a question about re-creation on #4267 that you demo'd with your patch, if you had a second.
[08:16:31]  Notify: 18kersom is online (29!irc.devel.redhat.com)
[09:45:17] <beav> sure, whats up
[09:48:27] <bherring So, I am still digging, but i am surprised that specific failure that you demo'd wasn't caught by the following test -- https://github.com/PulpQE/Pulp-2-Tests/blob/7a7e13cae1148475b90d75bede836ffcfe300b9a/pulp_2_tests/tests/rpm/cli/test_sync.py#L91
[09:48:57] <bherring The best I can tell from the history is that test has been in CI for about 2 years, since the advent of #1982.
[09:49:34] <beav> oh, that is for RPM
[09:49:44] <beav> try the same test but for distributor units
[09:49:55] <beav> that is the bug, it works ok for rpm but not for the extra units associated with kickstarts
[09:50:29] <beav> so to repro the bug, try syncing RHEL 7 server kickstart and do the same test, but with like the vmlinuz file
[09:50:47] <bherring ha.
[09:50:56] <beav> you may need to find a centos kickstart or something to do it in upstream
[09:51:42] <bherring well, that would explain the problem better :)
[09:55:19] <bherring thanks!
[09:59:44] <beav> np!

Thoughts¶

• Need to find a repo that is small to add to pulp-fixtures:
  • https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-kickstart/repodata/
• only applicable for distribution units
• only applicable for extra units associated specifically with kickstarts
• update ci/test_sync.py to include more unit types other than RPM
• Need to generate a test fixture instead of the large import bin/scp there now (that is correct and works).

References¶

• http://cdn.stage.redhat.com/content/dist/rhel/server/6/6.6/x86_64/kickstart/
• https://docs.pulpproject.org/plugins/pulp_rpm/user-guide/recipes.html
• https://github.com/PulpQE/pulp-fixtures
• https://docs.pulpproject.org/en/2.18/testing/dev-guide/newtypesupport/plugin/importers.html
• https://github.com/PulpQE/Pulp-2-Tests/pull/164
• https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-kickstart/
• http://cdn.stage.redhat.com/content/dist/rhel/server/6/6.6/x86_64/kickstart/
  • http://cdn.stage.redhat.com/content/dist/rhel/server/6/6.6/x86_64/kickstart/PULP_DISTRIBUTION.xml
• https://docs.python.org/2/library/string.html
  • https://www.python.org/dev/peps/pep-3101
  • https://www.tutorialspoint.com/python/python_tuples.htm
  • https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting
  • https://docs.python.org/3/library/subprocess.html
• https://bugzilla.redhat.com/show_bug.cgi?id=1648318
• https://docs.pulpproject.org/plugins/pulp_rpm/tech-reference/yum-plugins.html?highlight=pulp_distribution%20xml
• https://github.com/PulpQE/pulp-fixtures/issues/117

Tanya Convo¶

[10:04:45]<bherring stupid question: Are those files generally built by hand? PULP_DISTRIBUTION.xml?
[10:05:33]<ttereshc> as far as I know, yes
[10:05:45]<bherring ugh.
[10:06:21]<ttereshc> maybe rel-eng has some automation for that but I doubt that
[10:06:32]<ttereshc> I don't remember us providing any tool
[10:06:48]<ttereshc> mostly because it's not upstream focused
[10:06:54]<ttereshc> and used mostly internally
[10:07:06]<bherring I think I could use `sed` or `vim` + RegEx to generation the <file> tags. It isn't that bad, once I have a \n delimited file
[10:07:39]<ttereshc> sure, you can easily automate it
[10:07:58]<bherring ack. what I have in my fixture works, but there are a lot of `un-needed` files. I was trying to ensure to only include what was used and, if I needed more, how to include it.
[10:08:01]<bherring Thank you!
[10:08:19]<ttereshc> I'm looking for the BZ I mentioned
[10:08:42]<ttereshc> but in general, support for kickstart trees are bad in pulp2
[10:08:52]<ttereshc> very basic and badly designed
[10:10:03]<bherring re: #4267 ?
[10:12:30]<ttereshc> no, a different one
[10:18:54]<ttereshc> finally https://bugzilla.redhat.com/show_bug.cgi?id=1648318#c12
[10:19:10]<ttereshc> what I was looking for are the sections which are used for images
[10:19:24]<ttereshc> - uses also sections: images-*, stage2, checksums.
[10:19:35]<ttereshc> guess what... images-*
[10:19:39]<bherring not auth to see the bug
[10:20:03]<ttereshc> any images outside of images-* won't be downloaded
[10:20:21]<ttereshc> just 4 lines:
[10:20:24]<ttereshc> Sat6:
[10:20:24]<ttereshc> - detects kickstart tree by presence of .treeinfo or treeinfo file
[10:20:24]<ttereshc> - uses the [general] section of the .treeinfo file: 'family', 'version', 'arch', 'packagedir', 'timestamp' and optionally 'variant'
[10:20:24]<ttereshc> - uses also sections: images-*, stage2, checksums.
[10:20:47]<ttereshc> these are the only places pulp is looking at
[10:21:22]<bherring and PULP_DISTRIBUTION.xml, right?
[10:21:48]<bherring Listed above is the behavior I was seeing and experimented with, which is good.
[10:22:00]<ttereshc> ah yes
[10:22:07]<ttereshc> and  PULP_DISTRIBUTION.xml
[10:22:09]<bherring :)
[10:22:18]<ttereshc> I meant the parts of [.]treeinfo
[10:22:26]<bherring of course! 
[10:22:45]<bherring I will add some PULP_DISTRIBUTION.xml to my fixture to ensure that is picked up as a distribution unit under test.
[10:22:58]<bherring Thanks for the digging, ttereshc ! :D
[10:23:35]<ttereshc> sure, np

Test Fixture¶

• https://github.com/PulpQE/pulp-fixtures/issues/117

Initially, I brought in a large, actual F29 mounted ISO dir and scaled down what I needed for test. That is what is currently in `rpm-fixtures` at the time of this writing.

What needs to be done is the fixture needs to be mocked (and re-tested) with a valid `.treeinfo` file AND PULP_DISTRIBUTION.xml (two birds at once) that replicates the actual fixture that is there now.

Reason for Replication¶

While I am always a fan of actual real data, the data in this instance is over 400MB and balloons the test time to over 5 mins.

The fixture, now understood what we are importing, can be simplified to the `.treeinfo` and `PULP_DISTRIBUTION.xml` files telling Pulp2 which `distribution units` to copy on sync.

In this case, what is in the units is not as important as the empty files:

• being copied in the initial sync to the `/var/lib/pulp/units/distribution` location to be operated on
• The sym-link being broken and fixed again when pointing at the distribution target .img that is randomly chosen (from more than one target location -- e.g. distribution/LiveOS, isolinux, pxelinux)
• The test working and passing, as intended

Raw PULP_DISTRIBUTION¶

The file is hand-made from an `ls`+ command line magic and encapsulated within `<file />` delimiters.

Here is an example dump of the file:

<pulp_distribution version="1">
<file>isolinux/upgrade.img </file>
<file>isolinux/TRANS.TBL </file>
<file>isolinux/memtest </file>
<file>isolinux/vmlinuz </file>
<file>isolinux/vesamenu.c32 </file>
<file>isolinux/boot.msg </file>
<file>isolinux/initrd.img </file>
<file>isolinux/isolinux.cfg </file>
<file>isolinux/boot.cat </file>
<file>isolinux/splash.png </file>
<file>isolinux/isolinux.bin </file>
<file>isolinux/grub.conf </file>
</pulp_distribution>

.treeinfo¶

The other location where distribution units are selected and copied. Anything not listed here or in the `PULP_DISTRIBUTION.xml` are not going to be copied as distribution units, thus being useless to the fixture for this test (but not representing reality for a far-end feed).

[addon-Server-HighAvailability]
id = HighAvailability
name = High Availability
packages = addons/HighAvailability
parent = Server
repository = addons/HighAvailability
type = addon
uid = Server-HighAvailability

[addon-Server-ResilientStorage]
id = ResilientStorage
name = Resilient Storage
packages = addons/ResilientStorage
parent = Server
repository = addons/ResilientStorage
type = addon
uid = Server-ResilientStorage

[checksums]
LiveOS/squashfs.img = sha256:158657d4ff270db50204ae70c2256ca1523ee4f3c23eeedbabc3f723b69163a2
images/pxeboot/initrd.img = sha256:094631a67f1766688dfcf4ec1f07ab0921100878736afc3cc1b31f69e7b517a1
images/pxeboot/upgrade.img = sha256:bc884c1b4b520e57f81f3bf43156365d9ef651a47266b21e2c3d00b09e0da46b
images/pxeboot/vmlinuz = sha256:91052b444e73f3eebdb93d1fb1506597e96c92d8de9c1e3c3f36b07a57d0a18f

[general]
; WARNING.0 = This section provides compatibility with pre-productmd treeinfos.
; WARNING.1 = Read productmd documentation for details about new format.
arch = x86_64
family = Red Hat Enterprise Linux
name = Red Hat Enterprise Linux 7.6
packagedir = Packages
platforms = x86_64,xen
repository = .
timestamp = 1539194952
variant = Server
variants = Server
version = 7.6

[header]
type = productmd.treeinfo
version = 1.2

[images-x86_64]
initrd = images/pxeboot/initrd.img
kernel = images/pxeboot/vmlinuz
upgrade = images/pxeboot/upgrade.img

[images-xen]
initrd = images/pxeboot/initrd.img
kernel = images/pxeboot/vmlinuz
upgrade = images/pxeboot/upgrade.img

[media]
discnum = 1
totaldiscs = 1

[release]
name = Red Hat Enterprise Linux
short = RHEL
version = 7.6

[stage2]
mainimage = LiveOS/squashfs.img

[tree]
arch = x86_64
build_timestamp = 1539194952
platforms = x86_64,xen
variants = Server

[variant-Server]
addons = Server-HighAvailability,Server-ResilientStorage
id = Server
name = Server
packages = Packages
repository = .
type = variant
uid = Server

Added README¶

Just incase this is lost:

rpm-kickstart
============

Redmine Ticket: #4409

This test fixture was derived from the F29 ISO. Mounted, all content was copied
and the following information was removed:
- RPMs: Lots of unneeded size andthese caused issues with other tests in test_copy
  in the test fixture.
  Removed all RPMS to resolve unneeded possibility of cross-test contamination.
- IMG: All .img files were replaced with `dd if=/dev/zero of=$file bs=1M count=1`
  replacements. Testing does not care about contents, only the files being copied.
  There are a total of (6) .img files
- PULP_DISTRIBUTION.xml: Added to have the isolinx/* directory contents copied on
  sync of the repo. Else, this content was skipped (and the img files).
- repodata/: Regenerated the repodata dir with all the RPMs removed. Note there
  is nothing in the package list. Used createrepo_c to re-generate locally and
  then copied to the test fixture.

The test fixture can have all other extraneous files removed. However, these files
have been left at this time as they are small and not causing issues. This fixture 
better represents a kickstart dir in this configuration.

This should be the final configuration of this hand-crafted fixture. An issue in
pulp-fixtures has been created [0] to track adding this to the auto-generated
fixtures creation.

[0] - https://github.com/PulpQE/pulp-fixtures/issues/117
[1] - https://pulp.plan.io/issues/4409

Git Commit¶

In correctly wrote the `closes` ref: https://github.com/PulpQE/Pulp-2-Tests/pull/164