force_full is not honored for distribution units
If you attempt to sync a repo like a KS repo with distribution units, the "force_full" option is not honored. This can cause issues if you have units that are not downloaded to disk, and you want to force a sync.
pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[7d067474-52cc-42b9-859d-8abc28f83995] pulp: pulp.server.controllers.repository:INFO: [7d067474] Fully resyncing due to use of force_full in config pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7d067474] Downloading metadata files. pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7d067474] Generating metadata databases. pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7d067474] Determining which units need to be downloaded. pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7d067474] Downloading 0 RPMs. pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [7d067474] Removing missing units. pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7d067474] Downloading additional units. pulp: pulp_rpm.plugins.importers.yum.parse.treeinfo:INFO: [7d067474] upstream distribution unchanged; skipping pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7d067474] Sync complete.
Note that "upstream distribution unchanged" is logged, even though force_full is in effect.
Updated by bherring over 3 years ago
Conversation with `beav` about the original problem¶
[08:16:18] <bherring Good day, beav! I am Brian Herring from Pulp QE. I had a question about re-creation on #4267 that you demo'd with your patch, if you had a second. [08:16:31] Notify: 18kersom is online (29!irc.devel.redhat.com) [09:45:17] <beav> sure, whats up [09:48:27] <bherring So, I am still digging, but i am surprised that specific failure that you demo'd wasn't caught by the following test -- https://github.com/PulpQE/Pulp-2-Tests/blob/7a7e13cae1148475b90d75bede836ffcfe300b9a/pulp_2_tests/tests/rpm/cli/test_sync.py#L91 [09:48:57] <bherring The best I can tell from the history is that test has been in CI for about 2 years, since the advent of #1982. [09:49:34] <beav> oh, that is for RPM [09:49:44] <beav> try the same test but for distributor units [09:49:55] <beav> that is the bug, it works ok for rpm but not for the extra units associated with kickstarts [09:50:29] <beav> so to repro the bug, try syncing RHEL 7 server kickstart and do the same test, but with like the vmlinuz file [09:50:47] <bherring ha. [09:50:56] <beav> you may need to find a centos kickstart or something to do it in upstream [09:51:42] <bherring well, that would explain the problem better :) [09:55:19] <bherring thanks! [09:59:44] <beav> np!
- Need to find a repo that is small to add to pulp-fixtures:
- only applicable for distribution units
- only applicable for extra units associated specifically with kickstarts
- update ci/test_sync.py to include more unit types other than RPM
- Need to generate a test fixture instead of the large import bin/scp there now (that is correct and works).
[10:04:45]<bherring stupid question: Are those files generally built by hand? PULP_DISTRIBUTION.xml? [10:05:33]<ttereshc> as far as I know, yes [10:05:45]<bherring ugh. [10:06:21]<ttereshc> maybe rel-eng has some automation for that but I doubt that [10:06:32]<ttereshc> I don't remember us providing any tool [10:06:48]<ttereshc> mostly because it's not upstream focused [10:06:54]<ttereshc> and used mostly internally [10:07:06]<bherring I think I could use `sed` or `vim` + RegEx to generation the <file> tags. It isn't that bad, once I have a \n delimited file [10:07:39]<ttereshc> sure, you can easily automate it [10:07:58]<bherring ack. what I have in my fixture works, but there are a lot of `un-needed` files. I was trying to ensure to only include what was used and, if I needed more, how to include it. [10:08:01]<bherring Thank you! [10:08:19]<ttereshc> I'm looking for the BZ I mentioned [10:08:42]<ttereshc> but in general, support for kickstart trees are bad in pulp2 [10:08:52]<ttereshc> very basic and badly designed [10:10:03]<bherring re: #4267 ? [10:12:30]<ttereshc> no, a different one [10:18:54]<ttereshc> finally https://bugzilla.redhat.com/show_bug.cgi?id=1648318#c12 [10:19:10]<ttereshc> what I was looking for are the sections which are used for images [10:19:24]<ttereshc> - uses also sections: images-*, stage2, checksums. [10:19:35]<ttereshc> guess what... images-* [10:19:39]<bherring not auth to see the bug [10:20:03]<ttereshc> any images outside of images-* won't be downloaded [10:20:21]<ttereshc> just 4 lines: [10:20:24]<ttereshc> Sat6: [10:20:24]<ttereshc> - detects kickstart tree by presence of .treeinfo or treeinfo file [10:20:24]<ttereshc> - uses the [general] section of the .treeinfo file: 'family', 'version', 'arch', 'packagedir', 'timestamp' and optionally 'variant' [10:20:24]<ttereshc> - uses also sections: images-*, stage2, checksums. [10:20:47]<ttereshc> these are the only places pulp is looking at [10:21:22]<bherring and PULP_DISTRIBUTION.xml, right? [10:21:48]<bherring Listed above is the behavior I was seeing and experimented with, which is good. [10:22:00]<ttereshc> ah yes [10:22:07]<ttereshc> and PULP_DISTRIBUTION.xml [10:22:09]<bherring :) [10:22:18]<ttereshc> I meant the parts of [.]treeinfo [10:22:26]<bherring of course! [10:22:45]<bherring I will add some PULP_DISTRIBUTION.xml to my fixture to ensure that is picked up as a distribution unit under test. [10:22:58]<bherring Thanks for the digging, ttereshc ! :D [10:23:35]<ttereshc> sure, np
Initially, I brought in a large, actual F29 mounted ISO dir and scaled down what I needed for test. That is what is currently in `rpm-fixtures` at the time of this writing.
What needs to be done is the fixture needs to be mocked (and re-tested) with a valid `.treeinfo` file AND PULP_DISTRIBUTION.xml (two birds at once) that replicates the actual fixture that is there now.
Reason for Replication¶
While I am always a fan of actual real data, the data in this instance is over 400MB and balloons the test time to over 5 mins.
The fixture, now understood what we are importing, can be simplified to the `.treeinfo` and `PULP_DISTRIBUTION.xml` files telling Pulp2 which `distribution units` to copy on sync.
In this case, what is in the units is not as important as the empty files:
- being copied in the initial sync to the `/var/lib/pulp/units/distribution` location to be operated on
- The sym-link being broken and fixed again when pointing at the distribution target .img that is randomly chosen (from more than one target location -- e.g. distribution/LiveOS, isolinux, pxelinux)
- The test working and passing, as intended
The file is hand-made from an `ls`+ command line magic and encapsulated within `<file />` delimiters.
Here is an example dump of the file:
<pulp_distribution version="1"> <file>isolinux/upgrade.img </file> <file>isolinux/TRANS.TBL </file> <file>isolinux/memtest </file> <file>isolinux/vmlinuz </file> <file>isolinux/vesamenu.c32 </file> <file>isolinux/boot.msg </file> <file>isolinux/initrd.img </file> <file>isolinux/isolinux.cfg </file> <file>isolinux/boot.cat </file> <file>isolinux/splash.png </file> <file>isolinux/isolinux.bin </file> <file>isolinux/grub.conf </file> </pulp_distribution>
The other location where distribution units are selected and copied. Anything not listed here or in the `PULP_DISTRIBUTION.xml` are not going to be copied as distribution units, thus being useless to the fixture for this test (but not representing reality for a far-end feed).
[addon-Server-HighAvailability] id = HighAvailability name = High Availability packages = addons/HighAvailability parent = Server repository = addons/HighAvailability type = addon uid = Server-HighAvailability [addon-Server-ResilientStorage] id = ResilientStorage name = Resilient Storage packages = addons/ResilientStorage parent = Server repository = addons/ResilientStorage type = addon uid = Server-ResilientStorage [checksums] LiveOS/squashfs.img = sha256:158657d4ff270db50204ae70c2256ca1523ee4f3c23eeedbabc3f723b69163a2 images/pxeboot/initrd.img = sha256:094631a67f1766688dfcf4ec1f07ab0921100878736afc3cc1b31f69e7b517a1 images/pxeboot/upgrade.img = sha256:bc884c1b4b520e57f81f3bf43156365d9ef651a47266b21e2c3d00b09e0da46b images/pxeboot/vmlinuz = sha256:91052b444e73f3eebdb93d1fb1506597e96c92d8de9c1e3c3f36b07a57d0a18f [general] ; WARNING.0 = This section provides compatibility with pre-productmd treeinfos. ; WARNING.1 = Read productmd documentation for details about new format. arch = x86_64 family = Red Hat Enterprise Linux name = Red Hat Enterprise Linux 7.6 packagedir = Packages platforms = x86_64,xen repository = . timestamp = 1539194952 variant = Server variants = Server version = 7.6 [header] type = productmd.treeinfo version = 1.2 [images-x86_64] initrd = images/pxeboot/initrd.img kernel = images/pxeboot/vmlinuz upgrade = images/pxeboot/upgrade.img [images-xen] initrd = images/pxeboot/initrd.img kernel = images/pxeboot/vmlinuz upgrade = images/pxeboot/upgrade.img [media] discnum = 1 totaldiscs = 1 [release] name = Red Hat Enterprise Linux short = RHEL version = 7.6 [stage2] mainimage = LiveOS/squashfs.img [tree] arch = x86_64 build_timestamp = 1539194952 platforms = x86_64,xen variants = Server [variant-Server] addons = Server-HighAvailability,Server-ResilientStorage id = Server name = Server packages = Packages repository = . type = variant uid = Server
Updated by bherring over 3 years ago
Just incase this is lost:
rpm-kickstart ============ Redmine Ticket: #4409 This test fixture was derived from the F29 ISO. Mounted, all content was copied and the following information was removed: - RPMs: Lots of unneeded size andthese caused issues with other tests in test_copy in the test fixture. Removed all RPMS to resolve unneeded possibility of cross-test contamination. - IMG: All .img files were replaced with `dd if=/dev/zero of=$file bs=1M count=1` replacements. Testing does not care about contents, only the files being copied. There are a total of (6) .img files - PULP_DISTRIBUTION.xml: Added to have the isolinx/* directory contents copied on sync of the repo. Else, this content was skipped (and the img files). - repodata/: Regenerated the repodata dir with all the RPMs removed. Note there is nothing in the package list. Used createrepo_c to re-generate locally and then copied to the test fixture. The test fixture can have all other extraneous files removed. However, these files have been left at this time as they are small and not causing issues. This fixture better represents a kickstart dir in this configuration. This should be the final configuration of this hand-crafted fixture. An issue in pulp-fixtures has been created  to track adding this to the auto-generated fixtures creation.  - https://github.com/PulpQE/pulp-fixtures/issues/117  - https://pulp.plan.io/issues/4409
Added by bherring over 3 years ago
Add distribution unit checks for --force-full
Pulp 2.19 fixes a bug where distribution units for kickstart files were not synched on a --force-full.
Refactored ForceSyncTestCase for distribution units and any future type of unit to be tested with force-full with resync.
Added a test case to support checks on distribution units and make the test_sync.py test more extensible for other units using a tuple definition of the location and extension of the unit.
The rpm-kickstart is added to the fixtures for test. The fixture was derived from the Fedora 29 ISO and scaled down for test.