Test #4409


force_full is not honored for distribution units

Added by bherring about 5 years ago. Updated about 5 years ago.

Platform Release:
Pulp 2


If you attempt to sync a repo like a KS repo with distribution units, the "force_full" option is not honored. This can cause issues if you have units that are not downloaded to disk, and you want to force a sync.

For example:

pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[7d067474-52cc-42b9-859d-8abc28f83995]
pulp: pulp.server.controllers.repository:INFO: [7d067474] Fully resyncing due to use of force_full in config
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7d067474] Downloading metadata files.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7d067474] Generating metadata databases.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7d067474] Determining which units need to be downloaded. 
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7d067474] Downloading 0 RPMs.
pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [7d067474] Removing missing units.
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7d067474] Downloading additional units.
pulp: pulp_rpm.plugins.importers.yum.parse.treeinfo:INFO: [7d067474] upstream distribution unchanged; skipping
pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [7d067474] Sync complete.

Note that "upstream distribution unchanged" is logged, even though force_full is in effect.

Related issues

Copied from RPM Support - Issue #4267: force_full is not honored for distribution unitsCLOSED - CURRENTRELEASEcduryeeActions
Actions #1

Updated by bherring about 5 years ago

  • Copied from Issue #4267: force_full is not honored for distribution units added
Actions #2

Updated by bherring about 5 years ago

  • Status changed from MODIFIED to ASSIGNED
Actions #3

Updated by bherring about 5 years ago


Conversation with `beav` about the original problem

[08:16:18] <bherring Good day, beav! I am Brian Herring from Pulp QE. I had a question about re-creation on #4267 that you demo'd with your patch, if you had a second.
[08:16:31]  Notify: 18kersom is online (29!
[09:45:17] <beav> sure, whats up
[09:48:27] <bherring So, I am still digging, but i am surprised that specific failure that you demo'd wasn't caught by the following test --
[09:48:57] <bherring The best I can tell from the history is that test has been in CI for about 2 years, since the advent of #1982.
[09:49:34] <beav> oh, that is for RPM
[09:49:44] <beav> try the same test but for distributor units
[09:49:55] <beav> that is the bug, it works ok for rpm but not for the extra units associated with kickstarts
[09:50:29] <beav> so to repro the bug, try syncing RHEL 7 server kickstart and do the same test, but with like the vmlinuz file
[09:50:47] <bherring ha.
[09:50:56] <beav> you may need to find a centos kickstart or something to do it in upstream
[09:51:42] <bherring well, that would explain the problem better :)
[09:55:19] <bherring thanks!
[09:59:44] <beav> np!


  • Need to find a repo that is small to add to pulp-fixtures:
  • only applicable for distribution units
  • only applicable for extra units associated specifically with kickstarts
  • update ci/ to include more unit types other than RPM
  • Need to generate a test fixture instead of the large import bin/scp there now (that is correct and works).


Tanya Convo

[10:04:45]<bherring stupid question: Are those files generally built by hand? PULP_DISTRIBUTION.xml?
[10:05:33]<ttereshc> as far as I know, yes
[10:05:45]<bherring ugh.
[10:06:21]<ttereshc> maybe rel-eng has some automation for that but I doubt that
[10:06:32]<ttereshc> I don't remember us providing any tool
[10:06:48]<ttereshc> mostly because it's not upstream focused
[10:06:54]<ttereshc> and used mostly internally
[10:07:06]<bherring I think I could use `sed` or `vim` + RegEx to generation the <file> tags. It isn't that bad, once I have a \n delimited file
[10:07:39]<ttereshc> sure, you can easily automate it
[10:07:58]<bherring ack. what I have in my fixture works, but there are a lot of `un-needed` files. I was trying to ensure to only include what was used and, if I needed more, how to include it.
[10:08:01]<bherring Thank you!
[10:08:19]<ttereshc> I'm looking for the BZ I mentioned
[10:08:42]<ttereshc> but in general, support for kickstart trees are bad in pulp2
[10:08:52]<ttereshc> very basic and badly designed
[10:10:03]<bherring re: #4267 ?
[10:12:30]<ttereshc> no, a different one
[10:18:54]<ttereshc> finally
[10:19:10]<ttereshc> what I was looking for are the sections which are used for images
[10:19:24]<ttereshc> - uses also sections: images-*, stage2, checksums.
[10:19:35]<ttereshc> guess what... images-*
[10:19:39]<bherring not auth to see the bug
[10:20:03]<ttereshc> any images outside of images-* won't be downloaded
[10:20:21]<ttereshc> just 4 lines:
[10:20:24]<ttereshc> Sat6:
[10:20:24]<ttereshc> - detects kickstart tree by presence of .treeinfo or treeinfo file
[10:20:24]<ttereshc> - uses the [general] section of the .treeinfo file: 'family', 'version', 'arch', 'packagedir', 'timestamp' and optionally 'variant'
[10:20:24]<ttereshc> - uses also sections: images-*, stage2, checksums.
[10:20:47]<ttereshc> these are the only places pulp is looking at
[10:21:22]<bherring and PULP_DISTRIBUTION.xml, right?
[10:21:48]<bherring Listed above is the behavior I was seeing and experimented with, which is good.
[10:22:00]<ttereshc> ah yes
[10:22:07]<ttereshc> and  PULP_DISTRIBUTION.xml
[10:22:09]<bherring :)
[10:22:18]<ttereshc> I meant the parts of [.]treeinfo
[10:22:26]<bherring of course! 
[10:22:45]<bherring I will add some PULP_DISTRIBUTION.xml to my fixture to ensure that is picked up as a distribution unit under test.
[10:22:58]<bherring Thanks for the digging, ttereshc ! :D
[10:23:35]<ttereshc> sure, np

Test Fixture

Initially, I brought in a large, actual F29 mounted ISO dir and scaled down what I needed for test. That is what is currently in `rpm-fixtures` at the time of this writing.

What needs to be done is the fixture needs to be mocked (and re-tested) with a valid `.treeinfo` file AND PULP_DISTRIBUTION.xml (two birds at once) that replicates the actual fixture that is there now.

Reason for Replication

While I am always a fan of actual real data, the data in this instance is over 400MB and balloons the test time to over 5 mins.

The fixture, now understood what we are importing, can be simplified to the `.treeinfo` and `PULP_DISTRIBUTION.xml` files telling Pulp2 which `distribution units` to copy on sync.

In this case, what is in the units is not as important as the empty files:

  • being copied in the initial sync to the `/var/lib/pulp/units/distribution` location to be operated on
  • The sym-link being broken and fixed again when pointing at the distribution target .img that is randomly chosen (from more than one target location -- e.g. distribution/LiveOS, isolinux, pxelinux)
  • The test working and passing, as intended


The file is hand-made from an `ls`+ command line magic and encapsulated within `<file />` delimiters.

Here is an example dump of the file:

<pulp_distribution version="1">
<file>isolinux/upgrade.img </file>
<file>isolinux/TRANS.TBL </file>
<file>isolinux/memtest </file>
<file>isolinux/vmlinuz </file>
<file>isolinux/vesamenu.c32 </file>
<file>isolinux/boot.msg </file>
<file>isolinux/initrd.img </file>
<file>isolinux/isolinux.cfg </file>
<file>isolinux/ </file>
<file>isolinux/splash.png </file>
<file>isolinux/isolinux.bin </file>
<file>isolinux/grub.conf </file>


The other location where distribution units are selected and copied. Anything not listed here or in the `PULP_DISTRIBUTION.xml` are not going to be copied as distribution units, thus being useless to the fixture for this test (but not representing reality for a far-end feed).

id = HighAvailability
name = High Availability
packages = addons/HighAvailability
parent = Server
repository = addons/HighAvailability
type = addon
uid = Server-HighAvailability

id = ResilientStorage
name = Resilient Storage
packages = addons/ResilientStorage
parent = Server
repository = addons/ResilientStorage
type = addon
uid = Server-ResilientStorage

LiveOS/squashfs.img = sha256:158657d4ff270db50204ae70c2256ca1523ee4f3c23eeedbabc3f723b69163a2
images/pxeboot/initrd.img = sha256:094631a67f1766688dfcf4ec1f07ab0921100878736afc3cc1b31f69e7b517a1
images/pxeboot/upgrade.img = sha256:bc884c1b4b520e57f81f3bf43156365d9ef651a47266b21e2c3d00b09e0da46b
images/pxeboot/vmlinuz = sha256:91052b444e73f3eebdb93d1fb1506597e96c92d8de9c1e3c3f36b07a57d0a18f

; WARNING.0 = This section provides compatibility with pre-productmd treeinfos.
; WARNING.1 = Read productmd documentation for details about new format.
arch = x86_64
family = Red Hat Enterprise Linux
name = Red Hat Enterprise Linux 7.6
packagedir = Packages
platforms = x86_64,xen
repository = .
timestamp = 1539194952
variant = Server
variants = Server
version = 7.6

type = productmd.treeinfo
version = 1.2

initrd = images/pxeboot/initrd.img
kernel = images/pxeboot/vmlinuz
upgrade = images/pxeboot/upgrade.img

initrd = images/pxeboot/initrd.img
kernel = images/pxeboot/vmlinuz
upgrade = images/pxeboot/upgrade.img

discnum = 1
totaldiscs = 1

name = Red Hat Enterprise Linux
short = RHEL
version = 7.6

mainimage = LiveOS/squashfs.img

arch = x86_64
build_timestamp = 1539194952
platforms = x86_64,xen
variants = Server

addons = Server-HighAvailability,Server-ResilientStorage
id = Server
name = Server
packages = Packages
repository = .
type = variant
uid = Server
Actions #4

Updated by bherring about 5 years ago

Actions #5

Updated by bherring about 5 years ago

Actions #6

Updated by bherring about 5 years ago


Just incase this is lost:


Redmine Ticket: #4409

This test fixture was derived from the F29 ISO. Mounted, all content was copied
and the following information was removed:
- RPMs: Lots of unneeded size andthese caused issues with other tests in test_copy
  in the test fixture.
  Removed all RPMS to resolve unneeded possibility of cross-test contamination.
- IMG: All .img files were replaced with `dd if=/dev/zero of=$file bs=1M count=1`
  replacements. Testing does not care about contents, only the files being copied.
  There are a total of (6) .img files
- PULP_DISTRIBUTION.xml: Added to have the isolinx/* directory contents copied on
  sync of the repo. Else, this content was skipped (and the img files).
- repodata/: Regenerated the repodata dir with all the RPMs removed. Note there
  is nothing in the package list. Used createrepo_c to re-generate locally and
  then copied to the test fixture.

The test fixture can have all other extraneous files removed. However, these files
have been left at this time as they are small and not causing issues. This fixture 
better represents a kickstart dir in this configuration.

This should be the final configuration of this hand-crafted fixture. An issue in
pulp-fixtures has been created [0] to track adding this to the auto-generated
fixtures creation.

[0] -
[1] -

Added by bherring about 5 years ago

Revision c6bd2d39 | View on GitHub

Add distribution unit checks for --force-full

Pulp 2.19 fixes a bug where distribution units for kickstart files were not synched on a --force-full.

Refactored ForceSyncTestCase for distribution units and any future type of unit to be tested with force-full with resync.

Added a test case to support checks on distribution units and make the test more extensible for other units using a tuple definition of the location and extension of the unit.

The rpm-kickstart is added to the fixtures for test. The fixture was derived from the Fedora 29 ISO and scaled down for test.


Closes #4409

Actions #7

Updated by bherring about 5 years ago

  • Status changed from ASSIGNED to CLOSED - COMPLETE
Actions #8

Updated by bherring about 5 years ago

Actions #9

Updated by bherring about 5 years ago

Git Commit

In correctly wrote the `closes` ref:

Actions #10

Updated by bherring about 5 years ago

  • Sprint/Milestone set to 2.19.0
Actions #11

Updated by bherring about 5 years ago

  • Sprint/Milestone deleted (2.19.0)
Actions #12

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF