Pulp

PR builer fails testcase on fedora and yet passes on rhel. [0]

[0] https://github.com/pulp/pulp_docker/pull/229

It looks like env. script is different. Some observations from email thread discussion.

The restorecon fix looks good to me; for the remaining problem, it seems
to be a bug in the test environment/script.

I noticed that the failure is *exactly* what would happen if the fix was
not actually present.  From adding extra debug logs I found that the
tests on Fedora are using the code from pulp-docker-plugins installed
via RPM, rather than the version from git.  So, the test is running
against the old, buggy version of the code, and it correctly detects the
bug it was meant to detect.

See here in RHEL7: the test loads the code from the Jenkins workspace,
and as expected, order of last two steps is RedirectFileStep,
AtomicDirectoryPublishStep:

https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/unittest-pulp_docker-pr/469/node-type=rhel7-np/consoleText

-------------------- >> begin captured logging << --------------------
test_publish_steps: DEBUG: publish_steps comes from /home/jenkins/workspace/unittest-pulp_docker-pr/node-type/rhel7-np/plugin/plugins/pulp_docker/plugins/distributors/publish_steps.py
pulp.plugins.util.publish_step: DEBUG: Copying tree from /tmp/tmpWYEMZP/work to /tmp/tmpWYEMZP/publish/v2/master/foo/1520898214.84
pulp.plugins.util.misc: DEBUG: Clearing out directory: /tmp/tmpWYEMZP/publish/v2/master/foo
test_publish_steps: DEBUG: publish_steps comes from /home/jenkins/workspace/unittest-pulp_docker-pr/node-type/rhel7-np/plugin/plugins/pulp_docker/plugins/distributors/publish_steps.py
test_publish_steps: DEBUG: Child of step: <class 'pulp_docker.plugins.distributors.publish_steps.PublishBlobsStep'>
test_publish_steps: DEBUG: Invariant check: /tmp/tmpWYEMZP/publish/v2/app/foo.json => /tmp/tmpWYEMZP/publish/v2/master/foo/1520898214.84/foo.json
test_publish_steps: DEBUG: Child of step: <class 'pulp_docker.plugins.distributors.publish_steps.PublishManifestsStep'>
test_publish_steps: DEBUG: Invariant check: /tmp/tmpWYEMZP/publish/v2/app/foo.json => /tmp/tmpWYEMZP/publish/v2/master/foo/1520898214.84/foo.json
test_publish_steps: DEBUG: Child of step: <class 'pulp_docker.plugins.distributors.publish_steps.PublishManifestListsStep'>
test_publish_steps: DEBUG: Invariant check: /tmp/tmpWYEMZP/publish/v2/app/foo.json => /tmp/tmpWYEMZP/publish/v2/master/foo/1520898214.84/foo.json
test_publish_steps: DEBUG: Child of step: <class 'pulp_docker.plugins.distributors.publish_steps.PublishTagsStep'>
test_publish_steps: DEBUG: Invariant check: /tmp/tmpWYEMZP/publish/v2/app/foo.json => /tmp/tmpWYEMZP/publish/v2/master/foo/1520898214.84/foo.json
test_publish_steps: DEBUG: Child of step: <class 'pulp_docker.plugins.distributors.publish_steps.RedirectFileStep'>
test_publish_steps: DEBUG: Invariant check: /tmp/tmpWYEMZP/publish/v2/app/foo.json => /tmp/tmpWYEMZP/publish/v2/master/foo/1520898214.84/foo.json
test_publish_steps: DEBUG: Child of step: <class 'pulp.plugins.util.publish_step.AtomicDirectoryPublishStep'>
test_publish_steps: DEBUG: Invariant check: /tmp/tmpWYEMZP/publish/v2/app/foo.json => /tmp/tmpWYEMZP/publish/v2/master/foo/1520898214.84/foo.json
pulp.plugins.util.publish_step: DEBUG: Copying tree from /tmp/tmpWYEMZP/work to /tmp/tmpWYEMZP/publish/v2/master/foo/1520898214.9
pulp.plugins.util.misc: DEBUG: Clearing out directory: /tmp/tmpWYEMZP/publish/v2/master/foo
--------------------- >> end captured logging << ---------------------

Compare with Fedora: the code has been loaded from /usr/lib/python2.7.
The behavior is as it was before the bug was fixed; last two steps are
AtomicDirectoryPublishStep followed by RedirectFileStep:

https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/unittest-pulp_docker-pr/469/node-type=f27-np/consoleText

-------------------- >> begin captured logging << --------------------
test_publish_steps: DEBUG: publish_steps comes from /usr/lib/python2.7/site-packages/pulp_docker/plugins/distributors/publish_steps.pyc
pulp.plugins.util.publish_step: DEBUG: Copying tree from /tmp/tmppG6u6n/work to /tmp/tmppG6u6n/publish/v2/master/foo/1520898110.38
pulp.plugins.util.misc: DEBUG: Clearing out directory: /tmp/tmppG6u6n/publish/v2/master/foo
test_publish_steps: DEBUG: publish_steps comes from /usr/lib/python2.7/site-packages/pulp_docker/plugins/distributors/publish_steps.pyc
test_publish_steps: DEBUG: Child of step: <class 'pulp_docker.plugins.distributors.publish_steps.PublishBlobsStep'>
test_publish_steps: DEBUG: Invariant check: /tmp/tmppG6u6n/publish/v2/app/foo.json => /tmp/tmppG6u6n/publish/v2/master/foo/1520898110.38/foo.json
test_publish_steps: DEBUG: Child of step: <class 'pulp_docker.plugins.distributors.publish_steps.PublishManifestsStep'>
test_publish_steps: DEBUG: Invariant check: /tmp/tmppG6u6n/publish/v2/app/foo.json => /tmp/tmppG6u6n/publish/v2/master/foo/1520898110.38/foo.json
test_publish_steps: DEBUG: Child of step: <class 'pulp_docker.plugins.distributors.publish_steps.PublishManifestListsStep'>
test_publish_steps: DEBUG: Invariant check: /tmp/tmppG6u6n/publish/v2/app/foo.json => /tmp/tmppG6u6n/publish/v2/master/foo/1520898110.38/foo.json
test_publish_steps: DEBUG: Child of step: <class 'pulp_docker.plugins.distributors.publish_steps.PublishTagsStep'>
test_publish_steps: DEBUG: Invariant check: /tmp/tmppG6u6n/publish/v2/app/foo.json => /tmp/tmppG6u6n/publish/v2/master/foo/1520898110.38/foo.json
test_publish_steps: DEBUG: Child of step: <class 'pulp.plugins.util.publish_step.AtomicDirectoryPublishStep'>
test_publish_steps: DEBUG: Invariant check: /tmp/tmppG6u6n/publish/v2/app/foo.json => /tmp/tmppG6u6n/publish/v2/master/foo/1520898110.38/foo.json
pulp.plugins.util.publish_step: DEBUG: Copying tree from /tmp/tmppG6u6n/work to /tmp/tmppG6u6n/publish/v2/master/foo/1520898110.44
pulp.plugins.util.misc: DEBUG: Clearing out directory: /tmp/tmppG6u6n/publish/v2/master/foo
test_publish_steps: DEBUG: Child of step: <class 'pulp_docker.plugins.distributors.publish_steps.RedirectFileStep'>
test_publish_steps: DEBUG: Invariant check: /tmp/tmppG6u6n/publish/v2/app/foo.json => /tmp/tmppG6u6n/publish/v2/master/foo/1520898110.44/foo.json
--------------------- >> end captured logging << ---------------------

So I conclude that the Jenkins setup or test scripts are buggy.  This
probably resulted in accurate test results for various pull requests
already.
<pre>