Issue #3317
closed
rsync_distributors: 'rsync_extra_args' not used in all calls to rsync
Description
Support for 'rsync_extra_args' (see #2730) missed to add the rsync extra args to the rsync call in remote_mkdir().
Depending on the rsync arguments used, this may work, as remote_mkdir() only creates directories on the remote target. However, parameters like --dry-run or --timeout will not work as expected.
I will provide PRs with a proposed fix and a test for pulp-smash.
Updated by gmbnomis about 5 years ago
PR with fix is here: https://github.com/pulp/pulp/pull/3282
PR for pulp-smash is here: https://github.com/PulpQE/pulp-smash/pull/849
I am not sure about which branches to target and what markup is needed to properly link the PRs to this issue. Let me know if I need to change the PRs.
Updated by dalley about 5 years ago
- Status changed from NEW to POST
- Triaged changed from No to Yes
Added by Simon Baatz about 5 years ago
Updated by Anonymous about 5 years ago
- Status changed from POST to MODIFIED
Applied in changeset pulp|b7705381ce0fd370fe24037910df44ad9d12f310.
Updated by kersom about 5 years ago
Pulp-Smash test related to this issue failed.
[[https://git.io/vN5Ni]]
When publishing the --dry-run the following structure is present on the host.
[root@r7-p215n a0b5ab57-500]# tree
.
├── content
│ └── units
└── f430b0da-8d36-4b54-84f5-7def03e37b82
└── repodata
Then the set returned is not empty.
Please, clarify expected behaviour.
Updated by gmbnomis about 5 years ago
Strange. This is the behavior I expect to see before the bug fix (directories are created as the --dry-run flag is not used in remote_mkdir(). No files/links get created since --dry-run is used in the remaining rsync calls).
I can't reproduce the failure. For me, the test runs using the following setup:
- Use current master of pulp, pulp_rpm and pulp-smash
- Comment out the skiptest in the
RsyncExtraArgsTestCase - Run the test in a pulp-devel environment (docker). I get:
python -m unittest -v pulp_smash.tests.pulp2.rpm.api_v2.test_rsync_distributor.RsyncExtraArgsTestCase
/home/vagrant/.virtualenvs/pulp-smash/lib/python3.5/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
test_all (pulp_smash.tests.pulp2.rpm.api_v2.test_rsync_distributor.RsyncExtraArgsTestCase)
Use the ``rsync_extra_args`` RPM rsync distributor option. ... ok
----------------------------------------------------------------------
Ran 1 test in 38.948s
OK
Could you elaborate on your test setup?
Updated by kersom about 5 years ago
I am using nightly builds of Pulp. I just installed Pulp again, and tested against the following setup.
FAIL: test_all (pulp_smash.tests.pulp2.rpm.api_v2.test_rsync_distributor.RsyncExtraArgsTestCase)
Use the ``rsync_extra_args`` RPM rsync distributor option.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/kersom/00_Code/01_Git/pulp-smash/pulp_smash/tests/pulp2/rpm/api_v2/test_rsync_distributor.py", line 821, in test_all
self.assertSetEqual(files, set())
AssertionError: Items in the first set but not the second:
'content'
'9d1c794c-897f-44d0-a3d5-147ce0f4fafc'
----------------------------------------------------------------------
Ran 1 test in 40.420s
FAILED (failures=1)
[root@r74-p215n ~]# rpm -qa | sort | grep -i pulp
pulp-admin-client-2.15.1-1.git.6.55208a3.el7.noarch
pulp-deb-admin-extensions-1.6.0-1.el7.noarch
pulp-deb-plugins-1.6.0-1.el7.noarch
pulp-docker-admin-extensions-3.1.1-1.git.3.55208a3.el7.noarch
pulp-docker-plugins-3.1.1-1.git.3.55208a3.el7.noarch
pulp-ostree-admin-extensions-1.3.0-1.el7.noarch
pulp-ostree-plugins-1.3.0-1.el7.noarch
pulp-puppet-admin-extensions-2.15.1-1.git.4.55208a3.el7.noarch
pulp-puppet-plugins-2.15.1-1.git.4.55208a3.el7.noarch
pulp-puppet-tools-2.15.1-1.git.4.55208a3.el7.noarch
pulp-python-admin-extensions-2.0.2-1.el7.noarch
pulp-python-plugins-2.0.2-1.el7.noarch
pulp-rpm-admin-extensions-2.15.1-1.git.5.55208a3.el7.noarch
pulp-rpm-plugins-2.15.1-1.git.5.55208a3.el7.noarch
pulp-selinux-2.15.1-1.git.6.55208a3.el7.noarch
pulp-server-2.15.1-1.git.6.55208a3.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-8.pulp.el7.noarch
python-pulp-bindings-2.15.1-1.git.6.55208a3.el7.noarch
python-pulp-client-lib-2.15.1-1.git.6.55208a3.el7.noarch
python-pulp-common-2.15.1-1.git.6.55208a3.el7.noarch
python-pulp-deb-common-1.6.0-1.el7.noarch
python-pulp-docker-common-3.1.1-1.git.3.55208a3.el7.noarch
python-pulp-oid_validation-2.15.1-1.git.6.55208a3.el7.noarch
python-pulp-ostree-common-1.3.0-1.el7.noarch
python-pulp-puppet-common-2.15.1-1.git.4.55208a3.el7.noarch
python-pulp-python-common-2.0.2-1.el7.noarch
python-pulp-repoauth-2.15.1-1.git.6.55208a3.el7.noarch
python-pulp-rpm-common-2.15.1-1.git.5.55208a3.el7.noarch
python-pulp-streamer-2.15.1-1.git.6.55208a3.el7.noarch
[root@r74-p215n ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)
Updated by kersom about 5 years ago
It seems that code that was merged to fix this issue is not being added as part of the nightly build of Pulp.
As soon as I have more info about it, I will update this issue.
Updated by Ichimonji10 about 5 years ago
- Platform Release set to 2.15.2
There was a mis-understanding, and we goofed up. The fix for this issue is in the master branch, which will at some point turn into a 2.16 release. However, QE assumed that the fix for this issue was present on the 2.15 branch, and was testing against the 2.15 nightlies.
I anticipate that the fix for this issue will be cherry-picked from the master branch onto the 2.15 branch in time for the 2.15.2 release, so I've set the platform release field accordingly. This has the side-effect of fixing spurious test failures.
Added by Simon Baatz about 5 years ago
Actually use rsync_extra_args for all rsync calls
Commit c981b94de498 ("As user I want to be able to provide custom rsync arguments to rsync distributor") missed to add the rsync extra args to the rsync call in remote_mkdir().
closes #3317 https://pulp.plan.io/issues/3317
(cherry picked from commit b7705381ce0fd370fe24037910df44ad9d12f310)
Updated by Anonymous about 5 years ago
Applied in changeset pulp|5f1956a26d63505c75f08608145951571920a5b1.
Updated by bmbouter about 5 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
.
Actually use rsync_extra_args for all rsync calls
Commit c981b94de498 ("As user I want to be able to provide custom rsync arguments to rsync distributor") missed to add the rsync extra args to the rsync call in remote_mkdir().
closes #3317 https://pulp.plan.io/issues/3317