Project

Profile

Help

Issue #3317

closed

rsync_distributors: 'rsync_extra_args' not used in all calls to rsync

Added by gmbnomis over 6 years ago. Updated about 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Master
Platform Release:
2.15.2
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Easy Fix, Pulp 2
Sprint:
Quarter:

Description

Support for 'rsync_extra_args' (see #2730) missed to add the rsync extra args to the rsync call in remote_mkdir().

Depending on the rsync arguments used, this may work, as remote_mkdir() only creates directories on the remote target. However, parameters like --dry-run or --timeout will not work as expected.

I will provide PRs with a proposed fix and a test for pulp-smash.

Actions #1

Updated by gmbnomis over 6 years ago

PR with fix is here: https://github.com/pulp/pulp/pull/3282
PR for pulp-smash is here: https://github.com/PulpQE/pulp-smash/pull/849

I am not sure about which branches to target and what markup is needed to properly link the PRs to this issue. Let me know if I need to change the PRs.

Actions #2

Updated by dalley about 6 years ago

  • Status changed from NEW to POST
  • Triaged changed from No to Yes
Actions #3

Updated by dalley about 6 years ago

  • Assignee set to gmbnomis

Added by Simon Baatz about 6 years ago

Revision b7705381 | View on GitHub

Actually use rsync_extra_args for all rsync calls

Commit c981b94de498 ("As user I want to be able to provide custom rsync arguments to rsync distributor") missed to add the rsync extra args to the rsync call in remote_mkdir().

closes #3317 https://pulp.plan.io/issues/3317

Actions #4

Updated by Anonymous about 6 years ago

  • Status changed from POST to MODIFIED
Actions #5

Updated by kersom about 6 years ago

Pulp-Smash test related to this issue failed.

[[https://git.io/vN5Ni]]

When publishing the --dry-run the following structure is present on the host.

[root@r7-p215n a0b5ab57-500]# tree
.
├── content
│   └── units
└── f430b0da-8d36-4b54-84f5-7def03e37b82
    └── repodata

Then the set returned is not empty.

Please, clarify expected behaviour.

Actions #6

Updated by kersom about 6 years ago

  • Status changed from MODIFIED to ASSIGNED
Actions #7

Updated by gmbnomis about 6 years ago

Strange. This is the behavior I expect to see before the bug fix (directories are created as the --dry-run flag is not used in remote_mkdir(). No files/links get created since --dry-run is used in the remaining rsync calls).

I can't reproduce the failure. For me, the test runs using the following setup:

  • Use current master of pulp, pulp_rpm and pulp-smash
  • Comment out the skiptest in the RsyncExtraArgsTestCase
  • Run the test in a pulp-devel environment (docker). I get:
 python -m unittest -v pulp_smash.tests.pulp2.rpm.api_v2.test_rsync_distributor.RsyncExtraArgsTestCase
/home/vagrant/.virtualenvs/pulp-smash/lib/python3.5/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
test_all (pulp_smash.tests.pulp2.rpm.api_v2.test_rsync_distributor.RsyncExtraArgsTestCase)
Use the ``rsync_extra_args`` RPM rsync distributor option. ... ok

----------------------------------------------------------------------
Ran 1 test in 38.948s

OK

Could you elaborate on your test setup?

Actions #8

Updated by kersom about 6 years ago

I am using nightly builds of Pulp. I just installed Pulp again, and tested against the following setup.

FAIL: test_all (pulp_smash.tests.pulp2.rpm.api_v2.test_rsync_distributor.RsyncExtraArgsTestCase)
Use the ``rsync_extra_args`` RPM rsync distributor option.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kersom/00_Code/01_Git/pulp-smash/pulp_smash/tests/pulp2/rpm/api_v2/test_rsync_distributor.py", line 821, in test_all
    self.assertSetEqual(files, set())
AssertionError: Items in the first set but not the second:
'content'
'9d1c794c-897f-44d0-a3d5-147ce0f4fafc'

----------------------------------------------------------------------
Ran 1 test in 40.420s

FAILED (failures=1)
[root@r74-p215n ~]# rpm -qa | sort | grep -i pulp
pulp-admin-client-2.15.1-1.git.6.55208a3.el7.noarch
pulp-deb-admin-extensions-1.6.0-1.el7.noarch
pulp-deb-plugins-1.6.0-1.el7.noarch
pulp-docker-admin-extensions-3.1.1-1.git.3.55208a3.el7.noarch
pulp-docker-plugins-3.1.1-1.git.3.55208a3.el7.noarch
pulp-ostree-admin-extensions-1.3.0-1.el7.noarch
pulp-ostree-plugins-1.3.0-1.el7.noarch
pulp-puppet-admin-extensions-2.15.1-1.git.4.55208a3.el7.noarch
pulp-puppet-plugins-2.15.1-1.git.4.55208a3.el7.noarch
pulp-puppet-tools-2.15.1-1.git.4.55208a3.el7.noarch
pulp-python-admin-extensions-2.0.2-1.el7.noarch
pulp-python-plugins-2.0.2-1.el7.noarch
pulp-rpm-admin-extensions-2.15.1-1.git.5.55208a3.el7.noarch
pulp-rpm-plugins-2.15.1-1.git.5.55208a3.el7.noarch
pulp-selinux-2.15.1-1.git.6.55208a3.el7.noarch
pulp-server-2.15.1-1.git.6.55208a3.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-8.pulp.el7.noarch
python-pulp-bindings-2.15.1-1.git.6.55208a3.el7.noarch
python-pulp-client-lib-2.15.1-1.git.6.55208a3.el7.noarch
python-pulp-common-2.15.1-1.git.6.55208a3.el7.noarch
python-pulp-deb-common-1.6.0-1.el7.noarch
python-pulp-docker-common-3.1.1-1.git.3.55208a3.el7.noarch
python-pulp-oid_validation-2.15.1-1.git.6.55208a3.el7.noarch
python-pulp-ostree-common-1.3.0-1.el7.noarch
python-pulp-puppet-common-2.15.1-1.git.4.55208a3.el7.noarch
python-pulp-python-common-2.0.2-1.el7.noarch
python-pulp-repoauth-2.15.1-1.git.6.55208a3.el7.noarch
python-pulp-rpm-common-2.15.1-1.git.5.55208a3.el7.noarch
python-pulp-streamer-2.15.1-1.git.6.55208a3.el7.noarch
[root@r74-p215n ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)
Actions #9

Updated by kersom about 6 years ago

It seems that code that was merged to fix this issue is not being added as part of the nightly build of Pulp.

As soon as I have more info about it, I will update this issue.

Actions #10

Updated by kersom about 6 years ago

  • Status changed from ASSIGNED to MODIFIED
Actions #11

Updated by Ichimonji10 about 6 years ago

  • Platform Release set to 2.15.2

There was a mis-understanding, and we goofed up. The fix for this issue is in the master branch, which will at some point turn into a 2.16 release. However, QE assumed that the fix for this issue was present on the 2.15 branch, and was testing against the 2.15 nightlies.

I anticipate that the fix for this issue will be cherry-picked from the master branch onto the 2.15 branch in time for the 2.15.2 release, so I've set the platform release field accordingly. This has the side-effect of fixing spurious test failures.

Added by Simon Baatz about 6 years ago

Revision 5f1956a2 | View on GitHub

Actually use rsync_extra_args for all rsync calls

Commit c981b94de498 ("As user I want to be able to provide custom rsync arguments to rsync distributor") missed to add the rsync extra args to the rsync call in remote_mkdir().

closes #3317 https://pulp.plan.io/issues/3317

(cherry picked from commit b7705381ce0fd370fe24037910df44ad9d12f310)

Actions #12

Updated by Anonymous about 6 years ago

Actions #13

Updated by daviddavis about 6 years ago

  • Status changed from MODIFIED to 5
Actions #14

Updated by pthomas@redhat.com about 6 years ago

The automated test for this is passing.

Actions #15

Updated by bmbouter about 6 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #16

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF