Project

Profile

Help

Issue #2687

closed

Rsync publish removes symlinks and replace them with directories

Added by yuzheng almost 7 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.12.1
Platform Release:
2.13.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 17
Quarter:

Description

Symlinks in rsync targeted storage will be replaced with directories when doing rsync publish.

Reproducing steps:

1. Create 2 rpm repos with yum_distributor and rpm_rsync_distributor defined
[test-repo01]
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/ -H "Content-Type: application/json" -H  "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"display_name": "test-repo01", "description": "test repo", "distributors": [{"distributor_type_id":"yum_distributor", "distributor_id":"yum_distributor", "distributor_config":{"http":true, "https":false, "relative_url":"content/dist/test-repo01"}},{"distributor_type_id":"rpm_rsync_distributor","distributor_id":"cdn_distributor","distributor_config":{"remote": {"auth_type": "publickey","ssh_user": "cdn", "ssh_identity_file": "/var/www/.ssh/id_rsa", "host": "10.66.136.68", "root": "/home/cdn/"}, "predistributor_id": "yum_distributor"}}], "notes": {"_repo-type": "rpm-repo", "relative_url": "content/dist/test-repo01"}, "importer_type_id": "yum_importer", "id": "test-repo01"}'

[test-repo02]
Create it in the same way as test-repo01.

2. Upload rpm1 to test-repo01, yum publish and rsync publish the repo
# pulp-admin rpm repo uploads rpm --repo-id test-repo01 --file /tmp/dog-4.23-1.noarch.rpm
# pulp-admin rpm repo publish run --repo-id test-repo01
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/test-repo01/actions/publish/ -H "Content-Type: application/json" -H  "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"id": "cdn_distributor", "override_config": {}}'

3. Create symlink on remote server
# sudo -u cdn ln -s /home/cdn/content/dist/test-repo01/ /home/cdn/content/dist/test-repo02

4. Upload rpm2 to test-repo02, yum publish and rsync publish the repo
# pulp-admin rpm repo uploads rpm --repo-id test-repo02 --file /tmp/fox-1.1-2.noarch.rpm
# pulp-admin rpm repo publish run --repo-id test-repo02
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/test-repo02/actions/publish/ -H "Content-Type: application/json" -H  "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"id": "cdn_distributor", "override_config": {}}'

Expected result:

test-repo02 on remote server should contain content of both test-repo01 and test-repo02.

Actual result:

Before rsync publish in step4, test-repo02 is a symlink.
# ll /home/cdn/content/dist/test-repo02
lrwxrwxrwx. 1 cdn cdn 35 Mar 31 13:04 /home/cdn/content/dist/test-repo02 -> /home/cdn/content/dist/test-repo01/

However, after the publish, it's replaced with a directory and loses existing content (content of test-repo01) in it.
# ll /home/cdn/content/dist/test-repo02
total 4
drwxr-xr-x. 3 cdn cdn   15 Mar 31 13:10 Packages
drwxr-xr-x. 2 cdn cdn 4096 Mar 31 13:10 repodata

Adding -K (--keep-dirlinks) for rsync will avoid this issue.

/usr/lib/python2.7/site-packages/pulp/plugins/rsync/publish.py
 65     def remote_mkdir(self, path):
 66         """
 67         Creates path on remote server. The path is rooted in distributor's remote_root directory.
 ...
 78         args = ['rsync', '-avrK', '-f+ */']
Actions #2

Updated by bmbouter almost 7 years ago

+1 to adding this to the sprint

Actions #3

Updated by ttereshc almost 7 years ago

  • Sprint/Milestone set to 36
  • Triaged changed from No to Yes

Added by pcreech almost 7 years ago

Revision 74644572 | View on GitHub

Add provides for python2-* packages

Add provides for python2-* packages. Fedora downstream packages are named python2-* and provide python-*. Upstream needs to mirror this for dependency resolution to work properly

re #2687

Added by pcreech almost 7 years ago

Revision 74644572 | View on GitHub

Add provides for python2-* packages

Add provides for python2-* packages. Fedora downstream packages are named python2-* and provide python-*. Upstream needs to mirror this for dependency resolution to work properly

re #2687

Added by pcreech almost 7 years ago

Revision 74644572 | View on GitHub

Add provides for python2-* packages

Add provides for python2-* packages. Fedora downstream packages are named python2-* and provide python-*. Upstream needs to mirror this for dependency resolution to work properly

re #2687

Actions #5

Updated by fdobrovo almost 7 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to fdobrovo
Actions #6

Updated by fdobrovo almost 7 years ago

  • Status changed from ASSIGNED to POST

Added by fdobrovo almost 7 years ago

Revision 3ca0816b | View on GitHub

Use rsync's -K to preserve symlinks on publish with rsync distributor

closes #2687 https://pulp.plan.io/issues/2687

Actions #7

Updated by fdobrovo almost 7 years ago

  • Status changed from POST to MODIFIED
Actions #8

Updated by pcreech almost 7 years ago

  • Project changed from RPM Support to Pulp
  • Platform Release set to 2.12.2
Actions #9

Updated by pcreech almost 7 years ago

  • Platform Release changed from 2.12.2 to 2.13.0
Actions #10

Updated by pcreech almost 7 years ago

  • Status changed from MODIFIED to 5
Actions #11

Updated by pcreech almost 7 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #12

Updated by bmbouter about 6 years ago

  • Sprint set to Sprint 17
Actions #13

Updated by bmbouter about 6 years ago

  • Sprint/Milestone deleted (36)
Actions #14

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF