Project

Profile

Help

Issue #2687

Rsync publish removes symlinks and replace them with directories

Added by yuzheng over 3 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.12.1
Platform Release:
2.13.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 17

Description

Symlinks in rsync targeted storage will be replaced with directories when doing rsync publish.

Reproducing steps:

1. Create 2 rpm repos with yum_distributor and rpm_rsync_distributor defined
[test-repo01]
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/ -H "Content-Type: application/json" -H  "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"display_name": "test-repo01", "description": "test repo", "distributors": [{"distributor_type_id":"yum_distributor", "distributor_id":"yum_distributor", "distributor_config":{"http":true, "https":false, "relative_url":"content/dist/test-repo01"}},{"distributor_type_id":"rpm_rsync_distributor","distributor_id":"cdn_distributor","distributor_config":{"remote": {"auth_type": "publickey","ssh_user": "cdn", "ssh_identity_file": "/var/www/.ssh/id_rsa", "host": "10.66.136.68", "root": "/home/cdn/"}, "predistributor_id": "yum_distributor"}}], "notes": {"_repo-type": "rpm-repo", "relative_url": "content/dist/test-repo01"}, "importer_type_id": "yum_importer", "id": "test-repo01"}'

[test-repo02]
Create it in the same way as test-repo01.

2. Upload rpm1 to test-repo01, yum publish and rsync publish the repo
# pulp-admin rpm repo uploads rpm --repo-id test-repo01 --file /tmp/dog-4.23-1.noarch.rpm
# pulp-admin rpm repo publish run --repo-id test-repo01
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/test-repo01/actions/publish/ -H "Content-Type: application/json" -H  "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"id": "cdn_distributor", "override_config": {}}'

3. Create symlink on remote server
# sudo -u cdn ln -s /home/cdn/content/dist/test-repo01/ /home/cdn/content/dist/test-repo02

4. Upload rpm2 to test-repo02, yum publish and rsync publish the repo
# pulp-admin rpm repo uploads rpm --repo-id test-repo02 --file /tmp/fox-1.1-2.noarch.rpm
# pulp-admin rpm repo publish run --repo-id test-repo02
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/test-repo02/actions/publish/ -H "Content-Type: application/json" -H  "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"id": "cdn_distributor", "override_config": {}}'

Expected result:

test-repo02 on remote server should contain content of both test-repo01 and test-repo02.

Actual result:

Before rsync publish in step4, test-repo02 is a symlink.
# ll /home/cdn/content/dist/test-repo02
lrwxrwxrwx. 1 cdn cdn 35 Mar 31 13:04 /home/cdn/content/dist/test-repo02 -> /home/cdn/content/dist/test-repo01/

However, after the publish, it's replaced with a directory and loses existing content (content of test-repo01) in it.
# ll /home/cdn/content/dist/test-repo02
total 4
drwxr-xr-x. 3 cdn cdn   15 Mar 31 13:10 Packages
drwxr-xr-x. 2 cdn cdn 4096 Mar 31 13:10 repodata

Adding -K (--keep-dirlinks) for rsync will avoid this issue.

/usr/lib/python2.7/site-packages/pulp/plugins/rsync/publish.py
 65     def remote_mkdir(self, path):
 66         """
 67         Creates path on remote server. The path is rooted in distributor's remote_root directory.
 ...
 78         args = ['rsync', '-avrK', '-f+ */']

Associated revisions

Revision 74644572 View on GitHub
Added by pcreech over 3 years ago

Add provides for python2-* packages

Add provides for python2-* packages. Fedora downstream packages are named python2-* and provide python-*. Upstream needs to mirror this for dependency resolution to work properly

re #2687

Revision 74644572 View on GitHub
Added by pcreech over 3 years ago

Add provides for python2-* packages

Add provides for python2-* packages. Fedora downstream packages are named python2-* and provide python-*. Upstream needs to mirror this for dependency resolution to work properly

re #2687

Revision 74644572 View on GitHub
Added by pcreech over 3 years ago

Add provides for python2-* packages

Add provides for python2-* packages. Fedora downstream packages are named python2-* and provide python-*. Upstream needs to mirror this for dependency resolution to work properly

re #2687

Revision 3ca0816b View on GitHub
Added by fdobrovo over 3 years ago

Use rsync's -K to preserve symlinks on publish with rsync distributor

closes #2687 https://pulp.plan.io/issues/2687

History

#2 Updated by bmbouter over 3 years ago

+1 to adding this to the sprint

#3 Updated by ttereshc over 3 years ago

  • Sprint/Milestone set to 36
  • Triaged changed from No to Yes

#5 Updated by fdobrovo over 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to fdobrovo

#6 Updated by fdobrovo over 3 years ago

  • Status changed from ASSIGNED to POST

#7 Updated by fdobrovo over 3 years ago

  • Status changed from POST to MODIFIED

#8 Updated by pcreech over 3 years ago

  • Project changed from RPM Support to Pulp
  • Platform Release set to 2.12.2

#9 Updated by pcreech over 3 years ago

  • Platform Release changed from 2.12.2 to 2.13.0

#10 Updated by pcreech over 3 years ago

  • Status changed from MODIFIED to 5

#11 Updated by pcreech over 3 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#12 Updated by bmbouter over 2 years ago

  • Sprint set to Sprint 17

#13 Updated by bmbouter over 2 years ago

  • Sprint/Milestone deleted (36)

#14 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF