Project

Profile

Help

Issue #4550

rpm_rsync_distributor is broken for yum_repo_metadata_file units

Added by rmcgover 8 months ago. Updated 6 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
Start date:
Due date:
Severity:
3. High
Version:
2.18.1
Platform Release:
2.19.1
Blocks Release:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 51

Description

rpm_rsync_distributor doesn't work at all for units of type yum_repo_metadata_file. It will publish broken symlinks to the remote host.

This is due to the following interaction between yum_distributor and rpm_rsync_distributor:

First, in yum_distributor, the metadata file is symlinked from repo publish dir to storage path:

class PublishMetadataStep(platform_steps.UnitModelPluginStep):
...
    def process_main(self, item=None):
        """ 
        Copy the metadata file into place and add it to the repomd file.

        :param item: The unit to process
        :type item: pulp.server.db.model.ContentUnit
        """ 
        unit = item
        # Copy the file to the location on disk where the published repo is built
        publish_location_relative_path = os.path.join(self.get_working_dir(),
                                                      REPO_DATA_DIR_NAME)

        metadata_file_name = os.path.basename(unit._storage_path)
        link_path = os.path.join(publish_location_relative_path, metadata_file_name)
        plugin_misc.create_symlink(unit._storage_path, link_path)

        # Add the proper relative reference to the metadata file to repomd
        self.parent.repomd_file_context.\
            add_metadata_file_metadata(unit.data_type, link_path)

That results in a yum repo with a symlink as in following example:

$ ls -l /var/lib/pulp/published/yum/master/yum_distributor/rhel-ha-for-rhel-7-for-system-z-rpms__7Server__s390x/1552949091.82/repodata/
total 200
-rw-r--r--. 1 apache apache   2087 Mar 18 22:44 29e7003fe4c25a3291e80c16006ca0edc315c8b89f930f8be9f435bb10c1000b-updateinfo.xml.gz
-rw-r--r--. 1 apache apache 160494 Mar 18 22:44 792d01f805670173c58c8c55b1b960576de6b72215bd61d845b077af254d7e14-other.xml.gz
-rw-r--r--. 1 apache apache  16643 Mar 18 22:44 79f6997620d82b1085fb9dc22499b81d1b4ccb8a6113372039273d0dbab5be9f-filelists.xml.gz
lrwxrwxrwx. 1 apache apache    153 Mar 18 22:44 85771b84-06dd-4e4d-9e88-1b1d5616b3cc -> /var/lib/pulp/content/units/yum_repo_metadata_file/6b/74066bca70043c0cec6f1c43bc973a4bb84aa9ff6579757b8b0a9be1bcf81b/85771b84-06dd-4e4d-9e88-1b1d5616b3cc
-rw-r--r--. 1 apache apache    124 Mar 18 22:44 a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
-rw-r--r--. 1 apache apache   3575 Mar 18 22:44 a57cc25570ba03b0c7fc82ad4502fad14fb20b404f065801563bc58ea8340eda-primary.xml.gz
-rw-r--r--. 1 apache apache   2405 Mar 18 22:44 repomd.xml

Then, rpm_rsync_distributor performs rsync of repodata directory with links=True, meaning rsync --links, meaning "copy symlinks as symlinks":

            self.add_child(RSyncPublishStep("Rsync step (repodata)",
                                            repodata_file_list,
                                            "%s/" % os.path.join(master_dir, 'repodata'),
                                            "%s/" % os.path.join(remote_repo_path, 'repodata'),
                                            exclude=[".*", "repodata.old"],
                                            config=config, links=True,
                                            delete=self.config.get("delete")))

This indeed rsyncs the symlink onto the remote host exactly as it is:

$ ls -l /mnt/redhat/cdn/content/dist/rhel/system-z/7/7Server/s390x/highavailability/os/repodata | grep 85771b84-06dd-4e4d-9e88-1b1d5616b3cc
lrwxrwxrwx. 1 vagrant vagrant    153 Mar 18 22:44 85771b84-06dd-4e4d-9e88-1b1d5616b3cc -> /var/lib/pulp/content/units/yum_repo_metadata_file/6b/74066bca70043c0cec6f1c43bc973a4bb84aa9ff6579757b8b0a9be1bcf81b/85771b84-06dd-4e4d-9e88-1b1d5616b3cc

That's a broken symlink to /var/lib/pulp, which is not expected to exist on the remote. Hence the file is not able to be accessed and the published repo is incomplete.

Steps to reproduce

- Create a repo with yum_distributor, rpm_rsync_distributor
- Add any yum_repo_metadata_file to the repo
- Publish the repo with yum_distributor and then rpm_rsync_distributor

Expected result

All yum_repo_metadata_file in the repo are accessible in the repo published to remote host via rpm_rsync_distributor

Actual result

All yum_repo_metadata_file in published repo are broken symlinks.


Related issues

Related to RPM Support - Issue #4560: Publish YumMetadataFiles as files and not symlinks CLOSED - CURRENTRELEASE Actions
Copied to RPM Support - Test #4727: rpm_rsync_distributor is broken for yum_repo_metadata_file units CLOSED - COMPLETE Actions

Associated revisions

Revision 713fedaa View on GitHub
Added by dkliban@redhat.com 8 months ago

Problem: YumMetadataFile is a symlink

Solution: Copy YumMetadataFile to the publish directory

YumCloneDistributor and RPMRsyncDistributor both copy the content of repodata directory.
This results in broken symlinks for all YumMetadataFile content.

fixes: #4550
https://pulp.plan.io/issues/4550

fixes: #4560
https://pulp.plan.io/issues/4560

Revision 8a770299 View on GitHub
Added by dkliban@redhat.com 8 months ago

Problem: YumMetadataFile attributes are not preserved

Solution: use shutil.copy2() to copy YumMetadataFile content type

re: #4550
https://pulp.plan.io/issues/4550

Revision 60cbb8ef View on GitHub
Added by dkliban@redhat.com 6 months ago

Problem: YumMetadataFile is a symlink

Solution: Copy YumMetadataFile to the publish directory

YumCloneDistributor and RPMRsyncDistributor both copy the content of repodata directory.
This results in broken symlinks for all YumMetadataFile content.

fixes: #4550
https://pulp.plan.io/issues/4550

fixes: #4560
https://pulp.plan.io/issues/4560
(cherry picked from commit 713fedaa16b0c65bb2fbd78db47fd250c9fb343e)

Revision 63c3637f View on GitHub
Added by dkliban@redhat.com 6 months ago

Problem: YumMetadataFile attributes are not preserved

Solution: use shutil.copy2() to copy YumMetadataFile content type

re: #4550
https://pulp.plan.io/issues/4550
(cherry picked from commit 8a770299821929764317007c58f1fef7c0cbf2ef)

History

#1 Updated by dkliban@redhat.com 8 months ago

  • Related to Issue #4560: Publish YumMetadataFiles as files and not symlinks added

#2 Updated by dkliban@redhat.com 8 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com
  • Triaged changed from No to Yes
  • Sprint set to Sprint 50

This issue is going to be resolved when we implement a new behavior for YumMetadataFiles. I added that issue as a related to this.

#3 Updated by dkliban@redhat.com 8 months ago

  • Status changed from ASSIGNED to POST

#4 Updated by dkliban@redhat.com 8 months ago

  • Status changed from POST to MODIFIED

#5 Updated by dkliban@redhat.com 8 months ago

  • Status changed from MODIFIED to POST

#7 Updated by rchan 8 months ago

  • Sprint changed from Sprint 50 to Sprint 51

#8 Updated by ipanova@redhat.com 7 months ago

  • Status changed from POST to MODIFIED

#9 Updated by bmbouter 7 months ago

  • Tags Pulp 2 added

#10 Updated by bherring 7 months ago

  • Copied to Test #4727: rpm_rsync_distributor is broken for yum_repo_metadata_file units added

#11 Updated by dkliban@redhat.com 6 months ago

  • Platform Release set to 2.19.1

#12 Updated by dkliban@redhat.com 6 months ago

  • Sprint/Milestone set to 2.19.1

#14 Updated by dkliban@redhat.com 6 months ago

  • Status changed from MODIFIED to ON_QA

#15 Updated by dkliban@redhat.com 6 months ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF