Project

Profile

Help

Issue #2657

Publishing iso_rsync_distributor fails when iso_distributor is published with serve_https==False

Added by yuzheng about 3 years ago. Updated about 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Sprint/Milestone:
-
Start date:
Due date:
Severity:
2. Medium
Version:
2.12.1
Platform Release:
2.12.2
Blocks Release:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
Yes
Verification Required:
Yes
Sprint:
Sprint 17

Description

When ISO repo in Pulp is not published to https path, publishing iso_rsync_distributor fails due to trying to copy from a non-existing https path.

Reproducing steps:

1. Create an ISO repo which only serves via http
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/ -H "Content-Type: application/json" -H  "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"display_name": "test-repo23", "description": "test repo", "distributors": [{"distributor_id": "iso_distributor", "distributor_type_id": "iso_distributor", "distributor_config": {"serve_https": false, "serve_http": true, "relative_url": "content/dist/test-repo23"}}, {"distributor_id": "cdn_distributor", "distributor_type_id": "iso_rsync_distributor", "distributor_config": {"remote": {"auth_type": "publickey", "ssh_user": "cdn", "host": "10.66.136.68", "root": "/home/cdn/", "ssh_identity_file": "/var/www/.ssh/id_rsa"}, "predistributor_id": "iso_distributor"}}], "notes": {"rsync_repodata": "True", "_repo-type": "iso-repo", "relative_url": "content/dist/test-repo23"}, "importer_type_id": "iso_importer", "id": "test-repo23"}'
{"scratchpad": {}, "display_name": "test-repo23", "description": "test repo", "last_unit_added": null, "notes": {"rsync_repodata": "True", "_repo-type": "iso-repo", "relative_url": "content/dist/test-repo23"}, "last_unit_removed": null, "content_unit_counts": {}, "_ns": "repos", "_id": {"$oid": "58d3856020ed2e2185ee575b"}, "id": "test-repo23", "_href": "/pulp/api/v2/repositories/test-repo23/"}

2. Check distributors for created repo
db.repo_distributors.find({repo_id:"test-repo23"})
/* 0 */
{
    "_id" : ObjectId("58d3856020ed2e2185ee575d"),
    "repo_id" : "test-repo23",
    "distributor_id" : "iso_distributor",
    "distributor_type_id" : "iso_distributor",
    "config" : {
        "serve_https" : false,
        "relative_url" : "content/dist/test-repo23",
        "serve_http" : true
    },
    "auto_publish" : false,
    "last_updated" : ISODate("2017-03-23T08:20:48.592Z"),
    "last_override_config" : {},
    "scratchpad" : {},
    "_ns" : "repo_distributors",
    "last_publish" : ISODate("2017-03-23T08:22:15.719Z")
}

/* 1 */
{
    "_id" : ObjectId("58d3856020ed2e2185ee575e"),
    "repo_id" : "test-repo23",
    "distributor_id" : "cdn_distributor",
    "distributor_type_id" : "iso_rsync_distributor",
    "config" : {
        "remote" : {
            "auth_type" : "publickey",
            "ssh_user" : "cdn",
            "host" : "10.66.136.68",
            "root" : "/home/cdn/",
            "ssh_identity_file" : "/var/www/.ssh/id_rsa"
        },
        "predistributor_id" : "iso_distributor"
    },
    "auto_publish" : false,
    "last_updated" : ISODate("2017-03-23T08:20:48.595Z"),
    "last_override_config" : {},
    "scratchpad" : {},
    "_ns" : "repo_distributors"
}

3. Upload an ISO to this repo
# pulp-admin iso repo uploads upload --repo-id test-repo23 --file 3.iso
+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

...
[\]
Running...

Task Succeeded

Deleting the upload request...
... completed

4. Run ISO publish
# pulp-admin iso repo publish run --repo-id test-repo23
+----------------------------------------------------------------------+
                  Publishing Repository [test-repo18]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

The repository was successfully published.

Task Succeeded

4. Check ISO repo in Pulp
# ll /var/lib/pulp/published/http/isos/content/dist/test-repo23/
total 4
lrwxrwxrwx. 1 apache apache 97 Mar 23 16:22 3.iso -> /var/lib/pulp/content/units/iso/da/3ce6e699362190cb518be82ea132c67a38dda031afe8d79011ee1a2d2cf377
-rw-r--r--. 1 apache apache 80 Mar 23 16:22 PULP_MANIFEST

# ll /var/lib/pulp/published/https/isos/content/dist/test-repo23/
ls: cannot access /var/lib/pulp/published/https/isos/content/dist/test-repo23/: No such file or directory

Content is only published to http path as expected.

5. Run Rsycn publish
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/test-repo23/actions/publish/ -H "Content-Type: application/json" -H  "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"id": "cdn_distributor", "override_config": {}}'

The publish fails. Error log in Pulp server:
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:INFO: (8098-62688) ['rsync', '-rtKOzi', '--files-from', u'/var/cache/pulp/reserved_resource_worker-1@dhcp-136-68.nay.redhat.com/bd0c988e-0062-4556-9974-2c0c3427cb7b/f2360d71-7352-4abc-a967-f2bcaf1aa7a6', '--relative', '-e', u'ssh -l cdn -i /var/www/.ssh/id_rsa -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" -S /tmp/rsync_distributor-%r@%h:%p -o "ControlMaster auto" -o "ControlPersist 10"', '--copy-links', u'/var/lib/pulp/published/https/isos/content/dist/test-repo23', u'cdn@10.66.136.68:/home/cdn/content/dist/test-repo23']
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:INFO: (8098-62688) rsync: link_stat "/var/lib/pulp/published/https/isos/content/dist/test-repo23/PULP_MANIFEST" failed: No such file or directory (2)
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:INFO: (8098-62688) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.9]
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:INFO: (8098-62688)
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:ERROR: (8098-62688) ['rsync', '-rtKOzi', '--files-from', u'/var/cache/pulp/reserved_resource_worker-1@dhcp-136-68.nay.redhat.com/bd0c988e-0062-4556-9974-2c0c3427cb7b/f2360d71-7352-4abc-a967-f2bcaf1aa7a6', '--relative', '-e', u'ssh -l cdn -i /var/www/.ssh/id_rsa -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" -S /tmp/rsync_distributor-%r@%h:%p -o "ControlMaster auto" -o "ControlPersist 10"', '--copy-links', u'/var/lib/pulp/published/https/isos/content/dist/test-repo23', u'cdn@10.66.136.68:/home/cdn/content/dist/test-repo23']
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:ERROR: (8098-62688) rsync: link_stat "/var/lib/pulp/published/https/isos/content/dist/test-repo23/PULP_MANIFEST" failed: No such file or directory (2)
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:ERROR: (8098-62688) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.9]
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:ERROR: (8098-62688)

Associated revisions

Revision 4878b2e3 View on GitHub
Added by ipanova@redhat.com about 3 years ago

Fix iso distributor option config name.

closes #2657 https://pulp.plan.io/issues/2657

History

#1 Updated by ipanova@redhat.com about 3 years ago

The issue is options name

    def _get_root_publish_dir(self):
        """
        Returns the publish directory path for the predistributor.

        :return: absolute path to the master publish directory of predistributor
        :rtype: str
        """
        if self.predistributor["config"].get("http", False):
            return constants.ISO_HTTP_DIR
        else:
            return constants.ISO_HTTPS_DIR

We need to change code to look for 'serve_http'

#2 Updated by ipanova@redhat.com about 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com

#3 Updated by bizhang about 3 years ago

  • Sprint/Milestone set to 36
  • Triaged changed from No to Yes

#4 Updated by pthomas@redhat.com about 3 years ago

  • Smash Test set to 602

#5 Updated by ipanova@redhat.com about 3 years ago

  • Status changed from ASSIGNED to POST

#6 Updated by ipanova@redhat.com about 3 years ago

  • Verification Required changed from No to Yes

#7 Updated by ipanova@redhat.com about 3 years ago

  • Status changed from POST to MODIFIED

#8 Updated by bizhang about 3 years ago

  • Platform Release set to 2.12.2

#9 Updated by Ichimonji10 about 3 years ago

  • Verified changed from No to Yes

Verified against Pulp 2.12 and 2.13 nightlies on F24, F25 and RHEL 7. For reference, here's the packages installed on one of my test systems:

[root@rhel-7-3-pulp-2-12 ~]# rpm -qa | grep -i pulp | sort
pulp-admin-client-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
pulp-docker-admin-extensions-2.3.1-0.1.alpha.git.5.052c506.el7.noarch
pulp-docker-plugins-2.3.1-0.1.alpha.git.5.052c506.el7.noarch
pulp-ostree-admin-extensions-1.2.2-0.1.alpha.git.3.809be44.el7.noarch
pulp-ostree-plugins-1.2.2-0.1.alpha.git.3.809be44.el7.noarch
pulp-puppet-admin-extensions-2.12.3-0.1.alpha.git.7.89c545a.el7.noarch
pulp-puppet-plugins-2.12.3-0.1.alpha.git.7.89c545a.el7.noarch
pulp-python-admin-extensions-1.1.4-0.1.alpha.git.31.36b75e3.el7.noarch
pulp-python-plugins-1.1.4-0.1.alpha.git.31.36b75e3.el7.noarch
pulp-rpm-admin-extensions-2.12.3-0.1.alpha.git.20.5e18bff.el7.noarch
pulp-rpm-plugins-2.12.3-0.1.alpha.git.20.5e18bff.el7.noarch
pulp-selinux-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
pulp-server-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-6.pulp.el7.noarch
python-pulp-bindings-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
python-pulp-client-lib-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
python-pulp-common-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
python-pulp-docker-common-2.3.1-0.1.alpha.git.5.052c506.el7.noarch
python-pulp-oid_validation-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
python-pulp-ostree-common-1.2.2-0.1.alpha.git.3.809be44.el7.noarch
python-pulp-puppet-common-2.12.3-0.1.alpha.git.7.89c545a.el7.noarch
python-pulp-python-common-1.1.4-0.1.alpha.git.31.36b75e3.el7.noarch
python-pulp-repoauth-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
python-pulp-rpm-common-2.12.3-0.1.alpha.git.20.5e18bff.el7.noarch
python-pulp-streamer-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch

#10 Updated by bizhang about 3 years ago

  • Status changed from MODIFIED to ON_QA

#11 Updated by bizhang about 3 years ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#12 Updated by bmbouter about 2 years ago

  • Sprint set to Sprint 17

#13 Updated by bmbouter about 2 years ago

  • Sprint/Milestone deleted (36)

#14 Updated by bmbouter about 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF