Project

Profile

Help

Issue #2657

closed

Publishing iso_rsync_distributor fails when iso_distributor is published with serve_https==False

Added by yuzheng about 7 years ago. Updated about 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.12.1
Platform Release:
2.12.2
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 17
Quarter:

Description

When ISO repo in Pulp is not published to https path, publishing iso_rsync_distributor fails due to trying to copy from a non-existing https path.

Reproducing steps:

1. Create an ISO repo which only serves via http
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/ -H "Content-Type: application/json" -H  "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"display_name": "test-repo23", "description": "test repo", "distributors": [{"distributor_id": "iso_distributor", "distributor_type_id": "iso_distributor", "distributor_config": {"serve_https": false, "serve_http": true, "relative_url": "content/dist/test-repo23"}}, {"distributor_id": "cdn_distributor", "distributor_type_id": "iso_rsync_distributor", "distributor_config": {"remote": {"auth_type": "publickey", "ssh_user": "cdn", "host": "10.66.136.68", "root": "/home/cdn/", "ssh_identity_file": "/var/www/.ssh/id_rsa"}, "predistributor_id": "iso_distributor"}}], "notes": {"rsync_repodata": "True", "_repo-type": "iso-repo", "relative_url": "content/dist/test-repo23"}, "importer_type_id": "iso_importer", "id": "test-repo23"}'
{"scratchpad": {}, "display_name": "test-repo23", "description": "test repo", "last_unit_added": null, "notes": {"rsync_repodata": "True", "_repo-type": "iso-repo", "relative_url": "content/dist/test-repo23"}, "last_unit_removed": null, "content_unit_counts": {}, "_ns": "repos", "_id": {"$oid": "58d3856020ed2e2185ee575b"}, "id": "test-repo23", "_href": "/pulp/api/v2/repositories/test-repo23/"}

2. Check distributors for created repo
db.repo_distributors.find({repo_id:"test-repo23"})
/* 0 */
{
    "_id" : ObjectId("58d3856020ed2e2185ee575d"),
    "repo_id" : "test-repo23",
    "distributor_id" : "iso_distributor",
    "distributor_type_id" : "iso_distributor",
    "config" : {
        "serve_https" : false,
        "relative_url" : "content/dist/test-repo23",
        "serve_http" : true
    },
    "auto_publish" : false,
    "last_updated" : ISODate("2017-03-23T08:20:48.592Z"),
    "last_override_config" : {},
    "scratchpad" : {},
    "_ns" : "repo_distributors",
    "last_publish" : ISODate("2017-03-23T08:22:15.719Z")
}

/* 1 */
{
    "_id" : ObjectId("58d3856020ed2e2185ee575e"),
    "repo_id" : "test-repo23",
    "distributor_id" : "cdn_distributor",
    "distributor_type_id" : "iso_rsync_distributor",
    "config" : {
        "remote" : {
            "auth_type" : "publickey",
            "ssh_user" : "cdn",
            "host" : "10.66.136.68",
            "root" : "/home/cdn/",
            "ssh_identity_file" : "/var/www/.ssh/id_rsa"
        },
        "predistributor_id" : "iso_distributor"
    },
    "auto_publish" : false,
    "last_updated" : ISODate("2017-03-23T08:20:48.595Z"),
    "last_override_config" : {},
    "scratchpad" : {},
    "_ns" : "repo_distributors"
}

3. Upload an ISO to this repo
# pulp-admin iso repo uploads upload --repo-id test-repo23 --file 3.iso
+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

...
[\]
Running...

Task Succeeded

Deleting the upload request...
... completed

4. Run ISO publish
# pulp-admin iso repo publish run --repo-id test-repo23
+----------------------------------------------------------------------+
                  Publishing Repository [test-repo18]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

The repository was successfully published.

Task Succeeded

4. Check ISO repo in Pulp
# ll /var/lib/pulp/published/http/isos/content/dist/test-repo23/
total 4
lrwxrwxrwx. 1 apache apache 97 Mar 23 16:22 3.iso -> /var/lib/pulp/content/units/iso/da/3ce6e699362190cb518be82ea132c67a38dda031afe8d79011ee1a2d2cf377
-rw-r--r--. 1 apache apache 80 Mar 23 16:22 PULP_MANIFEST

# ll /var/lib/pulp/published/https/isos/content/dist/test-repo23/
ls: cannot access /var/lib/pulp/published/https/isos/content/dist/test-repo23/: No such file or directory

Content is only published to http path as expected.

5. Run Rsycn publish
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/test-repo23/actions/publish/ -H "Content-Type: application/json" -H  "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"id": "cdn_distributor", "override_config": {}}'

The publish fails. Error log in Pulp server:
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:INFO: (8098-62688) ['rsync', '-rtKOzi', '--files-from', u'/var/cache/pulp/reserved_resource_worker-1@dhcp-136-68.nay.redhat.com/bd0c988e-0062-4556-9974-2c0c3427cb7b/f2360d71-7352-4abc-a967-f2bcaf1aa7a6', '--relative', '-e', u'ssh -l cdn -i /var/www/.ssh/id_rsa -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" -S /tmp/rsync_distributor-%r@%h:%p -o "ControlMaster auto" -o "ControlPersist 10"', '--copy-links', u'/var/lib/pulp/published/https/isos/content/dist/test-repo23', u'cdn@10.66.136.68:/home/cdn/content/dist/test-repo23']
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:INFO: (8098-62688) rsync: link_stat "/var/lib/pulp/published/https/isos/content/dist/test-repo23/PULP_MANIFEST" failed: No such file or directory (2)
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:INFO: (8098-62688) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.9]
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:INFO: (8098-62688)
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:ERROR: (8098-62688) ['rsync', '-rtKOzi', '--files-from', u'/var/cache/pulp/reserved_resource_worker-1@dhcp-136-68.nay.redhat.com/bd0c988e-0062-4556-9974-2c0c3427cb7b/f2360d71-7352-4abc-a967-f2bcaf1aa7a6', '--relative', '-e', u'ssh -l cdn -i /var/www/.ssh/id_rsa -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" -S /tmp/rsync_distributor-%r@%h:%p -o "ControlMaster auto" -o "ControlPersist 10"', '--copy-links', u'/var/lib/pulp/published/https/isos/content/dist/test-repo23', u'cdn@10.66.136.68:/home/cdn/content/dist/test-repo23']
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:ERROR: (8098-62688) rsync: link_stat "/var/lib/pulp/published/https/isos/content/dist/test-repo23/PULP_MANIFEST" failed: No such file or directory (2)
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:ERROR: (8098-62688) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.9]
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:ERROR: (8098-62688)
Actions #1

Updated by ipanova@redhat.com about 7 years ago

The issue is options name

    def _get_root_publish_dir(self):
        """
        Returns the publish directory path for the predistributor.

        :return: absolute path to the master publish directory of predistributor
        :rtype: str
        """
        if self.predistributor["config"].get("http", False):
            return constants.ISO_HTTP_DIR
        else:
            return constants.ISO_HTTPS_DIR

We need to change code to look for 'serve_http'

Actions #2

Updated by ipanova@redhat.com about 7 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com
Actions #3

Updated by bizhang about 7 years ago

  • Sprint/Milestone set to 36
  • Triaged changed from No to Yes

Added by ipanova@redhat.com about 7 years ago

Revision 4878b2e3 | View on GitHub

Fix iso distributor option config name.

closes #2657 https://pulp.plan.io/issues/2657

Actions #5

Updated by ipanova@redhat.com about 7 years ago

  • Status changed from ASSIGNED to POST
Actions #7

Updated by ipanova@redhat.com about 7 years ago

  • Status changed from POST to MODIFIED
Actions #8

Updated by bizhang about 7 years ago

  • Platform Release set to 2.12.2
Actions #9

Updated by Ichimonji10 about 7 years ago

Verified against Pulp 2.12 and 2.13 nightlies on F24, F25 and RHEL 7. For reference, here's the packages installed on one of my test systems:

[root@rhel-7-3-pulp-2-12 ~]# rpm -qa | grep -i pulp | sort
pulp-admin-client-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
pulp-docker-admin-extensions-2.3.1-0.1.alpha.git.5.052c506.el7.noarch
pulp-docker-plugins-2.3.1-0.1.alpha.git.5.052c506.el7.noarch
pulp-ostree-admin-extensions-1.2.2-0.1.alpha.git.3.809be44.el7.noarch
pulp-ostree-plugins-1.2.2-0.1.alpha.git.3.809be44.el7.noarch
pulp-puppet-admin-extensions-2.12.3-0.1.alpha.git.7.89c545a.el7.noarch
pulp-puppet-plugins-2.12.3-0.1.alpha.git.7.89c545a.el7.noarch
pulp-python-admin-extensions-1.1.4-0.1.alpha.git.31.36b75e3.el7.noarch
pulp-python-plugins-1.1.4-0.1.alpha.git.31.36b75e3.el7.noarch
pulp-rpm-admin-extensions-2.12.3-0.1.alpha.git.20.5e18bff.el7.noarch
pulp-rpm-plugins-2.12.3-0.1.alpha.git.20.5e18bff.el7.noarch
pulp-selinux-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
pulp-server-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-6.pulp.el7.noarch
python-pulp-bindings-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
python-pulp-client-lib-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
python-pulp-common-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
python-pulp-docker-common-2.3.1-0.1.alpha.git.5.052c506.el7.noarch
python-pulp-oid_validation-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
python-pulp-ostree-common-1.2.2-0.1.alpha.git.3.809be44.el7.noarch
python-pulp-puppet-common-2.12.3-0.1.alpha.git.7.89c545a.el7.noarch
python-pulp-python-common-1.1.4-0.1.alpha.git.31.36b75e3.el7.noarch
python-pulp-repoauth-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
python-pulp-rpm-common-2.12.3-0.1.alpha.git.20.5e18bff.el7.noarch
python-pulp-streamer-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
Actions #10

Updated by bizhang about 7 years ago

  • Status changed from MODIFIED to 5
Actions #11

Updated by bizhang about 7 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #12

Updated by bmbouter about 6 years ago

  • Sprint set to Sprint 17
Actions #13

Updated by bmbouter about 6 years ago

  • Sprint/Milestone deleted (36)
Actions #14

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF