Project

Profile

Help

Issue #2657

Publishing iso_rsync_distributor fails when iso_distributor is published with serve_https==False

Added by yuzheng almost 3 years ago. Updated 9 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Severity:
2. Medium
Version:
2.12.1
Platform Release:
2.12.2
Blocks Release:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
Yes
Verification Required:
Yes
Sprint:
Sprint 17

Description

When ISO repo in Pulp is not published to https path, publishing iso_rsync_distributor fails due to trying to copy from a non-existing https path.

Reproducing steps:

1. Create an ISO repo which only serves via http
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/ -H "Content-Type: application/json" -H  "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"display_name": "test-repo23", "description": "test repo", "distributors": [{"distributor_id": "iso_distributor", "distributor_type_id": "iso_distributor", "distributor_config": {"serve_https": false, "serve_http": true, "relative_url": "content/dist/test-repo23"}}, {"distributor_id": "cdn_distributor", "distributor_type_id": "iso_rsync_distributor", "distributor_config": {"remote": {"auth_type": "publickey", "ssh_user": "cdn", "host": "10.66.136.68", "root": "/home/cdn/", "ssh_identity_file": "/var/www/.ssh/id_rsa"}, "predistributor_id": "iso_distributor"}}], "notes": {"rsync_repodata": "True", "_repo-type": "iso-repo", "relative_url": "content/dist/test-repo23"}, "importer_type_id": "iso_importer", "id": "test-repo23"}'
{"scratchpad": {}, "display_name": "test-repo23", "description": "test repo", "last_unit_added": null, "notes": {"rsync_repodata": "True", "_repo-type": "iso-repo", "relative_url": "content/dist/test-repo23"}, "last_unit_removed": null, "content_unit_counts": {}, "_ns": "repos", "_id": {"$oid": "58d3856020ed2e2185ee575b"}, "id": "test-repo23", "_href": "/pulp/api/v2/repositories/test-repo23/"}

2. Check distributors for created repo
db.repo_distributors.find({repo_id:"test-repo23"})
/* 0 */
{
    "_id" : ObjectId("58d3856020ed2e2185ee575d"),
    "repo_id" : "test-repo23",
    "distributor_id" : "iso_distributor",
    "distributor_type_id" : "iso_distributor",
    "config" : {
        "serve_https" : false,
        "relative_url" : "content/dist/test-repo23",
        "serve_http" : true
    },
    "auto_publish" : false,
    "last_updated" : ISODate("2017-03-23T08:20:48.592Z"),
    "last_override_config" : {},
    "scratchpad" : {},
    "_ns" : "repo_distributors",
    "last_publish" : ISODate("2017-03-23T08:22:15.719Z")
}

/* 1 */
{
    "_id" : ObjectId("58d3856020ed2e2185ee575e"),
    "repo_id" : "test-repo23",
    "distributor_id" : "cdn_distributor",
    "distributor_type_id" : "iso_rsync_distributor",
    "config" : {
        "remote" : {
            "auth_type" : "publickey",
            "ssh_user" : "cdn",
            "host" : "10.66.136.68",
            "root" : "/home/cdn/",
            "ssh_identity_file" : "/var/www/.ssh/id_rsa" 
        },
        "predistributor_id" : "iso_distributor" 
    },
    "auto_publish" : false,
    "last_updated" : ISODate("2017-03-23T08:20:48.595Z"),
    "last_override_config" : {},
    "scratchpad" : {},
    "_ns" : "repo_distributors" 
}

3. Upload an ISO to this repo
# pulp-admin iso repo uploads upload --repo-id test-repo23 --file 3.iso
+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

...
[\]
Running...

Task Succeeded

Deleting the upload request...
... completed

4. Run ISO publish
# pulp-admin iso repo publish run --repo-id test-repo23
+----------------------------------------------------------------------+
                  Publishing Repository [test-repo18]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

The repository was successfully published.

Task Succeeded

4. Check ISO repo in Pulp
# ll /var/lib/pulp/published/http/isos/content/dist/test-repo23/
total 4
lrwxrwxrwx. 1 apache apache 97 Mar 23 16:22 3.iso -> /var/lib/pulp/content/units/iso/da/3ce6e699362190cb518be82ea132c67a38dda031afe8d79011ee1a2d2cf377
-rw-r--r--. 1 apache apache 80 Mar 23 16:22 PULP_MANIFEST

# ll /var/lib/pulp/published/https/isos/content/dist/test-repo23/
ls: cannot access /var/lib/pulp/published/https/isos/content/dist/test-repo23/: No such file or directory

Content is only published to http path as expected.

5. Run Rsycn publish
$ curl -k -X POST https://10.66.136.68/pulp/api/v2/repositories/test-repo23/actions/publish/ -H "Content-Type: application/json" -H  "Authorization: Basic YWRtaW46YWRtaW4=" -d '{"id": "cdn_distributor", "override_config": {}}'

The publish fails. Error log in Pulp server:
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:INFO: (8098-62688) ['rsync', '-rtKOzi', '--files-from', u'/var/cache/pulp/reserved_resource_worker-1@dhcp-136-68.nay.redhat.com/bd0c988e-0062-4556-9974-2c0c3427cb7b/f2360d71-7352-4abc-a967-f2bcaf1aa7a6', '--relative', '-e', u'ssh -l cdn -i /var/www/.ssh/id_rsa -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" -S /tmp/rsync_distributor-%r@%h:%p -o "ControlMaster auto" -o "ControlPersist 10"', '--copy-links', u'/var/lib/pulp/published/https/isos/content/dist/test-repo23', u'cdn@10.66.136.68:/home/cdn/content/dist/test-repo23']
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:INFO: (8098-62688) rsync: link_stat "/var/lib/pulp/published/https/isos/content/dist/test-repo23/PULP_MANIFEST" failed: No such file or directory (2)
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:INFO: (8098-62688) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.9]
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:INFO: (8098-62688)
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:ERROR: (8098-62688) ['rsync', '-rtKOzi', '--files-from', u'/var/cache/pulp/reserved_resource_worker-1@dhcp-136-68.nay.redhat.com/bd0c988e-0062-4556-9974-2c0c3427cb7b/f2360d71-7352-4abc-a967-f2bcaf1aa7a6', '--relative', '-e', u'ssh -l cdn -i /var/www/.ssh/id_rsa -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" -S /tmp/rsync_distributor-%r@%h:%p -o "ControlMaster auto" -o "ControlPersist 10"', '--copy-links', u'/var/lib/pulp/published/https/isos/content/dist/test-repo23', u'cdn@10.66.136.68:/home/cdn/content/dist/test-repo23']
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:ERROR: (8098-62688) rsync: link_stat "/var/lib/pulp/published/https/isos/content/dist/test-repo23/PULP_MANIFEST" failed: No such file or directory (2)
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:ERROR: (8098-62688) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.9]
Mar 23 16:23:10 dhcp-136-68 pulp: pulp.plugins.rsync.publish:ERROR: (8098-62688)

Associated revisions

History

#1 Updated by ipanova@redhat.com almost 3 years ago

The issue is options name

    def _get_root_publish_dir(self):
        """ 
        Returns the publish directory path for the predistributor.

        :return: absolute path to the master publish directory of predistributor
        :rtype: str
        """ 
        if self.predistributor["config"].get("http", False):
            return constants.ISO_HTTP_DIR
        else:
            return constants.ISO_HTTPS_DIR

We need to change code to look for 'serve_http'

#2 Updated by ipanova@redhat.com almost 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com

#3 Updated by bizhang almost 3 years ago

  • Sprint/Milestone set to 36
  • Triaged changed from No to Yes

#4 Updated by pthomas@redhat.com almost 3 years ago

  • Smash Test set to 602

#5 Updated by ipanova@redhat.com almost 3 years ago

  • Status changed from ASSIGNED to POST

#6 Updated by ipanova@redhat.com almost 3 years ago

  • Verification Required changed from No to Yes

#7 Updated by ipanova@redhat.com almost 3 years ago

  • Status changed from POST to MODIFIED

#8 Updated by bizhang almost 3 years ago

  • Platform Release set to 2.12.2

#9 Updated by Ichimonji10 almost 3 years ago

  • Verified changed from No to Yes

Verified against Pulp 2.12 and 2.13 nightlies on F24, F25 and RHEL 7. For reference, here's the packages installed on one of my test systems:

[root@rhel-7-3-pulp-2-12 ~]# rpm -qa | grep -i pulp | sort
pulp-admin-client-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
pulp-docker-admin-extensions-2.3.1-0.1.alpha.git.5.052c506.el7.noarch
pulp-docker-plugins-2.3.1-0.1.alpha.git.5.052c506.el7.noarch
pulp-ostree-admin-extensions-1.2.2-0.1.alpha.git.3.809be44.el7.noarch
pulp-ostree-plugins-1.2.2-0.1.alpha.git.3.809be44.el7.noarch
pulp-puppet-admin-extensions-2.12.3-0.1.alpha.git.7.89c545a.el7.noarch
pulp-puppet-plugins-2.12.3-0.1.alpha.git.7.89c545a.el7.noarch
pulp-python-admin-extensions-1.1.4-0.1.alpha.git.31.36b75e3.el7.noarch
pulp-python-plugins-1.1.4-0.1.alpha.git.31.36b75e3.el7.noarch
pulp-rpm-admin-extensions-2.12.3-0.1.alpha.git.20.5e18bff.el7.noarch
pulp-rpm-plugins-2.12.3-0.1.alpha.git.20.5e18bff.el7.noarch
pulp-selinux-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
pulp-server-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-6.pulp.el7.noarch
python-pulp-bindings-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
python-pulp-client-lib-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
python-pulp-common-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
python-pulp-docker-common-2.3.1-0.1.alpha.git.5.052c506.el7.noarch
python-pulp-oid_validation-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
python-pulp-ostree-common-1.2.2-0.1.alpha.git.3.809be44.el7.noarch
python-pulp-puppet-common-2.12.3-0.1.alpha.git.7.89c545a.el7.noarch
python-pulp-python-common-1.1.4-0.1.alpha.git.31.36b75e3.el7.noarch
python-pulp-repoauth-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch
python-pulp-rpm-common-2.12.3-0.1.alpha.git.20.5e18bff.el7.noarch
python-pulp-streamer-2.12.3-0.1.alpha.git.1202.52400ed.el7.noarch

#10 Updated by bizhang almost 3 years ago

  • Status changed from MODIFIED to ON_QA

#11 Updated by bizhang almost 3 years ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#12 Updated by bmbouter almost 2 years ago

  • Sprint set to Sprint 17

#13 Updated by bmbouter almost 2 years ago

  • Sprint/Milestone deleted (36)

#14 Updated by bmbouter 9 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF