Project

Profile

Help

Issue #1986

Download Error When Syncing OpenShift Registry

Added by lucarval over 5 years ago. Updated over 2 years ago.

Status:
CLOSED - WORKSFORME
Priority:
High
Assignee:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version - Docker:
master
Platform Release:
Target Release - Docker:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

I'm unable to perform a docker repo sync with an Integrated Openshift Registry.
I modified the "docker_importer" config to contain "basic_auth_username" and "basic_auth_password" attributes as suggested in https://pulp.plan.io/issues/1975.

Authentication seems to work properly as some of the task steps are successful.

However, it seems to fail during the "Downloading remote files" stage.

$ pulp-admin docker repo sync run --repo-id devops-osbs-openshift
+----------------------------------------------------------------------+
            Synchronizing Repository [devops-osbs-openshift]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading manifests
[\]
... completed

Copying units already in pulp
[-]
... completed

Copying units already in pulp
[-]
... completed

Downloading remote files
[                                                  ] 0%
0 of 4 items

Task Failed

Content import of
/var/cache/pulp/reserved_resource_worker-2@brew-pulp-docker01.web.qa.ext.phx1.re
dhat.com/fb14c674-0d47-4948-9662-3000bcb10f2f/sha256:478be19d61cd9d6871c440ac574
f7ce9b966c35d51a392dabf99a8dc01a2e924 failed - must be an existing file.

We've identified the file mentioned above is an image layer.

Debugging information indicates that download failed but file was still attempted to be copied, which seems odd. All error messages/tracebacks available are from this copy step which does not appear to be the root cause of the issue considering download failed.

{
"num_success": 0,
"description": "Downloading remote files",
"step_type": "sync_step_download",
"items_total": 3,
"state": "FAILED",
"error_details": [],
"details": "",
"num_failures": 3,
"step_id": "88062216-6722-4522-aebb-f029737450cf",
"num_processed": 3
},
{
"num_success": 0,
"description": "Saving Manifests and Blobs",
"step_type": "sync_step_save",
"items_total": 1,
"state": "FAILED",
"error_details": [
{
    "traceback": "  File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 232, in process\n    self._process_block(item=item)\n\n  File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.
py\", line 291, in _process_block\n    self.process_main(item=item)\n\n  File \"/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py\", line 282, in process_main\n    item.save_and_import_content(os.path.join(self.get_wo
rking_dir(), item.digest))\n\n  File \"/usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py\", line 801, in save_and_import_content\n    self.safe_import_content(path, location)\n\n  File \"/usr/lib/python2.7/site-packages/pu
lp/server/db/model/__init__.py\", line 814, in safe_import_content\n    self.import_content(path, location)\n\n  File \"/usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py\", line 786, in import_content\n    raise exceptions
.PulpCodedException(error_code=error_codes.PLP0037, path=path)\n",
            "error": "Content import of /var/cache/pulp/reserved_resource_worker-2@brew-pulp-docker01.web.qa.ext.phx1.redhat.com/121483e4-7453-454d-ba83-8bc83dba115f/sha256:18c96d12bd777da143dbdac5daaaaa5abb8b77d5697c43611b5be58a5097e220
failed - must be an existing file."
}
],
"details": "",
"num_failures": 1,
"step_id": "755c0fa6-6e9f-48ed-a5fd-212a7ea67028",
"num_processed": 1
},

Importer config being used:


{
"_href": "/pulp/api/v2/repositories/devops-osbs-openshift/importers/docker_importer/",
"_id": {
    "$oid": "57582fed11573f410613dba6"
},
"_ns": "repo_importers",
"config": {
    "basic_auth_password": "*****",
    "basic_auth_username": "lucarval",
    "enable_v1": false,
    "enable_v2": true,
    "feed": "https://my-registry-url",
    "upstream_name": "osbs-qa01/helloworld"
},
"id": "docker_importer",
"importer_type_id": "docker_importer",
"last_sync": null,
"repo_id": "devops-osbs-openshift",
"scratchpad": null
}

History

#1 Updated by lucarval over 5 years ago

Another observation is that if the sync command is executed enough times, it'll eventually succeed.

[vagrant@localhost ~]$ pulp-admin docker repo sync run --repo-id devops-osbs-openshift
+----------------------------------------------------------------------+
            Synchronizing Repository [devops-osbs-openshift]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading manifests
[-]
... completed

Copying units already in pulp
[-]
... completed

Copying units already in pulp
[-]
... completed

Downloading remote files
[/]

Task Failed

Content import of
/var/cache/pulp/reserved_resource_worker-2@brew-pulp-docker01.web.qa.ext.phx1.re
dhat.com/7be9f2a0-0db1-42ec-b501-ebc9c7bc4ab5/sha256:385e281300cc6d88bdd155e0931
fbdfbb1801c2b0265340a40481ee2b733ae66 failed - must be an existing file.

[vagrant@localhost ~]$ pulp-admin docker repo sync run --repo-id devops-osbs-openshift
+----------------------------------------------------------------------+
            Synchronizing Repository [devops-osbs-openshift]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading manifests
[-]
... completed

Copying units already in pulp
[-]
... completed

Copying units already in pulp
[-]
... completed

Downloading remote files
[-]
... completed

Saving Manifests and Blobs
[-]
... completed

Saving Tags
[-]
... completed

Task Succeeded

Task Succeeded

[vagrant@localhost ~]$ pa_tool --password admin --user admin --scheme https         --port 443 --server brew-pulp-docker01.web.qa.ext.phx1.redhat.com         docker repo get --REPO_ID devops-osbs-openshift
/usr/lib/python2.7/site-packages/requests-2.10.0-py2.7.egg/requests/packages/urllib3/connectionpool.py:821: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://
urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests-2.10.0-py2.7.egg/requests/packages/urllib3/connectionpool.py:821: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://
urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
[
    {
        "_href": "/pulp/api/v2/repositories/devops-osbs-openshift/",
        "_id": {
            "$oid": "5759654d11573f410613dc1f"
        },
        "_ns": "repos",
        "content_unit_counts": {
            "docker_blob": 2,
            "docker_manifest": 1,
            "docker_tag": 1
        },
        "description": null,
        "display_name": null,
        "id": "devops-osbs-openshift",
        "last_unit_added": null,
        "last_unit_removed": null,
        "notes": {
            "_repo-type": "docker-repo"
        },
        "scratchpad": {}
    }
]

However, there is no "docker_images" key in "content_unit_counts". I'm not sure if this is expected for v2 registries.

#2 Updated by mhrivnak over 5 years ago

Could you add log output from the pulp processes, showing their output during a failed sync? That will help determine what exactly is going wrong.

Here is info about how to access the log output: http://pulp.readthedocs.io/en/latest/user-guide/troubleshooting.html#logging

Thanks!

#4 Updated by lucarval over 5 years ago

Verified that blob files were not actually copied, but this doesn't seem to fail the sync.

#5 Updated by amacdona@redhat.com over 5 years ago

  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High
  • Triaged changed from No to Yes

#6 Updated by lucarval over 5 years ago

The issue seems to be related to firewall rules. Accessing the Openshift registry redirects to Amazon S3 URL which is blocked on pulp server for this deployment.

Verified this flow works properly on a deployment that does not have such firewall restriction.

#7 Updated by mhrivnak about 5 years ago

  • Status changed from NEW to CLOSED - WORKSFORME

Thanks for following up. I'm closing since the problem was caused by firewall rules.

#8 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF