Issue #2536
closed
error for docker pull of non-existent tag not clear
Description
docker version: docker-1.10.3-53.gite03ddb8.fc24.x86_64
Against a katello dev server
# docker pull devel.example.com:5000/examplecorp-production-rhelbase-r
hcc-rhel:7.3-45
Trying to pull repository devel.example.com:5000/examplecorp-production-rhelbase-rhcc-rhel ...
Error parsing HTTP response: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML
PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>400 Bad Request</title>\n</head><body
>\n<h1>Bad Request</h1>\n<p>Your browser sent a request that this server could not understand.<br
/>\nReason: You're speaking plain HTTP to an SSL-enabled server port.<br />\n Instead use the HTTP
S scheme to access this URL, please.<br />\n</p>\n</body></html>\n"
From /var/log/httpd/crane_access_ssl.log
192.168.100.1 - - [19/Jan/2017:16:29:59 +0000] "GET /v2/ HTTP/1.1" 200 2 "-" "docker/1.10.3 go/go1.6.3 kernel/4.7.9-200.fc24.x86_64 os/linux arch/amd64"
192.168.100.1 - - [19/Jan/2017:16:29:59 +0000] "GET /v2/examplecorp-production-rhelbase-rhcc-rhel/manifests/7.3-45 HTTP/1.1" 302 405 "-" "docker/1.10.3 go/go1.6.3 kernel/4.7.9-200.fc24.x86_64 os/linux arch/amd64"
192.168.100.1 - - [19/Jan/2017:16:30:00 +0000] "GET / HTTP/1.0" 400 362 "-" "-"
192.168.100.1 - - [19/Jan/2017:16:30:00 +0000] "GET / HTTP/1.0" 400 362 "-" "-"
Other examples
# docker pull docker.io/thomasmckay/hammer:abcdef
Trying to pull repository docker.io/thomasmckay/hammer ...
Pulling repository docker.io/thomasmckay/hammer
Tag abcdef not found in repository docker.io/thomasmckay/hammer
# docker pull registry.access.redhat.com/rhel7/rhel:abcdef123
Trying to pull repository registry.access.redhat.com/rhel7/rhel ...
Error parsing HTTP response: invalid character 'F' looking for beginning of value: "File not found.\""
Related issues
Updated by tomckay@redhat.com almost 8 years ago
[root@devel ~]# rpm -qa | grep crane
python-crane-2.0.2-1.el7.noarch
[root@devel ~]# rpm -qa | grep pulp
python-pulp-repoauth-2.11.0-1.el7.noarch
pulp-selinux-2.11.0-1.el7.noarch
pulp-rpm-admin-extensions-2.11.0-1.el7.noarch
python-pulp-ostree-common-1.2.0-1.el7.noarch
pulp-puppet-plugins-2.11.0-1.el7.noarch
rubygem-smart_proxy_pulp-1.3.0-1.el7.noarch
python-pulp-common-2.11.0-1.el7.noarch
python-pulp-oid_validation-2.11.0-1.el7.noarch
pulp-admin-client-2.11.0-1.el7.noarch
pulp-docker-plugins-2.2.0-1.el7.noarch
pulp-ostree-plugins-1.2.0-1.el7.noarch
pulp-client-1.0-1.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-6.pulp.el7.noarch
python-pulp-rpm-common-2.11.0-1.el7.noarch
python-pulp-bindings-2.11.0-1.el7.noarch
pulp-server-2.11.0-1.el7.noarch
python-pulp-streamer-2.11.0-1.el7.noarch
python-pulp-client-lib-2.11.0-1.el7.noarch
pulp-rpm-plugins-2.11.0-1.el7.noarch
python-pulp-docker-common-2.2.0-1.el7.noarch
python-pulp-puppet-common-2.11.0-1.el7.noarch
pulp-docker-admin-extensions-2.2.0-1.el7.noarch
pulp-katello-1.0.2-1.el7.noarch
Updated by mhrivnak almost 8 years ago
We need to figure out why "You're speaking plain HTTP to an SSL-enabled server port."
It looks like a 302 redirect happened correctly, which should be to the final destination where a tag might live. Then the 400 happens.
Updated by tomckay@redhat.com almost 8 years ago
# docker pull sat62.example.com:5000/default_organization-rhcc-openshift3_jenkins-1-rhel7:latest
Trying to pull repository sat62.example.com:5000/default_organization-rhcc-openshift3_jenkins-1-rhel7 ...
latest: Pulling from sat62.example.com:5000/default_organization-rhcc-openshift3_jenkins-1-rhel7
7bd78273b666: Pull complete
c196631bd9ac: Pull complete
e14fc2b1e39f: Pull complete
Digest: sha256:4b32e3826a5391610d73a9a0097c328007ab1fa25d9021ee4b07e810acca9c34
Status: Downloaded newer image for sat62.example.com:5000/default_organization-rhcc-openshift3_jenkins-1-rhel7:latest
# tail -f /var/log/httpd/crane_*.log
==> /var/log/httpd/crane_access_ssl.log <==
==> /var/log/httpd/crane_error_ssl.log <==
==> /var/log/httpd/crane_access_ssl.log <==
192.168.121.1 - - [24/Jan/2017:15:52:20 +0000] "GET /v2/ HTTP/1.1" 200 2 "-" "docker/1.10.3 go/go1.6.3 kernel/4.6.7-300.fc24.x86_64 os/linux arch/amd64"
192.168.121.1 - - [24/Jan/2017:15:52:20 +0000] "GET /v2/default_organization-rhcc-openshift3_jenkins-1-rhel7/manifests/latest HTTP/1.1" 302 427 "-" "docker/1.10.3 go/go1.6.3 kernel/4.6.7-300.fc24.x86_64 os/linux arch/amd64"
192.168.121.1 - - [24/Jan/2017:15:52:20 +0000] "GET /v2/default_organization-rhcc-openshift3_jenkins-1-rhel7/blobs/sha256:c196631bd9ac47f0e62cd3b0160159ccf34a88b47a9487a0c3dd3c55b457d607 HTTP/1.1" 302 549 "-" "docker/1.10.3 go/go1.6.3 kernel/4.6.7-300.fc24.x86_64 os/linux arch/amd64"
192.168.121.1 - - [24/Jan/2017:15:52:20 +0000] "GET /v2/default_organization-rhcc-openshift3_jenkins-1-rhel7/blobs/sha256:7bd78273b66657ac8b3e800506047866ce94eea0b50e23ecdb76b0a8fbc5cdcc HTTP/1.1" 302 549 "-" "docker/1.10.3 go/go1.6.3 kernel/4.6.7-300.fc24.x86_64 os/linux arch/amd64"
192.168.121.1 - - [24/Jan/2017:15:52:20 +0000] "GET /v2/default_organization-rhcc-openshift3_jenkins-1-rhel7/blobs/sha256:e14fc2b1e39fc223ecd4e20dca2c04b84057715f5d81528e12285e3ce162254b HTTP/1.1" 302 549 "-" "docker/1.10.3 go/go1.6.3 kernel/4.6.7-300.fc24.x86_64 os/linux arch/amd64"
# docker pull sat62.example.com:5000/default_organization-rhcc-openshift3_jenkins-1-rhel7:wrong
Trying to pull repository sat62.example.com:5000/default_organization-rhcc-openshift3_jenkins-1-rhel7 ...
Error parsing HTTP response: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>400 Bad Request</title>\n</head><body>\n<h1>Bad Request</h1>\n<p>Your browser sent a request that this server could not understand.<br >\nReason: You're speaking plain HTTP to an SSL-enabled server port.<br >\n Instead use the HTTPS scheme to access this URL, please.<br >\n<p>\n<body><html>\n"
==> /var/log/httpd/crane_access_ssl.log <==
192.168.121.1 - - [24/Jan/2017:15:53:54 +0000] "GET /v2/ HTTP/1.1" 200 2 "-" "docker/1.10.3 go/go1.6.3 kernel/4.6.7-300.fc24.x86_64 os/linux arch/amd64"
192.168.121.1 - - [24/Jan/2017:15:53:54 +0000] "GET /v2/default_organization-rhcc-openshift3_jenkins-1-rhel7/manifests/wrong HTTP/1.1" 302 425 "-" "docker/1.10.3 go/go1.6.3 kernel/4.6.7-300.fc24.x86_64 os/linux arch/amd64"
192.168.121.1 - - [24/Jan/2017:15:53:54 +0000] "GET / HTTP/1.0" 400 362 "-" "-"
192.168.121.1 - - [24/Jan/2017:15:53:54 +0000] "GET / HTTP/1.0" 400 362 "-" "-"
- rpm -qa | grep pulp-server
pulp-server-2.8.7.5-1.el7sat.noarch - rpm -qa | grep crane
python-crane-2.0.2.1-1.el7sat.noarch
Updated by bizhang almost 8 years ago
- Sprint/Milestone set to 32
- Triaged changed from No to Yes
Updated by daviddavis almost 8 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to daviddavis
Updated by daviddavis almost 8 years ago
I'm getting a similar but different error:
[vagrant@dev ~]$ sudo docker pull localhost:5001/busybox:abcdef
Trying to pull repository localhost:5001/busybox ...
Error parsing HTTP response: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /pulp/docker/v2/busybox/manifests/abcdef was not found on this server.</p>\n</body></html>\n"
Going to dig into the docker code to see what it's expecting. It looks like a 404 is being returned successfully (at least in my case). I think the difference in what I am seeing might be that I'm hitting crane directly (not through httpd).
Updated by daviddavis almost 8 years ago
Talking with mhrivnak, it looks like docker is not handling redirects properly. I've opened an upstream issue against docker:
Updated by daviddavis over 7 years ago
So it turns out the redirect wasn't the problem. The problem is the body. It's html and docker wants json. We need a response with something like this which is what we return if the repo is missing:
{"errors": [{"code": "404", "message": "Not Found"}]}
I think we can check the tags in crane and then return a 404 if the tag does not exist instead of redirecting to httpd.
Updated by daviddavis over 7 years ago
Assuming we want to have crane return 404 for a tag that doesn't exist, getting the tags for a v2 repo is a bit tricky. In v1, the tags are simply loaded from the <data_dir>/app/<repo>.json file:
https://github.com/pulp/crane/blob/75ae1ecd4b7a2fe4fdcd021863f2c08f908f81c4/crane/data.py#L57
For v2, the tags aren't in this file. It looks like they're in <data_dir>/web/<repo>/tags/list. Therefore, I think we need to crawl those tag files and pull out the tags when we start crane. Then we'll save the tags on the V2 tuple so we can validate them later when a request is made for a particular tag.
Does that make sense? Is there another solution?
Updated by daviddavis over 7 years ago
Per @ipanova, the format of the json needs to be:
{"errors": [{"code": "MANIFEST_UNKNOWN","message": "manifest unknown","detail": {}}]}
Also, need to make sure the headers are properly set:
response = current_app.make_response(json.dumps(x))
response.headers['Content-Type'] = 'application/json'
response.headers['Docker-Distribution-API-Version'] = 'registry/2.0'
response.status_code = 404
return response
Updated by daviddavis over 7 years ago
- Related to Story #2735: Add docs about v1 protocol limitation added
Updated by ipanova@redhat.com over 7 years ago
- Status changed from ASSIGNED to NEW
- Assignee deleted (
daviddavis)
I am moving this back to NEW state since noone is working on this at the moment.
Updated by tomckay@redhat.com over 7 years ago
That's unfortunate; it is a bad experience for users.
Updated by bmbouter over 5 years ago
- Status changed from NEW to CLOSED - WONTFIX
Updated by bmbouter over 5 years ago
Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.
.