Project

Profile

Help

Issue #2536

error for docker pull of non-existent tag not clear

Added by tomckay@redhat.com almost 4 years ago. Updated almost 2 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version - Crane:
Platform Release:
Target Release - Crane:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description


docker version: docker-1.10.3-53.gite03ddb8.fc24.x86_64

Against a katello dev server

# docker pull devel.example.com:5000/examplecorp-production-rhelbase-r
hcc-rhel:7.3-45
Trying to pull repository devel.example.com:5000/examplecorp-production-rhelbase-rhcc-rhel ... 
Error parsing HTTP response: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML
 PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>400 Bad Request</title>\n</head><body
>\n<h1>Bad Request</h1>\n<p>Your browser sent a request that this server could not understand.<br 
/>\nReason: You're speaking plain HTTP to an SSL-enabled server port.<br />\n Instead use the HTTP
S scheme to access this URL, please.<br />\n</p>\n</body></html>\n"

From /var/log/httpd/crane_access_ssl.log

192.168.100.1 - - [19/Jan/2017:16:29:59 +0000] "GET /v2/ HTTP/1.1" 200 2 "-" "docker/1.10.3 go/go1.6.3 kernel/4.7.9-200.fc24.x86_64 os/linux arch/amd64"
192.168.100.1 - - [19/Jan/2017:16:29:59 +0000] "GET /v2/examplecorp-production-rhelbase-rhcc-rhel/manifests/7.3-45 HTTP/1.1" 302 405 "-" "docker/1.10.3 go/go1.6.3 kernel/4.7.9-200.fc24.x86_64 os/linux arch/amd64"
192.168.100.1 - - [19/Jan/2017:16:30:00 +0000] "GET / HTTP/1.0" 400 362 "-" "-"
192.168.100.1 - - [19/Jan/2017:16:30:00 +0000] "GET / HTTP/1.0" 400 362 "-" "-"

Other examples

# docker pull docker.io/thomasmckay/hammer:abcdef
Trying to pull repository docker.io/thomasmckay/hammer ... 
Pulling repository docker.io/thomasmckay/hammer
Tag abcdef not found in repository docker.io/thomasmckay/hammer

# docker pull registry.access.redhat.com/rhel7/rhel:abcdef123
Trying to pull repository registry.access.redhat.com/rhel7/rhel ... 
Error parsing HTTP response: invalid character 'F' looking for beginning of value: "File not found.\""

Related issues

Related to Docker Support - Story #2735: Add docs about v1 protocol limitation CLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

History

#1 Updated by tomckay@redhat.com almost 4 years ago

[root@devel ~]# rpm -qa | grep crane
python-crane-2.0.2-1.el7.noarch
[root@devel ~]# rpm -qa | grep pulp
python-pulp-repoauth-2.11.0-1.el7.noarch
pulp-selinux-2.11.0-1.el7.noarch
pulp-rpm-admin-extensions-2.11.0-1.el7.noarch
python-pulp-ostree-common-1.2.0-1.el7.noarch
pulp-puppet-plugins-2.11.0-1.el7.noarch
rubygem-smart_proxy_pulp-1.3.0-1.el7.noarch
python-pulp-common-2.11.0-1.el7.noarch
python-pulp-oid_validation-2.11.0-1.el7.noarch
pulp-admin-client-2.11.0-1.el7.noarch
pulp-docker-plugins-2.2.0-1.el7.noarch
pulp-ostree-plugins-1.2.0-1.el7.noarch
pulp-client-1.0-1.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-6.pulp.el7.noarch
python-pulp-rpm-common-2.11.0-1.el7.noarch
python-pulp-bindings-2.11.0-1.el7.noarch
pulp-server-2.11.0-1.el7.noarch
python-pulp-streamer-2.11.0-1.el7.noarch
python-pulp-client-lib-2.11.0-1.el7.noarch
pulp-rpm-plugins-2.11.0-1.el7.noarch
python-pulp-docker-common-2.2.0-1.el7.noarch
python-pulp-puppet-common-2.11.0-1.el7.noarch
pulp-docker-admin-extensions-2.2.0-1.el7.noarch
pulp-katello-1.0.2-1.el7.noarch

#2 Updated by mhrivnak almost 4 years ago

We need to figure out why "You're speaking plain HTTP to an SSL-enabled server port."

It looks like a 302 redirect happened correctly, which should be to the final destination where a tag might live. Then the 400 happens.

#3 Updated by tomckay@redhat.com almost 4 years ago

# docker pull sat62.example.com:5000/default_organization-rhcc-openshift3_jenkins-1-rhel7:latest
Trying to pull repository sat62.example.com:5000/default_organization-rhcc-openshift3_jenkins-1-rhel7 ... 
latest: Pulling from sat62.example.com:5000/default_organization-rhcc-openshift3_jenkins-1-rhel7
7bd78273b666: Pull complete 
c196631bd9ac: Pull complete 
e14fc2b1e39f: Pull complete 
Digest: sha256:4b32e3826a5391610d73a9a0097c328007ab1fa25d9021ee4b07e810acca9c34
Status: Downloaded newer image for sat62.example.com:5000/default_organization-rhcc-openshift3_jenkins-1-rhel7:latest

# tail -f /var/log/httpd/crane_*.log
==> /var/log/httpd/crane_access_ssl.log <==

==> /var/log/httpd/crane_error_ssl.log <==

==> /var/log/httpd/crane_access_ssl.log <==
192.168.121.1 - - [24/Jan/2017:15:52:20 +0000] "GET /v2/ HTTP/1.1" 200 2 "-" "docker/1.10.3 go/go1.6.3 kernel/4.6.7-300.fc24.x86_64 os/linux arch/amd64"
192.168.121.1 - - [24/Jan/2017:15:52:20 +0000] "GET /v2/default_organization-rhcc-openshift3_jenkins-1-rhel7/manifests/latest HTTP/1.1" 302 427 "-" "docker/1.10.3 go/go1.6.3 kernel/4.6.7-300.fc24.x86_64 os/linux arch/amd64"
192.168.121.1 - - [24/Jan/2017:15:52:20 +0000] "GET /v2/default_organization-rhcc-openshift3_jenkins-1-rhel7/blobs/sha256:c196631bd9ac47f0e62cd3b0160159ccf34a88b47a9487a0c3dd3c55b457d607 HTTP/1.1" 302 549 "-" "docker/1.10.3 go/go1.6.3 kernel/4.6.7-300.fc24.x86_64 os/linux arch/amd64"
192.168.121.1 - - [24/Jan/2017:15:52:20 +0000] "GET /v2/default_organization-rhcc-openshift3_jenkins-1-rhel7/blobs/sha256:7bd78273b66657ac8b3e800506047866ce94eea0b50e23ecdb76b0a8fbc5cdcc HTTP/1.1" 302 549 "-" "docker/1.10.3 go/go1.6.3 kernel/4.6.7-300.fc24.x86_64 os/linux arch/amd64"
192.168.121.1 - - [24/Jan/2017:15:52:20 +0000] "GET /v2/default_organization-rhcc-openshift3_jenkins-1-rhel7/blobs/sha256:e14fc2b1e39fc223ecd4e20dca2c04b84057715f5d81528e12285e3ce162254b HTTP/1.1" 302 549 "-" "docker/1.10.3 go/go1.6.3 kernel/4.6.7-300.fc24.x86_64 os/linux arch/amd64"
# docker pull sat62.example.com:5000/default_organization-rhcc-openshift3_jenkins-1-rhel7:wrong
Trying to pull repository sat62.example.com:5000/default_organization-rhcc-openshift3_jenkins-1-rhel7 ... 
Error parsing HTTP response: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>400 Bad Request</title>\n</head><body>\n<h1>Bad Request</h1>\n<p>Your browser sent a request that this server could not understand.<br >\nReason: You're speaking plain HTTP to an SSL-enabled server port.<br >\n Instead use the HTTPS scheme to access this URL, please.<br >\n<p>\n<body><html>\n"

==> /var/log/httpd/crane_access_ssl.log <==
192.168.121.1 - - [24/Jan/2017:15:53:54 +0000] "GET /v2/ HTTP/1.1" 200 2 "-" "docker/1.10.3 go/go1.6.3 kernel/4.6.7-300.fc24.x86_64 os/linux arch/amd64"
192.168.121.1 - - [24/Jan/2017:15:53:54 +0000] "GET /v2/default_organization-rhcc-openshift3_jenkins-1-rhel7/manifests/wrong HTTP/1.1" 302 425 "-" "docker/1.10.3 go/go1.6.3 kernel/4.6.7-300.fc24.x86_64 os/linux arch/amd64"
192.168.121.1 - - [24/Jan/2017:15:53:54 +0000] "GET / HTTP/1.0" 400 362 "-" "-"
192.168.121.1 - - [24/Jan/2017:15:53:54 +0000] "GET / HTTP/1.0" 400 362 "-" "-"
  1. rpm -qa | grep pulp-server
    pulp-server-2.8.7.5-1.el7sat.noarch
  2. rpm -qa | grep crane
    python-crane-2.0.2.1-1.el7sat.noarch

#4 Updated by bizhang almost 4 years ago

  • Sprint/Milestone set to 32
  • Triaged changed from No to Yes

#5 Updated by daviddavis almost 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to daviddavis

#6 Updated by mhrivnak almost 4 years ago

  • Sprint/Milestone changed from 32 to 33

#7 Updated by daviddavis almost 4 years ago

I'm getting a similar but different error:

[vagrant@dev ~]$ sudo docker pull localhost:5001/busybox:abcdef
Trying to pull repository localhost:5001/busybox ... 
Error parsing HTTP response: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /pulp/docker/v2/busybox/manifests/abcdef was not found on this server.</p>\n</body></html>\n"

Going to dig into the docker code to see what it's expecting. It looks like a 404 is being returned successfully (at least in my case). I think the difference in what I am seeing might be that I'm hitting crane directly (not through httpd).

#8 Updated by daviddavis almost 4 years ago

Talking with mhrivnak, it looks like docker is not handling redirects properly. I've opened an upstream issue against docker:

https://github.com/docker/docker/issues/30798

#9 Updated by mhrivnak almost 4 years ago

  • Sprint/Milestone deleted (33)

#10 Updated by daviddavis over 3 years ago

So it turns out the redirect wasn't the problem. The problem is the body. It's html and docker wants json. We need a response with something like this which is what we return if the repo is missing:

{"errors": [{"code": "404", "message": "Not Found"}]}

I think we can check the tags in crane and then return a 404 if the tag does not exist instead of redirecting to httpd.

#11 Updated by daviddavis over 3 years ago

Assuming we want to have crane return 404 for a tag that doesn't exist, getting the tags for a v2 repo is a bit tricky. In v1, the tags are simply loaded from the <data_dir>/app/<repo>.json file:

https://github.com/pulp/crane/blob/75ae1ecd4b7a2fe4fdcd021863f2c08f908f81c4/crane/data.py#L57

For v2, the tags aren't in this file. It looks like they're in <data_dir>/web/<repo>/tags/list. Therefore, I think we need to crawl those tag files and pull out the tags when we start crane. Then we'll save the tags on the V2 tuple so we can validate them later when a request is made for a particular tag.

Does that make sense? Is there another solution?

#12 Updated by daviddavis over 3 years ago

Per @ipanova, the format of the json needs to be:

{"errors": [{"code": "MANIFEST_UNKNOWN","message": "manifest unknown","detail": {}}]}

Also, need to make sure the headers are properly set:

    response = current_app.make_response(json.dumps(x))
    response.headers['Content-Type'] = 'application/json'
    response.headers['Docker-Distribution-API-Version'] = 'registry/2.0'
    response.status_code = 404
    return response

#13 Updated by daviddavis over 3 years ago

  • Related to Story #2735: Add docs about v1 protocol limitation added

#14 Updated by ipanova@redhat.com over 3 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (daviddavis)

I am moving this back to NEW state since noone is working on this at the moment.

#15 Updated by tomckay@redhat.com over 3 years ago

That's unfortunate; it is a bad experience for users.

#16 Updated by bmbouter almost 2 years ago

  • Status changed from NEW to CLOSED - WONTFIX

#17 Updated by bmbouter almost 2 years ago

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

#18 Updated by bmbouter almost 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF