Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #2643

closed

DKR1008: Could not find registry API at https://docker-registry.engineering.redhat.com

Added by tomckay@redhat.com about 7 years ago. Updated about 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
ipanova@redhat.com
Start date:
Due date:
Estimated time:
Severity:
3. High
Version - Docker:
Platform Release:
2.13.0
Target Release - Docker:
2.4.0
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 18
Quarter:

Description

Internal registry https://docker-registry.engineering.redhat.com cannot sync repos into foreman. 'docker pull' works

docker pull docker-registry.engineering.redhat.com/thomasmckay/freewill:v2

Packages:

[root@devel ~]# rpm -qa | grep pulp
pulp-server-2.12.0-1.el7.noarch
pulp-puppet-plugins-2.12.0-1.el7.noarch
pulp-selinux-2.12.0-1.el7.noarch
python-pulp-client-lib-2.12.0-1.el7.noarch
python-pulp-ostree-common-1.2.0-1.el7.noarch
pulp-client-1.0-1.noarch
python-pulp-docker-common-2.3.0-1.el7.noarch
pulp-docker-admin-extensions-2.3.0-1.el7.noarch
python-pulp-puppet-common-2.12.0-1.el7.noarch
python-pulp-oid_validation-2.12.0-1.el7.noarch
pulp-rpm-admin-extensions-2.12.0-1.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-6.pulp.el7.noarch
python-pulp-common-2.12.0-1.el7.noarch
python-pulp-rpm-common-2.12.0-1.el7.noarch
pulp-admin-client-2.12.0-1.el7.noarch
pulp-puppet-tools-2.12.0-1.el7.noarch
pulp-docker-plugins-2.3.0-1.el7.noarch
python-pulp-bindings-2.12.0-1.el7.noarch
pulp-ostree-plugins-1.2.0-1.el7.noarch
pulp-katello-1.0.2-1.el7.noarch
python-pulp-repoauth-2.12.0-1.el7.noarch
pulp-rpm-plugins-2.12.0-1.el7.noarch
python-pulp-streamer-2.12.0-1.el7.noarch
rubygem-smart_proxy_pulp-1.3.0-1.el7.noarch

Log:

Mar 17 15:36:30 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: Running task : [9337920c-83ba-454f-a9b4-8cfb79d8f478]
Mar 17 15:36:30 devel.example.com pulp[10325]: pulp_docker.plugins.importers.sync:DEBUG: v1 API skipped due to config
Mar 17 15:36:30 devel.example.com pulp[10325]: pulp_docker.plugins.registry:DEBUG: Determining if the registry URL can do v2 of the Docker API.
Mar 17 15:36:30 devel.example.com pulp[10325]: pulp_docker.plugins.registry:DEBUG: Retrieving https://docker-registry.engineering.redhat.com/v2/
Mar 17 15:36:30 devel.example.com pulp[10325]: nectar.downloaders.threaded:DEBUG: Attempting to connect to https://docker-registry.engineering.redhat.com/v2/.
Mar 17 15:36:30 devel.example.com pulp[10325]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): docker-registry.engineering.redhat.com
Mar 17 15:36:30 devel.example.com pulp[10312]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[4d1f4ce3-c31c-44c6-8637-301bbb902a2a] succeeded in 0.0419807829894s: None
Mar 17 15:36:30 devel.example.com pulp[10180]: celery.worker.job:DEBUG: Task accepted: pulp.server.managers.repo.sync.sync[9337920c-83ba-454f-a9b4-8cfb79d8f478] pid:10325
Mar 17 15:36:30 devel.example.com pulp[10180]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[2bff1f0d-b39c-435a-97b9-dff17929a209]
Mar 17 15:36:30 devel.example.com pulp[10456]: pulp.server.webservices.views.decorators:DEBUG: User preauthenticated: admin
Mar 17 15:36:31 devel.example.com pulp[10325]: requests.packages.urllib3.connectionpool:DEBUG: "GET /v2/ HTTP/1.1" 401 87
Mar 17 15:36:31 devel.example.com pulp[10325]: nectar.downloaders.threaded:DEBUG: download failed: Download of https://docker-registry.engineering.redhat.com/v2/ failed with code 401: Unauthorized
Mar 17 15:36:31 devel.example.com pulp[10325]: pulp_docker.plugins.registry:DEBUG: Download unauthorized, attempting to retrieve a token.
Mar 17 15:36:31 devel.example.com pulp[10325]: pulp_docker.plugins.token_util:DEBUG: Requesting token from https://docker-registry.engineering.redhat.com/openshift/token
Mar 17 15:36:31 devel.example.com pulp[10325]: nectar.downloaders.threaded:DEBUG: Attempting to connect to https://docker-registry.engineering.redhat.com/openshift/token.
Mar 17 15:36:31 devel.example.com pulp[10325]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): docker-registry.engineering.redhat.com
Mar 17 15:36:31 devel.example.com pulp[10456]: pulp.server.webservices.views.decorators:DEBUG: User preauthenticated: admin
Mar 17 15:36:31 devel.example.com pulp[10325]: requests.packages.urllib3.connectionpool:DEBUG: "GET /openshift/token HTTP/1.1" 200 49
Mar 17 15:36:31 devel.example.com pulp[10325]: nectar.downloaders.threaded:DEBUG: Attempting to connect to https://docker-registry.engineering.redhat.com/v2/.
Mar 17 15:36:31 devel.example.com pulp[10325]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): docker-registry.engineering.redhat.com
Mar 17 15:36:31 devel.example.com pulp[10456]: pulp.server.webservices.views.decorators:DEBUG: User preauthenticated: admin
Mar 17 15:36:32 devel.example.com pulp[10325]: requests.packages.urllib3.connectionpool:DEBUG: "GET /v2/ HTTP/1.1" 401 87
Mar 17 15:36:32 devel.example.com pulp[10325]: nectar.downloaders.threaded:DEBUG: download failed: Download of https://docker-registry.engineering.redhat.com/v2/ failed with code 401: Unauthorized
Mar 17 15:36:32 devel.example.com pulp[10459]: pulp.server.webservices.views.decorators:DEBUG: User preauthenticated: admin
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:INFO: Task failed : [9337920c-83ba-454f-a9b4-8cfb79d8f478] : Could not find registry API at https://docker-registry.engineering.redhat.com
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296) Traceback (most recent call last):
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)     R = retval = fun(*args, **kwargs)
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 505, in __call__
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)     return super(Task, self).__call__(*args, **kwargs)
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 108, in __call__
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)     return super(PulpTask, self).__call__(*args, **kwargs)
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)     return self.run(*args, **kwargs)
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 762, in sync
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)     sync_report = sync_repo(transfer_repo, conduit, call_config)
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 708, in wrap_f
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)     return f(*args, **kwargs)
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)   File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 83, in sync_repo
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)     self.sync_step = sync.SyncStep(repo=repo, conduit=sync_conduit, config=config)
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)   File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py", line 89, in __init__
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)     raise PulpCodedException(error_code=error_codes.DKR1008, registry=url)
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296) PulpCodedException: Could not find registry API at https://docker-registry.engineering.redhat.com
Mar 17 15:36:32 devel.example.com pulp[10325]: pulp.server.async.tasks:DEBUG: (10325-43296)
Actions
Copy link
 #2

Updated by tomckay@redhat.com about 7 years ago

Comparing the docker-registry.engineering.redhat.com with atomic-registry.usersys.redhat.com, the docker-registry is almost four hours behind current time. Could this lead to problems during authentication?

From what I understand, the atomic-registry is using GoogleAuthProvider while docker-registry is using LDAPPasswordIdentityProvider.

Actions
Copy link
 #3

Updated by ipanova@redhat.com about 7 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com
Actions
Copy link
 #4

Updated by ipanova@redhat.com about 7 years ago

I did some investigation, pulp sync works correctly with 2 out of 3 registries:
registry-1.docker.io
atomic-registry.usersys.redhat.com
docker-registry.engineering.redhat.com

Pulp tries to reach <registry>/v2 enpoint and after it receives 401 response, it parses the headers, which contain realm, scope, service and other needed parameters to fetch the token later. Then it goes to fetch the token with(if any) query params it processed from the 401 response headers. With fetched token from the registry it then tries to reach again the /v2/ endpoint.

The listed above registries work, beside the docker-registry.engineering.redhat.com.

I can confirm that pulp processes the received headers from the registry correctly as well as fetches the token.
I guess there is some issue on the registry side. Maybe it expects some additional query params at the fetch moment of the token. If it is the case, then the response headers from the registry lack some info. In any case further investigation needs to be done not on pulp side but on the registry side.

Actions
Copy link
 #5

Updated by ipanova@redhat.com about 7 years ago

Another point that confirms that the issue is on registry side is:

[ipanova@ina myprojects]$ curl -k -I -L -H 'Authorization: Bearer anonymous' https://docker-registry.engineering.redhat.com/v2/
HTTP/2.0 401
content-type:application/json; charset=utf-8
docker-distribution-api-version:registry/2.0
www-authenticate:Basic realm=openshift,error="access denied"
content-length:87
date:Wed, 22 Mar 2017 14:12:02 GMT

[ipanova@ina myprojects]$ curl -k -I -L -H 'Authorization: Bearer anonymous' -X GET https://docker-registry.engineering.redhat.com/v2/thomasmckay/freewill/tags/list
HTTP/2.0 200
content-type:application/json; charset=utf-8
docker-distribution-api-version:registry/2.0
content-length:46
date:Wed, 22 Mar 2017 14:45:07 GMT

I gives 401 auth error when reaching to the /v2 endpoint but allows to reach the /repo/tags/list endpoint

Actions
Copy link
 #6

Updated by ipanova@redhat.com about 7 years ago

ok ..so i have more results. There are more discrepancies in more projects.

Why 'docker pull' works':
Docker pull works because docker engine client code looks at the presence of the "Docker-Distribution-API-Version” and does not care about the response status code. ( [1] https://github.com/docker/distribution/blob/master/registry/client/auth/api_version.go#L26) So basically you can have even 500 response code, and Docker-Distribution-API-Version header present that would enable docker client to continue to work because he will say "v2 found!' what's funny is this behaviour contradicts their specs which say https://docs.docker.com/registry/spec/api/#api-version-check:

if 404 Not Found response status, or other unexpected status, is returned, the client should proceed with the assumption that the registry does not implement V2 of the API.

That means(based on spec) code should look at the status code and search for 404( docker client does not do that)
Docker client should look at the "Docker-Distribution-API-Version" header just in case 200 or 401 is returned( baes on spec)

Another discrepancy:
In the code:
https://github.com/docker/docker/blob/master/registry/auth.go#L286

// The version header indicates we're definitely
            // talking to a v2 registry. So don't allow future
// fallbacks to the v1 protocol.

Note the word definitely

And in specs they say

if a 200 OK response is returned, the registry implements the V2(.1) registry API and the client may proceed safely with other V2 operations.

Note the word safely

Basically they contradict what's written in specs and how actually the docker-client was implemented. That's why docker pull works.

What's wrong with docker-registry.engineering.redhat.com:

Because it does not behave correctly by reaching the /v2 endpoint , we cannot validate the token

If a 401 Unauthorized response is returned, the client should take action based on the contents of the “WWW-Authenticate” header and try the endpoint again. Depending on access control setup, the client may still have to authenticate against different resources, even if this check succeeds.

What's wrong in pulp:

I will update the code so it would look at the status code and not just at the presence of the header. The change will enable pulp to sync from docker-registry.engineering.redhat.com, even if token validation at /v2 did not pass

Actions
Copy link
 #7

Updated by ipanova@redhat.com about 7 years ago

  • Sprint/Milestone set to 36
Actions
Copy link
 #8

Updated by ipanova@redhat.com about 7 years ago

  • Status changed from ASSIGNED to POST
Actions
Copy link
 #10

Updated by mhrivnak about 7 years ago

  • Sprint/Milestone changed from 36 to 37
Actions
Copy link
 #11

Updated by ipanova@redhat.com about 7 years ago

  • Status changed from POST to MODIFIED
Actions
Copy link
 #12

Updated by pcreech about 7 years ago

  • Platform Release set to 2.13.0
  • Target Release - Docker set to 2.4.0
Actions
Copy link
 #13

Updated by pcreech about 7 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #14

Updated by Ichimonji10 almost 7 years ago

Verified with Pulp 2.13 beta on F24. Verification required editing /usr/lib/systemd/system/docker.service and inserting an --insecure-registry line:

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target
Wants=docker-storage-setup.service

[Service]
Type=notify
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
ExecStart=/usr/bin/docker-current daemon \
          --insecure-registry=docker-registry.engineering.redhat.com \
          --exec-opt native.cgroupdriver=systemd \
          $OPTIONS \
          $DOCKER_STORAGE_OPTIONS \
          $DOCKER_NETWORK_OPTIONS \
          $INSECURE_REGISTRY
ExecReload=/bin/kill -s HUP $MAINPID
TasksMax=infinity
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
Restart=on-abnormal

[Install]
WantedBy=multi-user.target
Actions
Copy link
 #15

Updated by pcreech almost 7 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions
Copy link
 #16

Updated by bmbouter about 6 years ago

  • Sprint set to Sprint 18
Actions
Copy link
 #17

Updated by bmbouter about 6 years ago

  • Sprint/Milestone deleted (37)
Actions
Copy link
 #18

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF