Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #2956

closed

As a user, I can sync from registries that use basic auth

Added by vrutkovs over 7 years ago. Updated over 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
bizhang
Start date:
Due date:
Estimated time:
Severity:
3. High
Version - Docker:
3.0.1
Platform Release:
2.14.1
Target Release - Docker:
OS:
Triaged:
Yes
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 25
Quarter:

Description

Versions:

pulp-server-2.14.0-0.3.rc.el7.noarch
pulp-puppet-admin-extensions-2.14.0-0.3.rc.el7.noarch
pulp-rpm-handlers-2.14.0-0.3.rc.el7.noarch
pulp-selinux-2.14.0-0.3.rc.el7.noarch
pulp-docker-plugins-3.0.0-0.3.rc.el7.noarch
pulp-rpm-plugins-2.14.0-0.3.rc.el7.noarch
pulp-admin-client-2.14.0-0.3.rc.el7.noarch
pulp-rpm-admin-extensions-2.14.0-0.3.rc.el7.noarch
pulp-consumer-client-2.14.0-0.3.rc.el7.noarch
pulp-puppet-handlers-2.14.0-0.3.rc.el7.noarch
pulp-rpm-consumer-extensions-2.14.0-0.3.rc.el7.noarch
pulp-rpm-yumplugins-2.14.0-0.3.rc.el7.noarch
pulp-puppet-plugins-2.14.0-0.3.rc.el7.noarch
pulp-docker-admin-extensions-3.0.0-0.3.rc.el7.noarch
pulp-agent-2.14.0-0.3.rc.el7.noarch
pulp-puppet-consumer-extensions-2.14.0-0.3.rc.el7.noarch

Commands:

$ pulp-admin docker repo create --repo-id redhat-rhel7-e2e-container-test-product-docker --feed https://pnt-rcm-distribution.web.qa.ext.phx1.redhat.com/ --upstream-name rhel7/e2e-container-test-product-docker --basicauth-user osbs --basicauth-pass craycray --verify-feed-ssl=false
Repository [redhat-rhel7-e2e-container-test-product-docker] successfully created

$ pulp-admin docker repo list --details --repo-id=redhat-rhel7-e2e-container-test-product-docker
+----------------------------------------------------------------------+
                          Docker Repositories
+----------------------------------------------------------------------+

Id:                   redhat-rhel7-e2e-container-test-product-docker
Display Name:         None
Description:          None
Content Unit Counts:  
Notes:                
Scratchpad:           
Importers:            
  Config:               
    Basic Auth Password: *****
    Basic Auth Username: osbs
    Feed:                https://pnt-rcm-distribution.web.qa.ext.phx1.redhat.com
                         /
    SSL Validation:      False
    Upstream Name:       rhel7/e2e-container-test-product-docker
  Id:                   docker_importer
  Importer Type Id:     docker_importer
  Last Override Config: 
  Last Sync:            None
  Last Updated:         2017-08-04T16:36:29Z
  Repo Id:              redhat-rhel7-e2e-container-test-product-docker
  Scratchpad:           None
Distributors:         
  Auto Publish:         True
  Config:               
  Distributor Type Id:  docker_distributor_web
  Id:                   docker_web_distributor_name_cli
  Last Override Config: 
  Last Publish:         None
  Last Updated:         2017-08-04T16:36:29Z
  Repo Id:              redhat-rhel7-e2e-container-test-product-docker
  Scratchpad:           
  Auto Publish:         False
  Config:               
  Distributor Type Id:  docker_distributor_export
  Id:                   docker_export_distributor_name_cli
  Last Override Config: 
  Last Publish:         None
  Last Updated:         2017-08-04T16:36:29Z
  Repo Id:              redhat-rhel7-e2e-container-test-product-docker
  Scratchpad:           

$ pulp-admin -vvv docker repo sync run --repo-id=redhat-rhel7-e2e-container-test-product-docker
+----------------------------------------------------------------------+
Synchronizing Repository [redhat-rhel7-e2e-container-test-product-docker]
+----------------------------------------------------------------------+

2017-08-04 18:39:30,702 - DEBUG - sending POST request to /pulp/api/v2/tasks/search/
2017-08-04 18:39:32,160 - INFO - POST request to /pulp/api/v2/tasks/search/ with parameters {"criteria": {"filters": {"state": {"$nin": ["finished", "error", "canceled", "skipped"]}, "tags": {"$all": ["pulp:repository:redhat-rhel7-e2e-container-test-product-docker", "pulp:action:sync"]}}}}
2017-08-04 18:39:32,160 - INFO - Response status : 200 

2017-08-04 18:39:32,161 - INFO - Response body :
 []

2017-08-04 18:39:32,161 - DEBUG - sending POST request to /pulp/api/v2/repositories/redhat-rhel7-e2e-container-test-product-docker/actions/sync/
2017-08-04 18:39:33,680 - INFO - POST request to /pulp/api/v2/repositories/redhat-rhel7-e2e-container-test-product-docker/actions/sync/ with parameters {"override_config": null}
2017-08-04 18:39:33,680 - INFO - Response status : 202 

2017-08-04 18:39:33,680 - INFO - Response body :
 {
  "spawned_tasks": [
    {
      "_href": "/pulp/api/v2/tasks/87aa974e-aa17-4a95-9b55-a9230b03c5ba/", 
      "task_id": "87aa974e-aa17-4a95-9b55-a9230b03c5ba"
    }
  ], 
  "result": null, 
  "error": null
}

This command may be exited via ctrl+c without affecting the request.

2017-08-04 18:39:34,682 - DEBUG - sending GET request to /pulp/api/v2/tasks/87aa974e-aa17-4a95-9b55-a9230b03c5ba/
2017-08-04 18:39:36,125 - INFO - GET request to /pulp/api/v2/tasks/87aa974e-aa17-4a95-9b55-a9230b03c5ba/ with parameters None
2017-08-04 18:39:36,125 - INFO - Response status : 200 

2017-08-04 18:39:36,126 - INFO - Response body :
 {
  "exception": null, 
  "task_type": "pulp.server.managers.repo.sync.sync", 
  "_href": "/pulp/api/v2/tasks/87aa974e-aa17-4a95-9b55-a9230b03c5ba/", 
  "task_id": "87aa974e-aa17-4a95-9b55-a9230b03c5ba", 
  "tags": [
    "pulp:repository:redhat-rhel7-e2e-container-test-product-docker", 
    "pulp:action:sync"
  ], 
  "finish_time": "2017-08-04T16:39:33Z", 
  "_ns": "task_status", 
  "start_time": "2017-08-04T16:39:33Z", 
  "traceback": "Traceback (most recent call last):\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 240, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 522, in __call__\n    return super(Task, self).__call__(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 107, in __call__\n    return super(PulpTask, self).__call__(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 438, in __protected_call__\n    return self.run(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 769, in sync\n    sync_report = sync_repo(transfer_repo, conduit, call_config)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 725, in wrap_f\n    return f(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py\", line 83, in sync_repo\n    self.sync_step = sync.SyncStep(repo=repo, conduit=sync_conduit, config=config)\n  File \"/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py\", line 89, in __init__\n    raise PulpCodedException(error_code=error_codes.DKR1008, registry=url)\nPulpCodedException: Could not find registry API at https://pnt-rcm-distribution.web.qa.ext.phx1.redhat.com/\n", 
  "spawned_tasks": [], 
  "progress_report": {}, 
  "queue": "reserved_resource_worker-1@osbs-pulp-stage.host.prod.eng.rdu2.redhat.com.dq", 
  "state": "error", 
  "worker_name": "reserved_resource_worker-1@osbs-pulp-stage.host.prod.eng.rdu2.redhat.com", 
  "result": null, 
  "error": {
    "code": "DKR1008", 
    "data": {
      "registry": "https://pnt-rcm-distribution.web.qa.ext.phx1.redhat.com/"
    }, 
    "description": "Could not find registry API at https://pnt-rcm-distribution.web.qa.ext.phx1.redhat.com/", 
    "sub_errors": []
  }, 
  "_id": {
    "$oid": "5984a345bdcd50fac25a8bcd"
  }, 
  "id": "5984a345bdcd50fac25a8bcd"
}

Task Failed

Could not find registry API at
https://pnt-rcm-distribution.web.qa.ext.phx1.redhat.com/

Server logs say the download failed:

Aug 04 16:32:08 osbs-pulp-stage.host.prod.eng.rdu2.redhat.com pulp[2272]: nectar.downloaders.threaded:INFO: [f7e60ab0] Download failed: Download of https://pnt-rcm-distribution.web.qa.ext.phx1.redhat.com/v2/ failed with code 401: Unauthorized
Aug 04 16:32:08 osbs-pulp-stage.host.prod.eng.rdu2.redhat.com pulp[2272]: pulp_docker.plugins.registry:DEBUG: [f7e60ab0] Download unauthorized, attempting to retrieve a token.

Curl with basicauth works:

$ curl -klvs -u osbs:craycray https://pnt-rcm-distribution.web.qa.ext.phx1.redhat.com/v2/      
*   Trying 10.27.119.22...
* TCP_NODELAY set
* Connected to pnt-rcm-distribution.web.qa.ext.phx1.redhat.com (10.27.119.22) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* ALPN, server accepted to use http/1.1
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*     subject: E=jmainguy@redhat.com,CN=distribution,OU=IT-PNT,O=Redhat,L=Raleigh,ST=North Carolina,C=US
*     start date: Jun 15 17:53:05 2016 GMT
*     expire date: Jun 13 17:53:05 2026 GMT
*     common name: distribution
*     issuer: E=jmainguy@redhat.com,CN=distribution,OU=IT-PNT,O=Redhat,L=Raleigh,ST=North Carolina,C=US
* Server auth using Basic with user 'osbs'
> GET /v2/ HTTP/1.1
> Host: pnt-rcm-distribution.web.qa.ext.phx1.redhat.com
> Authorization: Basic b3NiczpjcmF5Y3JheQ==
> User-Agent: curl/7.53.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Length: 2
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< Date: Fri, 04 Aug 2017 16:43:07 GMT
< 
* Connection #0 to host pnt-rcm-distribution.web.qa.ext.phx1.redhat.com left intact
Actions
Copy link
 #1

Updated by vrutkovs over 7 years ago

After some investigation with Ina it seems its caused by https://github.com/pulp/pulp_docker/commit/2a99e19f197902c647485679ec6d590cc219ac90.

Pulp 2.8 instance works fine for OSBS team, packages versions:

python-kombu-3.0.24-10.pulp.el7.noarch
pulp-server-2.8.0.2-1.el7sat.noarch
python-pulp-repoauth-2.8.0.2-1.el7sat.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
pulp-admin-client-2.8.0.2-1.el7sat.noarch
python-pulp-common-2.8.0.2-1.el7sat.noarch
python-pulp-bindings-2.8.0.2-1.el7sat.noarch
pulp-docker-admin-extensions-2.0.0.2-1.el7sat.noarch
python-pulp-agent-lib-2.8.0.2-1.el7sat.noarch
python-pulp-client-lib-2.8.0.2-1.el7sat.noarch
pulp-selinux-2.8.0.2-1.el7sat.noarch
pulp-docker-plugins-2.0.0.2-1.el7sat.noarch
python-pulp-docker-common-2.0.0.2-1.el7sat.noarch
Actions
Copy link
 #2

Updated by amacdona@redhat.com over 7 years ago

Docker registries use token authentication, and the tokens are retrieved using basic auth.

https://docs.docker.com/registry/spec/auth/token/

Following that spec, Pulp Docker uses basic auth only on token requests.

Actions
Copy link
 #3

Updated by amacdona@redhat.com over 7 years ago

To be more explicit, curl with basic auth to the /v2/ endpoint should not work according to the docker token auth specification.

Actions
Copy link
 #4

Updated by twaugh over 7 years ago

Docker registries can use token authentication, but they do not all use it. For example, the configuration key "htpasswd" allows them to use HTTP Basic auth instead.

https://docs.docker.com/registry/configuration/#htpasswd

Actions
Copy link
 #5

Updated by amacdona@redhat.com over 7 years ago

@twaugh, thanks for the link.

After some discussion with ipanova, we think that this issue should become a story to support basic auth. Basic auth did work in 2.8, but it was not the correct flow, so I would say that this is not a regression.

Instead, we can modify how we respond to the 401, which has a "WWW-Authenticate" header that indicates the auth scheme. We already parse this header to request a bearer token here: https://github.com/pulp/pulp_docker/blob/master/plugins/pulp_docker/plugins/token_util.py#L30

If the scheme is "basic" we should update the request to include basic auth credentials and try again.

Actions
Copy link
 #6

Updated by twaugh over 7 years ago

That definitely sounds reasonable.

Actions
Copy link
 #7

Updated by ttereshc over 7 years ago

  • Tracker changed from Issue to Story
  • % Done set to 0

@ipanova and @asmacdo will groom it and mark as sprint candidate

Actions
Copy link
 #8

Updated by amacdona@redhat.com over 7 years ago

  • Subject changed from Pulp docker ignores basic auth credentials to As a user, I can sync from registries that use basic auth
Actions
Copy link
 #9

Updated by ipanova@redhat.com over 7 years ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes
Actions
Copy link
 #10

Updated by ipanova@redhat.com over 7 years ago

groomed based on comment #5
Just a small note so it does not get lost - during the first request we are not going to send any kind of credentials.

Actions
Copy link
 #11

Updated by mhrivnak over 7 years ago

  • Tracker changed from Story to Issue
  • Sprint/Milestone set to 43
  • Severity set to 2. Medium
  • Triaged set to Yes
Actions
Copy link
 #12

Updated by mhrivnak over 7 years ago

  • Priority changed from Normal to High
Actions
Copy link
 #13

Updated by mhrivnak over 7 years ago

  • Severity changed from 2. Medium to 3. High
Actions
Copy link
 #14

Updated by bizhang over 7 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bizhang
Actions
Copy link
 #15

Updated by bizhang over 7 years ago

  • Status changed from ASSIGNED to POST
Actions
Copy link
 #16

Updated by bizhang over 7 years ago

@vrutkovs
While fixing this I found that the docker repository you mentioned [0] had tags:null. This was an issue already fixed in docker [1] and you should open a bug with whomever owns the registry. I've added a defensive patch to pulp as a part of the #2956 fix, so it shouldn't blow up pulp.

[0] https://pnt-rcm-distribution.web.qa.ext.phx1.redhat.com/v2/rhel7/e2e-container-test-product-docker/tags/list
[1] https://github.com/docker/distribution/issues/1186

Actions
Copy link
 #17

Updated by vrutkovs over 7 years ago

Thanks! Yes, that repo is probably empty, but its easy to fill it in and test the fix

Added by werwty over 7 years ago

Revision 5ed1c5c0 | View on GitHub

Use basic auth to sync when scheme is "basic"

closes #2956 https://pulp.plan.io/issues/2956

Added by werwty over 7 years ago

Revision 5ed1c5c0 | View on GitHub

Use basic auth to sync when scheme is "basic"

closes #2956 https://pulp.plan.io/issues/2956

Added by werwty over 7 years ago

Revision 5ed1c5c0 | View on GitHub

Use basic auth to sync when scheme is "basic"

closes #2956 https://pulp.plan.io/issues/2956

Added by werwty over 7 years ago

Revision 5ed1c5c0 | View on GitHub

Use basic auth to sync when scheme is "basic"

closes #2956 https://pulp.plan.io/issues/2956

Actions
Copy link
 #18

Updated by werwty over 7 years ago

  • Status changed from POST to MODIFIED
Actions
Copy link
 #19

Updated by jortel@redhat.com over 7 years ago

  • Sprint/Milestone changed from 43 to 44
Actions
Copy link
 #20

Updated by pcreech over 7 years ago

  • Version - Docker set to 3.0.1
  • Platform Release set to 2.14.1
Actions
Copy link
 #22

Updated by pcreech about 7 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions
Copy link
 #23

Updated by bmbouter almost 7 years ago

  • Sprint set to Sprint 25
Actions
Copy link
 #24

Updated by bmbouter almost 7 years ago

  • Sprint/Milestone deleted (44)
Actions
Copy link
 #27

Updated by bmbouter over 5 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF