sandeepc1988 wrote:
sandeepc1988 wrote:
- The remote API call is invoked to configure the AWS ECR container registry with username and password.
- AWS token expiration is 12 hrs.
- Once the token is expired, When the docker pull request is invoked on the pulp server, the remote returns a 403 forbidden error.
- Now a Remote API call is patched with a new valid AWS token, the docker pull request is expected to pull the image from the remote.
- But pulp container fails to take the updated token and fails to pull the image.
Observation:
The pulp container plugin is not handling 403 forbidden errors.
Update:
The Basic auth credentials is cached here: https://github.com/pulp/pulp_container/blob/af01b442daa7a47b2ffc4674eafbc7a0331e2fa0/pulp_container/app/downloaders.py#L56
with the expired token AWS registry always returns 403.
Token auth has been implemented following these specs and the code relies on 401 auth response https://docs.docker.com/registry/spec/auth/token/. Whether the token is invalid because it has expired or has wrong format, or has insufficient scope a 401 is expected.
www-authenticate header tells where to fetch a new token together with the scope details, if a 403 is returned that header does not contain any of the information and client would not know where to retrieve the token
$ curl -i -H "Authorization:Bearer $ACCESS_TOKEN" 'https://registry-1.docker.io/v2/_catalog' -L
HTTP/1.1 401 Unauthorized
content-type: application/json
docker-distribution-api-version: registry/2.0
www-authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="registry:catalog:*",error="insufficient_scope"
date: Tue, 22 Jun 2021 11:46:28 GMT
content-length: 145
strict-transport-security: max-age=31536000
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"registry","Class":"","Name":"catalog","Action":"*"}]}]}
$ curl -i -H "Authorization:Bearer $ACCESS_TOKEN" 'https://registry-1.docker.io/v2/library/busybox/manifests/latest' -L
HTTP/1.1 401 Unauthorized
content-type: application/json
docker-distribution-api-version: registry/2.0
www-authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/busybox:pull",error="invalid_token"
date: Tue, 22 Jun 2021 11:50:24 GMT
content-length: 158
strict-transport-security: max-age=31536000
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"library/busybox","Action":"pull"}]}]}
The new Token is not read from remote until 401 error occurs.
https://github.com/pulp/pulp_container/blob/af01b442daa7a47b2ffc4674eafbc7a0331e2fa0/pulp_container/app/downloaders.py#L67
Clearing the cache for error 403 would result in 401 responses on retry.
This forces code to pick the new token from the remote. We Need a better way of handling basic auth credentials rotation/update.
.