Crane

IMO, this issue will be more difficult than it may seem to address. Pulp is not currently involved in serving out the Manifests to clients, as they are just files on disk that httpd is serving when clients make requests. Thus, the current architecture does not allow for Pulp to do things like detecting which client version is communicating with Pulp and directing them to the correct file.

Additionally, Pulp has no awareness of the v2 schema 2 Manifest format, so there will be some engineering effort there to determine how to work with that as well.

A short-term workaround might be to make sure that Pulp identifies itself as a Docker 1.9 client so that it always received v2 schema 1 Manifests, which should work with all Docker v2 versions. It's possible that is already what happens, but if not it's likely a header we can add to our requests during sync. I suggest that we investigate this option and get it working before trying to address the complications I mentioned above.

rbarlow wrote:

IMO, this issue will be more difficult than it may seem to address. Pulp is not currently involved in serving out the Manifests to clients, as they are just files on disk that httpd is serving when clients make requests. Thus, the current architecture does not allow for Pulp to do things like detecting which client version is communicating with Pulp and directing them to the correct file.

Additionally, Pulp has no awareness of the v2 schema 2 Manifest format, so there will be some engineering effort there to determine how to work with that as well.

A short-term workaround might be to make sure that Pulp identifies itself as a Docker 1.9 client so that it always received v2 schema 1 Manifests, which should work with all Docker v2 versions. It's possible that is already what happens, but if not it's likely a header we can add to our requests during sync. I suggest that we investigate this option and get it working before trying to address the complications I mentioned above.

+1 to that. Transforming data is a business pulp tries to stay out of.

How does this work with the upstream registry? Does it detect the client version starting all the way back at the tag listing request, and give tag values (which identify manifest hashes) that correspond to schema 1 or 2 depending on the client? Does upstream convert between the schemas on the fly?

I like the idea of pulp pretending to be a 1.9 client and hoping to get compatible content. It's simple, and hopefully effective!

It sounds like RCM is going to do some testing with various clients to see what the impact is. We are deferring triage until we can get more information along those lines, to know more concretely how this impacts users.

The OSBS development team has not had time to do any testing for this.

twaugh wrote:

The OSBS development team has not had time to do any testing for this.

At this point I think the best course of action is for the Pulp team to evaluate the severity of this issue.

If Randy's work-around has already been implemented, make sure to disable it prior to testing. :-)

Just noting that v2 support must land in production before July 20 to support the Satellite 6.2 release. We do not plan to go live with 1.10 support but will move to 1.10 soon after (likely early August).

Docker 1.11 is now out. So testing needs to be performed using Docker 1.9 and 1.11.

The testing involve the following:

1. Install docker engine on a machine, version 1.11
2. Build an image using Docker 1.11. The instructions are here https://docs.docker.com/engine/userguide/containers/dockerimages/
3. Deploy a docker registry running in a container (https://hub.docker.com/_/registry/)
4. Push that image to your local registry
5. Using curl and docker API, request the manifest for the new image in dockerhub. Make sure that schema 2 is being used in the response.
6. Once schema version has been confirmed, sync the image from your registry into pulp
7. publish that repo and have it served by crane
8. install docker 1.9 on another machine
9. perform docker pull from crane using docker 1.9

Then perform the same test in reverse. Create image using Docker 1.9. Then sync it into Pulp. Then do a docker pull using Docker 1.11.

Some facts about docker architecture:
docker client can talk to the docker registry in 2 ways

1. directly via API
2 talk through docker-daemon

If you are making a request to API directly with curl and want to fetch manifest with schemav2 you need to specify that in headers - that you accept schemav2 [0], otherwise schemav1 will be returned [1]( that is done because transition needs to be backwards-compatible)

[0] https://github.com/docker/distribution/blob/master/docs/spec/manifest-v2-2.md#media-types
[1] https://github.com/docker/distribution/blob/master/registry/handlers/images.go#L104

]$ curl -L -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET 'http://localhost:5000/v2/ipanova/base/manifests/latest'
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1078,
      "digest": "sha256:3ccdc139f077afebfef63dc26a0a2c4eb6f0ff0864f94ebf0069e3665c19ae6f"
   },
   "layers": []

]$ curl -L -X GET 'http://localhost:5000/v2/ipanova/base/manifests/latest'
{
   "schemaVersion": 1,
   "name": "ipanova/base",
   "tag": "latest",
   "architecture": "amd64",
   "fsLayers": [
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      }
   ],
   "history": [
      {
         "v1Compatibility": "{\"architecture\":\"amd64\",\"config\":{\"Hostname\":\"01bbbc4c98fc\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"x=a\"],\"Cmd\":null,\"Image\":\"\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":null},\"container\":\"01bbbc4c98fcc15aceb958db6f1a472858d8fbdd2aec4632b901ced5fe036f28\",\"container_config\":{\"Hostname\":\"01bbbc4c98fc\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"x=a\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) ENV x=a\"],\"Image\":\"\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":null},\"created\":\"2016-06-13T11:41:24.936334311Z\",\"docker_version\":\"1.11.2\",\"id\":\"a3446e94ae61426abfb23e2263b19c992fc470957ae529cb6d75d3c48fe34590\",\"os\":\"linux\",\"throwaway\":true}"
      }
   ],
   "signatures": [
      {
         "header": {
            "jwk": {
               "crv": "P-256",
               "kid": "4WBO:SDRD:4VMB:TBSP:CQAD:5EXK:ZODJ:B75I:AGIH:M4EN:ZNG2:TBGZ",
               "kty": "EC",
               "x": "-fg9H9Io3hFCYjsD4zq_AKIyNm2kmmWKN52G5BpZxP8",
               "y": "8HdSl8EF0TEmJ8sZP6vzenOgimRgBBhgEqtnbCkykTQ"
            },
            "alg": "ES256"
         },
         "signature": "-Rpyg_bx-EAUuv1tQIFDwAcEPXay0v9q14eQ4_Izt1I2iP1bA1MnL2XcVglt_9UdD12T85TsXmC8JI6mC4vmsQ",
         "protected": "eyJmb3JtYXRMZW5ndGgiOjE0NTUsImZvcm1hdFRhaWwiOiJDbjAiLCJ0aW1lIjoiMjAxNi0wNi0xM1QxNjoyODozNVoifQ"

If the client is talking to the registry via docker-daemon, docker daemon>=1.10 sets headers to receive the scnemav2.

When we sync docker content into pulp, and in the feed provide the registry to sync from, we talk to the registry directly via api, that means that we do not set any headers, and schemav1 version is returned in any case [2], plus we pull manifest by tag, so we are out of the issue that docker compatibility it is having right now. [3]

[2] https://github.com/pulp/pulp_docker/blob/master/plugins/pulp_docker/plugins/registry.py#L277
[3] https://docs.docker.com/registry/compatibility/

When the image is built by docker 1.9, then pushed to the registry, then synced into pulp and served by crane, the manifest digest does not change, i.e digest from docker push into registry coincides with the manifest digest that is in crane.

When the image is built by docker 1.11, then pushed to the registry, then synced into pulp and served by crane, the manifest digest changes, i.e digest from docker push into registry is different from the manifest digest that is in crane. Manifest digest changes because of the fact that in the registry schemav2 is registered, but when we sync content into pulp we pull schemav1.

So that's why when we do docker pull from crane by manifest digest (noted in the 'push' step to the registry) we fail.

Outcome of the research

If we want to docker pull by digest, we need to use the digest that is specified in crane and not in the registry.