Issue #1893
closedpulp_docker does not support Docker < 1.10 with manifests that were published from Docker >= 1.10
Added by rbarlow over 8 years ago. Updated over 5 years ago.
Description
Tim Waugh reported this issue, so I will use the text he wrote up here:
The V2 manifest that Docker 1.10 pushes to the Distribution registry, and which will get synced into Pulp, is Schema 2, which Docker 1.9 does not understand. When Docker engine 1.9 pulls from a Distribution registry, Distribution sees that it is talking to a client that does not understand Schema 2, and converts the manifest to Schema 1 before sending its response.
This means there are effectively three things:
V1, supported by all Docker engine versions
V2 (Schema 1), supported by 1.6 up to 1.9
V2 (Schema 2), only supported by 1.10
In order for Pulp to be able to satisfy both 1.6 – 1.9 clients and 1.10 clients, it will have to learn to convert Schema 2 manifests to Schema 1 manifests.
https://github.com/docker/distribution/blob/master/docs/compatibility.md#manifest-push-with-docker-110
https://gist.github.com/aaronlehmann/b42a2eaf633fc949f93b#interoperability-with-old-image-formats
Related issues
Updated by rbarlow over 8 years ago
IMO, this issue will be more difficult than it may seem to address. Pulp is not currently involved in serving out the Manifests to clients, as they are just files on disk that httpd is serving when clients make requests. Thus, the current architecture does not allow for Pulp to do things like detecting which client version is communicating with Pulp and directing them to the correct file.
Additionally, Pulp has no awareness of the v2 schema 2 Manifest format, so there will be some engineering effort there to determine how to work with that as well.
A short-term workaround might be to make sure that Pulp identifies itself as a Docker 1.9 client so that it always received v2 schema 1 Manifests, which should work with all Docker v2 versions. It's possible that is already what happens, but if not it's likely a header we can add to our requests during sync. I suggest that we investigate this option and get it working before trying to address the complications I mentioned above.
Updated by mhrivnak over 8 years ago
rbarlow wrote:
IMO, this issue will be more difficult than it may seem to address. Pulp is not currently involved in serving out the Manifests to clients, as they are just files on disk that httpd is serving when clients make requests. Thus, the current architecture does not allow for Pulp to do things like detecting which client version is communicating with Pulp and directing them to the correct file.
Additionally, Pulp has no awareness of the v2 schema 2 Manifest format, so there will be some engineering effort there to determine how to work with that as well.
A short-term workaround might be to make sure that Pulp identifies itself as a Docker 1.9 client so that it always received v2 schema 1 Manifests, which should work with all Docker v2 versions. It's possible that is already what happens, but if not it's likely a header we can add to our requests during sync. I suggest that we investigate this option and get it working before trying to address the complications I mentioned above.
+1 to that. Transforming data is a business pulp tries to stay out of.
How does this work with the upstream registry? Does it detect the client version starting all the way back at the tag listing request, and give tag values (which identify manifest hashes) that correspond to schema 1 or 2 depending on the client? Does upstream convert between the schemas on the fly?
I like the idea of pulp pretending to be a 1.9 client and hoping to get compatible content. It's simple, and hopefully effective!
Updated by mhrivnak over 8 years ago
It sounds like RCM is going to do some testing with various clients to see what the impact is. We are deferring triage until we can get more information along those lines, to know more concretely how this impacts users.
Updated by twaugh about 8 years ago
The OSBS development team has not had time to do any testing for this.
Updated by rbarlow about 8 years ago
twaugh wrote:
The OSBS development team has not had time to do any testing for this.
At this point I think the best course of action is for the Pulp team to evaluate the severity of this issue.
Updated by dkliban@redhat.com about 8 years ago
- Priority changed from Normal to High
- Triaged changed from No to Yes
This issue needs to be investigated soon. Mark it as untriaged after the investigation.
Updated by twaugh about 8 years ago
If Randy's work-around has already been implemented, make sure to disable it prior to testing. :-)
Updated by acarter@redhat.com about 8 years ago
Just noting that v2 support must land in production before July 20 to support the Satellite 6.2 release. We do not plan to go live with 1.10 support but will move to 1.10 soon after (likely early August).
Updated by dkliban@redhat.com about 8 years ago
Docker 1.11 is now out. So testing needs to be performed using Docker 1.9 and 1.11.
The testing involve the following:
1. Install docker engine on a machine, version 1.11
2. Build an image using Docker 1.11. The instructions are here https://docs.docker.com/engine/userguide/containers/dockerimages/
3. Deploy a docker registry running in a container (https://hub.docker.com/_/registry/)
4. Push that image to your local registry
5. Using curl and docker API, request the manifest for the new image in dockerhub. Make sure that schema 2 is being used in the response.
6. Once schema version has been confirmed, sync the image from your registry into pulp
7. publish that repo and have it served by crane
8. install docker 1.9 on another machine
9. perform docker pull from crane using docker 1.9
Then perform the same test in reverse. Create image using Docker 1.9. Then sync it into Pulp. Then do a docker pull using Docker 1.11.
Updated by ipanova@redhat.com about 8 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to ipanova@redhat.com
Updated by ipanova@redhat.com about 8 years ago
Some facts about docker architecture:
docker client can talk to the docker registry in 2 ways
1. directly via API
2 talk through docker-daemon
If you are making a request to API directly with curl and want to fetch manifest with schemav2 you need to specify that in headers - that you accept schemav2 [0], otherwise schemav1 will be returned [1]( that is done because transition needs to be backwards-compatible)
[0] https://github.com/docker/distribution/blob/master/docs/spec/manifest-v2-2.md#media-types
[1] https://github.com/docker/distribution/blob/master/registry/handlers/images.go#L104
]$ curl -L -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET 'http://localhost:5000/v2/ipanova/base/manifests/latest'
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1078,
"digest": "sha256:3ccdc139f077afebfef63dc26a0a2c4eb6f0ff0864f94ebf0069e3665c19ae6f"
},
"layers": []
]$ curl -L -X GET 'http://localhost:5000/v2/ipanova/base/manifests/latest'
{
"schemaVersion": 1,
"name": "ipanova/base",
"tag": "latest",
"architecture": "amd64",
"fsLayers": [
{
"blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
}
],
"history": [
{
"v1Compatibility": "{\"architecture\":\"amd64\",\"config\":{\"Hostname\":\"01bbbc4c98fc\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"x=a\"],\"Cmd\":null,\"Image\":\"\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":null},\"container\":\"01bbbc4c98fcc15aceb958db6f1a472858d8fbdd2aec4632b901ced5fe036f28\",\"container_config\":{\"Hostname\":\"01bbbc4c98fc\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"x=a\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) ENV x=a\"],\"Image\":\"\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":null},\"created\":\"2016-06-13T11:41:24.936334311Z\",\"docker_version\":\"1.11.2\",\"id\":\"a3446e94ae61426abfb23e2263b19c992fc470957ae529cb6d75d3c48fe34590\",\"os\":\"linux\",\"throwaway\":true}"
}
],
"signatures": [
{
"header": {
"jwk": {
"crv": "P-256",
"kid": "4WBO:SDRD:4VMB:TBSP:CQAD:5EXK:ZODJ:B75I:AGIH:M4EN:ZNG2:TBGZ",
"kty": "EC",
"x": "-fg9H9Io3hFCYjsD4zq_AKIyNm2kmmWKN52G5BpZxP8",
"y": "8HdSl8EF0TEmJ8sZP6vzenOgimRgBBhgEqtnbCkykTQ"
},
"alg": "ES256"
},
"signature": "-Rpyg_bx-EAUuv1tQIFDwAcEPXay0v9q14eQ4_Izt1I2iP1bA1MnL2XcVglt_9UdD12T85TsXmC8JI6mC4vmsQ",
"protected": "eyJmb3JtYXRMZW5ndGgiOjE0NTUsImZvcm1hdFRhaWwiOiJDbjAiLCJ0aW1lIjoiMjAxNi0wNi0xM1QxNjoyODozNVoifQ"
If the client is talking to the registry via docker-daemon, docker daemon>=1.10 sets headers to receive the scnemav2.
When we sync docker content into pulp, and in the feed provide the registry to sync from, we talk to the registry directly via api, that means that we do not set any headers, and schemav1 version is returned in any case [2], plus we pull manifest by tag, so we are out of the issue that docker compatibility it is having right now. [3]
[2] https://github.com/pulp/pulp_docker/blob/master/plugins/pulp_docker/plugins/registry.py#L277
[3] https://docs.docker.com/registry/compatibility/
When the image is built by docker 1.9, then pushed to the registry, then synced into pulp and served by crane, the manifest digest does not change, i.e digest from docker push into registry coincides with the manifest digest that is in crane.
When the image is built by docker 1.11, then pushed to the registry, then synced into pulp and served by crane, the manifest digest changes, i.e digest from docker push into registry is different from the manifest digest that is in crane. Manifest digest changes because of the fact that in the registry schemav2 is registered, but when we sync content into pulp we pull schemav1.
So that's why when we do docker pull from crane by manifest digest (noted in the 'push' step to the registry) we fail.
Outcome of the research
If we want to docker pull by digest, we need to use the digest that is specified in crane and not in the registry.
Updated by ipanova@redhat.com about 8 years ago
- Status changed from ASSIGNED to POST
Added by ipanova@redhat.com about 8 years ago
Updated by ipanova@redhat.com about 8 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset crane:80699c2c8a8f7fb674658c6f66ea99701cdaf903.
Updated by ipanova@redhat.com about 8 years ago
- Platform Release set to 2.8.6
- Target Release - Docker set to 2.0.2
Updated by semyers about 8 years ago
- Project changed from Docker Support to Crane
- Platform Release changed from 2.8.6 to 2.8.7
This was addressed with a fix to Crane, so I've moved it over to that project. It didn't make the cut for 2.8.6, but will be in 2.8.7.
Updated by semyers about 8 years ago
- Related to Task #2083: Issues common to 2.9.1 and 2.8 stream added
Updated by pthomas@redhat.com about 8 years ago
- Status changed from 5 to 6
Updated by semyers almost 8 years ago
- Status changed from 6 to CLOSED - CURRENTRELEASE
Manifest digest changes when there is schemav2-v1 conversion.
closes #1893 https://pulp.plan.io/issues/1893