As a user, I can sync v2 schema manifests
As a result of this story, Pulp Docker will support both schema versions, 1 and 2 for sync, publish and copy.
To support both schema 1 and 2 we need to:
1. during sync always make 2 requests to the registry and in each request set properly the headers.
2. as a result of 2 requests we will have to save to the DB manifest with both schemas.
3. change Tag model and add schema_version field and include it into the unit_key( having 2 instances of tags will facilitate copy operation)
4. make sure copy operation works with above changes related to Tag model.
5. write a migration for Tag model change, also write a migration to drop old index
6. in manifest schema 2 there is a config array that contains a config blob, which is treated as regular layer. In order to have a runable docker image config layer should also be present. That means that we would need to download it and store in the DB( as a regular blob layer). That means that most likely we would need to change Manifest model and add a config layer blob field to it. No migration needs to be written for this field
7. change the schema of the crane metadata file to have two manifest lists: one for schema 1, and one for schema 2
8. change publish to create the new crane metadata file, also change publish directory structure for manifests
this work will not enable support for manifest list( aka fat manifest)
#2 Updated by twaugh over 4 years ago
Discussed this with firstname.lastname@example.org today and I think this came from a misunderstanding.
However, thinking around the issue today I realised something else relating to it.
Once we update our container image builder to docker 1.10, we will be pushing images to a docker-distribution instance and these images will have V2 schema 2 manifests. Our build system notes the manifest digest reported by docker after the push, then tells Pulp to sync from docker-distribution.
However, the digest noted by the build system and stored in Koji for the build will be different to the one available in Crane: after 'docker push' the digest is for a schema 2 manifest, but after 'sync' the manifest digest Crane will know will be for a schema 1 manifest.
To work around this we will need to query the docker-distribution instance for the schema 1 manifest and note the Docker-Content-Digest header value.
Is support for V2 schema 2 on the roadmap for pulp_docker/crane? I have an idea about how it could work.
#10 Updated by vrutkovs about 4 years ago
Discussed it with Amanda: currently we need just schema 2 (as we're planning to migrate to Docker 1.10 soon).
Note that multiarch implementation is also planned (thus fat manifest is needed as well), but it first has to be discussed with Content Delivery team
#11 Updated by mhrivnak about 4 years ago
To support schema 1 and 2, we would need to:
- change the manifest DB model to accomodate schema 2
- retrieve both schema 1 and 2 during sync and save both
- change the tag model to store both schema 1 and schema 2 manifest digests on the same instance, or add a schema field as part of the unit key
- possibly change the tag copy logic to make sure it's replacing tag models correctly now that there are two digests to worry about
- change the schema of the crane metadata file to have two manifest lists: one for schema 1, and one for schema 2
- change publish to create the new crane metadata file
- change crane to look for a optional header from the client and use that to decide whether to return a schema 1 or schema 2 manifest
@ipanova looked up that docker version 1.9 and older only do schema 1. We would consider just dropping schema 1 support, but that probably would not be acceptable at this time to only support docker 1.10+.
#12 Updated by mhrivnak about 4 years ago
#18 Updated by email@example.com about 4 years ago
- Checklist item add necessary code to enable schema 2 support added
- Checklist item write necessary migrations ( 2 migrations) added
- Checklist item add docs added
- Checklist item add release notes added
- Checklist item check that other operations like copy and publish are not broken added
#30 Updated by firstname.lastname@example.org over 3 years ago
N.B docker rsync distributor needs to be modified as well
Docker rsync dist publish path was changed because of publish directory structure change in general
We solved this by adding to apache config to set proper response headers based on directory match
#34 Updated by Ichimonji10 over 3 years ago
#36 Updated by Ichimonji10 over 3 years ago
This issue needs additional verification. See: https://github.com/PulpQE/pulp-smash/issues/555#issuecomment-297168241
#37 Updated by Ichimonji10 over 3 years ago
Re-verified. See: https://github.com/PulpQE/pulp-smash/issues/555#issuecomment-297447723
Please register to edit this issue