Project

Profile

Help

Story #2099

closed

As a user, I can sync v2 schema manifests

Added by twaugh over 7 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Start date:
Due date:
% Done:

100%

Estimated time:
(Total: 0:00 h)
Platform Release:
2.13.0
Target Release - Docker:
2.4.0
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 16
Quarter:

Description

As a result of this story, Pulp Docker will support both schema versions, 1 and 2 for sync, publish and copy.

To support both schema 1 and 2 we need to:
1. during sync always make 2 requests to the registry and in each request set properly the headers.
2. as a result of 2 requests we will have to save to the DB manifest with both schemas.
3. change Tag model and add schema_version field and include it into the unit_key( having 2 instances of tags will facilitate copy operation)
4. make sure copy operation works with above changes related to Tag model.
5. write a migration for Tag model change, also write a migration to drop old index
6. in manifest schema 2 there is a config array that contains a config blob, which is treated as regular layer. In order to have a runable docker image config layer should also be present. That means that we would need to download it and store in the DB( as a regular blob layer). That means that most likely we would need to change Manifest model and add a config layer blob field to it. No migration needs to be written for this field
7. change the schema of the crane metadata file to have two manifest lists: one for schema 1, and one for schema 2
8. change publish to create the new crane metadata file, also change publish directory structure for manifests

Note:
this work will not enable support for manifest list( aka fat manifest)


Sub-issues 1 (0 open1 closed)

Crane - Story #2336: Update crane to serve manifest schema 2CLOSED - CURRENTRELEASEipanova@redhat.com

Actions

Related issues

Blocks Crane - Story #2102: Easier way to configure Crane to serve both V1 and V2CLOSED - WONTFIX

Actions
Blocks Docker Support - Story #2384: As a user I can sync manifest listsCLOSED - CURRENTRELEASEipanova@redhat.com

Actions
Actions #1

Updated by mhrivnak over 7 years ago

Can you elaborate on the openshift need for schema 2? After investigating schema 2, we determined in consultation with @vbatts that all docker-related tooling should still be compatible with schema 1, and there should not be a need to support both.

Actions #2

Updated by twaugh over 7 years ago

Discussed this with today and I think this came from a misunderstanding.

However, thinking around the issue today I realised something else relating to it.

Once we update our container image builder to docker 1.10, we will be pushing images to a docker-distribution instance and these images will have V2 schema 2 manifests. Our build system notes the manifest digest reported by docker after the push, then tells Pulp to sync from docker-distribution.

However, the digest noted by the build system and stored in Koji for the build will be different to the one available in Crane: after 'docker push' the digest is for a schema 2 manifest, but after 'sync' the manifest digest Crane will know will be for a schema 1 manifest.

To work around this we will need to query the docker-distribution instance for the schema 1 manifest and note the Docker-Content-Digest header value.

Is support for V2 schema 2 on the roadmap for pulp_docker/crane? I have an idea about how it could work.

Actions #3

Updated by dkliban@redhat.com over 7 years ago

  • Tracker changed from Issue to Story
  • Project changed from Crane to Docker Support
  • Subject changed from Does not provide v2 schema 2 images to As a user, I can sync v2 schema manifests
  • Description updated (diff)
Actions #4

Updated by dkliban@redhat.com over 7 years ago

  • Sprint Candidate changed from No to Yes
  • Tags 2.11 added
Actions #5

Updated by amacdona@redhat.com over 7 years ago

  • Blocks Story #2102: Easier way to configure Crane to serve both V1 and V2 added
Actions #6

Updated by amacdona@redhat.com over 7 years ago

  • Groomed changed from No to Yes
Actions #7

Updated by jortel@redhat.com over 7 years ago

  • Sprint/Milestone set to 26
Actions #8

Updated by ipanova@redhat.com over 7 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com
Actions #9

Updated by ipanova@redhat.com over 7 years ago

Can you please specify if you need just schema v2 support of regular image manifest or you also need support manifests list, so called 'fat manifest'?

Actions #10

Updated by vrutkovs over 7 years ago

Discussed it with Amanda: currently we need just schema 2 (as we're planning to migrate to Docker 1.10 soon).

Note that multiarch implementation is also planned (thus fat manifest is needed as well), but it first has to be discussed with Content Delivery team

Actions #11

Updated by mhrivnak over 7 years ago

To support schema 1 and 2, we would need to:

  • change the manifest DB model to accomodate schema 2
  • retrieve both schema 1 and 2 during sync and save both
  • change the tag model to store both schema 1 and schema 2 manifest digests on the same instance, or add a schema field as part of the unit key
  • possibly change the tag copy logic to make sure it's replacing tag models correctly now that there are two digests to worry about
  • change the schema of the crane metadata file to have two manifest lists: one for schema 1, and one for schema 2
  • change publish to create the new crane metadata file
  • change crane to look for a optional header from the client and use that to decide whether to return a schema 1 or schema 2 manifest

@ipanova looked up that docker version 1.9 and older only do schema 1. We would consider just dropping schema 1 support, but that probably would not be acceptable at this time to only support docker 1.10+.

Actions #12

Updated by mhrivnak over 7 years ago

This issue is at least loosely related to this atomic registry bug:

https://github.com/openshift/origin/issues/10730

which was resolved here:

https://github.com/openshift/origin/pull/10805

and the fix appears to be available already upstream. Pulp has been tested syncing against that by @asmacdo

Actions #13

Updated by ipanova@redhat.com over 7 years ago

  • Description updated (diff)
Actions #14

Updated by ipanova@redhat.com over 7 years ago

  • Description updated (diff)
Actions #15

Updated by ipanova@redhat.com over 7 years ago

  • Status changed from ASSIGNED to NEW
  • Sprint/Milestone deleted (26)
Actions #16

Updated by ipanova@redhat.com over 7 years ago

  • Assignee deleted (ipanova@redhat.com)
Actions #17

Updated by ipanova@redhat.com over 7 years ago

  • Description updated (diff)
Actions #19

Updated by mhrivnak over 7 years ago

  • Description updated (diff)
  • Tags deleted (2.11)
Actions #20

Updated by ipanova@redhat.com over 7 years ago

  • Blocks Story #2384: As a user I can sync manifest lists added
Actions #21

Updated by mhrivnak over 7 years ago

  • Sprint/Milestone set to 30
Actions #22

Updated by ipanova@redhat.com over 7 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com
Actions #23

Updated by ipanova@redhat.com about 7 years ago

---N.B. 'docker repo tag' is broken. needs to be updated, --schema-version needs to be added as another required-field---

This requirement was doen on server side. From client perspective nothing changes

Actions #24

Updated by ipanova@redhat.com about 7 years ago

N.B docker rsync distributor needs to be modified as well

Docker rsync dist publish path was changed because of publish directory structure change in general

Actions #25

Updated by ipanova@redhat.com about 7 years ago

  • Sprint/Milestone changed from 30 to 33
Actions #27

Updated by mhrivnak about 7 years ago

  • Sprint/Milestone changed from 33 to 34
Actions #28

Updated by ipanova@redhat.com about 7 years ago

  • Status changed from ASSIGNED to POST
Actions #29

Updated by ipanova@redhat.com about 7 years ago

  • Description updated (diff)
Actions #30

Updated by ipanova@redhat.com about 7 years ago

wrote:

N.B docker rsync distributor needs to be modified as well

Docker rsync dist publish path was changed because of publish directory structure change in general

We solved this by adding to apache config to set proper response headers based on directory match

Added by ipanova@redhat.com about 7 years ago

Revision 35dc19c8 | View on GitHub

As a user, I can sync manifests schema version 2.

closes #2099

Added by ipanova@redhat.com about 7 years ago

Revision 35dc19c8 | View on GitHub

As a user, I can sync manifests schema version 2.

closes #2099

Added by ipanova@redhat.com about 7 years ago

Revision 35dc19c8 | View on GitHub

As a user, I can sync manifests schema version 2.

closes #2099

Added by ipanova@redhat.com about 7 years ago

Revision 35dc19c8 | View on GitHub

As a user, I can sync manifests schema version 2.

closes #2099

Actions #31

Updated by ipanova@redhat.com about 7 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #32

Updated by pcreech almost 7 years ago

  • Platform Release set to 2.13.0
  • Target Release - Docker set to 2.4.0
Actions #33

Updated by pcreech almost 7 years ago

  • Status changed from MODIFIED to 5
Actions #36

Updated by Ichimonji10 almost 7 years ago

This issue needs additional verification. See: https://github.com/PulpQE/pulp-smash/issues/555#issuecomment-297168241

Actions #38

Updated by pcreech almost 7 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #39

Updated by bmbouter about 6 years ago

  • Sprint set to Sprint 16
Actions #40

Updated by bmbouter about 6 years ago

  • Sprint/Milestone deleted (34)
Actions #41

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF