Project

Profile

Help

Story #2099

As a user, I can sync v2 schema manifests

Added by twaugh over 3 years ago. Updated 11 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Platform Release:
2.13.0
Blocks Release:
Target Release - Docker:
2.4.0
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
Yes
Verification Required:
Yes
Sprint:
Sprint 16

Description

As a result of this story, Pulp Docker will support both schema versions, 1 and 2 for sync, publish and copy.

To support both schema 1 and 2 we need to:
1. during sync always make 2 requests to the registry and in each request set properly the headers.
2. as a result of 2 requests we will have to save to the DB manifest with both schemas.
3. change Tag model and add schema_version field and include it into the unit_key( having 2 instances of tags will facilitate copy operation)
4. make sure copy operation works with above changes related to Tag model.
5. write a migration for Tag model change, also write a migration to drop old index
6. in manifest schema 2 there is a config array that contains a config blob, which is treated as regular layer. In order to have a runable docker image config layer should also be present. That means that we would need to download it and store in the DB( as a regular blob layer). That means that most likely we would need to change Manifest model and add a config layer blob field to it. No migration needs to be written for this field
7. change the schema of the crane metadata file to have two manifest lists: one for schema 1, and one for schema 2
8. change publish to create the new crane metadata file, also change publish directory structure for manifests

Note:
this work will not enable support for manifest list( aka fat manifest)


Checklist


Subtasks

Crane - Story #2336: Update crane to serve manifest schema 2CLOSED - CURRENTRELEASEipanova@redhat.com

Actions

Related issues

Blocks Crane - Story #2102: Easier way to configure Crane to serve both V1 and V2 CLOSED - WONTFIX Actions
Blocks Docker Support - Story #2384: As a user I can sync manifest lists CLOSED - CURRENTRELEASE Actions

Associated revisions

Revision 35dc19c8 View on GitHub
Added by ipanova@redhat.com almost 3 years ago

As a user, I can sync manifests schema version 2.

closes #2099

Revision 35dc19c8 View on GitHub
Added by ipanova@redhat.com almost 3 years ago

As a user, I can sync manifests schema version 2.

closes #2099

Revision 35dc19c8 View on GitHub
Added by ipanova@redhat.com almost 3 years ago

As a user, I can sync manifests schema version 2.

closes #2099

Revision 35dc19c8 View on GitHub
Added by ipanova@redhat.com almost 3 years ago

As a user, I can sync manifests schema version 2.

closes #2099

History

#1 Updated by mhrivnak over 3 years ago

Can you elaborate on the openshift need for schema 2? After investigating schema 2, we determined in consultation with @vbatts that all docker-related tooling should still be compatible with schema 1, and there should not be a need to support both.

#2 Updated by twaugh over 3 years ago

Discussed this with today and I think this came from a misunderstanding.

However, thinking around the issue today I realised something else relating to it.

Once we update our container image builder to docker 1.10, we will be pushing images to a docker-distribution instance and these images will have V2 schema 2 manifests. Our build system notes the manifest digest reported by docker after the push, then tells Pulp to sync from docker-distribution.

However, the digest noted by the build system and stored in Koji for the build will be different to the one available in Crane: after 'docker push' the digest is for a schema 2 manifest, but after 'sync' the manifest digest Crane will know will be for a schema 1 manifest.

To work around this we will need to query the docker-distribution instance for the schema 1 manifest and note the Docker-Content-Digest header value.

Is support for V2 schema 2 on the roadmap for pulp_docker/crane? I have an idea about how it could work.

#3 Updated by dkliban@redhat.com over 3 years ago

  • Tracker changed from Issue to Story
  • Project changed from Crane to Docker Support
  • Subject changed from Does not provide v2 schema 2 images to As a user, I can sync v2 schema manifests
  • Description updated (diff)

#4 Updated by dkliban@redhat.com over 3 years ago

  • Sprint Candidate changed from No to Yes
  • Tags 2.11 added

#5 Updated by amacdona@redhat.com over 3 years ago

  • Blocks Story #2102: Easier way to configure Crane to serve both V1 and V2 added

#6 Updated by amacdona@redhat.com over 3 years ago

  • Groomed changed from No to Yes

#7 Updated by jortel@redhat.com over 3 years ago

  • Sprint/Milestone set to 26

#8 Updated by ipanova@redhat.com over 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com

#9 Updated by ipanova@redhat.com over 3 years ago

Can you please specify if you need just schema v2 support of regular image manifest or you also need support manifests list, so called 'fat manifest'?

#10 Updated by vrutkovs over 3 years ago

Discussed it with Amanda: currently we need just schema 2 (as we're planning to migrate to Docker 1.10 soon).

Note that multiarch implementation is also planned (thus fat manifest is needed as well), but it first has to be discussed with Content Delivery team

#11 Updated by mhrivnak over 3 years ago

To support schema 1 and 2, we would need to:

  • change the manifest DB model to accomodate schema 2
  • retrieve both schema 1 and 2 during sync and save both
  • change the tag model to store both schema 1 and schema 2 manifest digests on the same instance, or add a schema field as part of the unit key
  • possibly change the tag copy logic to make sure it's replacing tag models correctly now that there are two digests to worry about
  • change the schema of the crane metadata file to have two manifest lists: one for schema 1, and one for schema 2
  • change publish to create the new crane metadata file
  • change crane to look for a optional header from the client and use that to decide whether to return a schema 1 or schema 2 manifest

@ipanova looked up that docker version 1.9 and older only do schema 1. We would consider just dropping schema 1 support, but that probably would not be acceptable at this time to only support docker 1.10+.

#12 Updated by mhrivnak over 3 years ago

This issue is at least loosely related to this atomic registry bug:

https://github.com/openshift/origin/issues/10730

which was resolved here:

https://github.com/openshift/origin/pull/10805

and the fix appears to be available already upstream. Pulp has been tested syncing against that by @asmacdo

#13 Updated by ipanova@redhat.com over 3 years ago

  • Description updated (diff)

#14 Updated by ipanova@redhat.com over 3 years ago

  • Description updated (diff)

#15 Updated by ipanova@redhat.com over 3 years ago

  • Status changed from ASSIGNED to NEW
  • Sprint/Milestone deleted (26)

#16 Updated by ipanova@redhat.com over 3 years ago

  • Assignee deleted (ipanova@redhat.com)

#17 Updated by ipanova@redhat.com over 3 years ago

  • Description updated (diff)

#18 Updated by ipanova@redhat.com over 3 years ago

  • Checklist item add necessary code to enable schema 2 support added
  • Checklist item write necessary migrations ( 2 migrations) added
  • Checklist item add docs added
  • Checklist item add release notes added
  • Checklist item check that other operations like copy and publish are not broken added

#19 Updated by mhrivnak over 3 years ago

  • Description updated (diff)
  • Tags deleted (2.11)

#20 Updated by ipanova@redhat.com over 3 years ago

  • Blocks Story #2384: As a user I can sync manifest lists added

#21 Updated by mhrivnak about 3 years ago

  • Sprint/Milestone set to 30

#22 Updated by ipanova@redhat.com about 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com

#23 Updated by ipanova@redhat.com about 3 years ago

---N.B. 'docker repo tag' is broken. needs to be updated, --schema-version needs to be added as another required-field---

This requirement was doen on server side. From client perspective nothing changes

#24 Updated by ipanova@redhat.com about 3 years ago

N.B docker rsync distributor needs to be modified as well

Docker rsync dist publish path was changed because of publish directory structure change in general

#25 Updated by ipanova@redhat.com about 3 years ago

  • Sprint/Milestone changed from 30 to 33

#26 Updated by dkliban@redhat.com about 3 years ago

  • Smash Test set to 555

#27 Updated by mhrivnak almost 3 years ago

  • Sprint/Milestone changed from 33 to 34

#28 Updated by ipanova@redhat.com almost 3 years ago

  • Status changed from ASSIGNED to POST
  • Verification Required changed from No to Yes

#29 Updated by ipanova@redhat.com almost 3 years ago

  • Description updated (diff)

#30 Updated by ipanova@redhat.com almost 3 years ago

wrote:

N.B docker rsync distributor needs to be modified as well

Docker rsync dist publish path was changed because of publish directory structure change in general

We solved this by adding to apache config to set proper response headers based on directory match

#31 Updated by ipanova@redhat.com almost 3 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#32 Updated by pcreech almost 3 years ago

  • Platform Release set to 2.13.0
  • Target Release - Docker set to 2.4.0

#33 Updated by pcreech almost 3 years ago

  • Status changed from MODIFIED to ON_QA

#34 Updated by Ichimonji10 almost 3 years ago

  • Verified changed from No to Yes

#35 Updated by Ichimonji10 almost 3 years ago

  • Verified changed from Yes to No

#36 Updated by Ichimonji10 almost 3 years ago

This issue needs additional verification. See: https://github.com/PulpQE/pulp-smash/issues/555#issuecomment-297168241

#37 Updated by Ichimonji10 almost 3 years ago

  • Verified changed from No to Yes

#38 Updated by pcreech almost 3 years ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#39 Updated by bmbouter almost 2 years ago

  • Sprint set to Sprint 16

#40 Updated by bmbouter almost 2 years ago

  • Sprint/Milestone deleted (34)

#41 Updated by bmbouter 11 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF