Project

Profile

Help

Story #3401

closed

Create content models for pulp_docker 4.0

Added by bizhang about 6 years ago. Updated over 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Sprint:
Sprint 42
Quarter:

Description

Manifest(Content) models a docker manifest:

  • digest: CharField(255, db_index=True, unique=True)
  • schema_version: IntegerField()
  • media_type: CharField(choices=)

media_type choices:

  • application/vnd.docker.distribution.manifest.v1+json
  • application/vnd.docker.distribution.manifest.v2+json

ManifestBlob(Content) models the relationship between Manifests and Blobs. Each blob is stored as an Artifact and is related by ContentArtifact.

  • digest: CharField(max_length=255, db_index=True, unique=True)
  • manifest: ForeignKey(Manifest, related_name='blobs', on_delete=CASCADE)
  • media_type: CharField(choices=)

media_type choices:

  • application/vnd.docker.container.image.v1+json - Container config json represented as a blob
  • application/vnd.docker.image.rootfs.diff.tar.gzip - Regular blob
  • application/vnd.docker.image.rootfs.foreign.diff.tar.gzip - Foreign blob ( we do not intend to store foreign blobs. Docker client during pull will fetch it from the remote location)

ManifestList(Content) models a docker manifest-list.

  • digest: CharField(max_length=255, db_index=True, unique=True)
  • schema_version: IntegerField()
  • manifests: ManyToManyField('Manifest', through='ManifestListManifest')
  • media_type: CharField(choices=)

media_type choices:

  • application/vnd.docker.distribution.manifest.list.v2+json

ManifestListManifest - models the many-to-many relationship between Manifest and ManifestList.

  • manifest: ForeignKey(Manifest, related_name='manifests', on_delete=CASCADE)
  • manifest_list: ForeignKey(ManifestList, related_name='manifest_lists', on_delete=CASCADE)
  • architecture: CharField(255)
  • os: CharField(255)
  • os_version: charField(255)
  • os_features: TextField()
  • variant: CharField(255)
  • features: TextField()

Tag(Content)

  • name CharField(255, db_index=True)
  • manifest ForeignKey(Manifest, null=True, related_name='tags', on_delete=CASCADE)
  • manifest_list ForeignKey(ManifestList, null=True, related_name='tags', on_delete=CASCADE)

unique_together = (name, manifest)
unique_together = (name, manifest_list)

Design assumptions:

  • Pre-save validation on the model ensures that one-of but not both manifest and manifest_list are set.
  • Tag name uniqueness and referential integrity{1} will be enforced by the plugin. During sync, this happens in the task logic. For content add/remove API calls, this happens in the (docker specific) view. Both can re-use the same code.

[1] Tag referential integrity is ensuring that when a manifest or manifest_list is removed from a repository, referencing tags are also removed. Also, when a tag is added to a repository, referenced manifest or manifest_list are also added.


This story will be complete when CRUD is possible for Manifests (for this story, the metadata is not expected to be extracted from the file, but provided by the user in the request body.)


Related issues

Has duplicate Container Support - Task #3950: Tag modelCLOSED - CURRENTRELEASE

Actions
Blocked by Container Support - Task #3398: Make pulp_docker/4.0-dev branch ready for plugin developmentCLOSED - CURRENTRELEASEamacdona@redhat.com

Actions
Blocks Container Support - Story #3989: Create content model serializers and viewsets for pulp_docker 4.0CLOSED - CURRENTRELEASEamacdona@redhat.com

Actions
Actions #1

Updated by bizhang about 6 years ago

  • Tags Pulp 3 added
Actions #2

Updated by ipanova@redhat.com about 6 years ago

before hardcoding the digest using sha256 hashing algoritm, i'd like to confirm that it is indeed so.
There are the docs, [0] but i don't believe them. I'd look into the code

[0] https://docs.docker.com/registry/spec/api/#content-digests

Actions #4

Updated by bizhang about 6 years ago

  • Description updated (diff)

Updated length of digest field, since sha512 might be supported

Actions #5

Updated by bizhang about 6 years ago

  • Description updated (diff)
Actions #6

Updated by bizhang about 6 years ago

  • Description updated (diff)
Actions #7

Updated by bizhang about 6 years ago

  • Description updated (diff)
Actions #8

Updated by bizhang about 6 years ago

  • Subject changed from Create Image Manifest Models for pulp_docker 4.0 to Create Manifest Models for pulp_docker 4.0
  • Description updated (diff)
Actions #9

Updated by bizhang about 6 years ago

  • Description updated (diff)
Actions #10

Updated by ipanova@redhat.com about 6 years ago

  • Description updated (diff)
Actions #11

Updated by bizhang about 6 years ago

  • Description updated (diff)
Actions #12

Updated by ipanova@redhat.com about 6 years ago

there can be a signed manifest application/vnd.docker.distribution.manifest.v1+prettyjws, we need to investigate on this, and extend the choice field in the Manifest model.

Actions #13

Updated by ipanova@redhat.com about 6 years ago

I gave some more thought to the mediatype field. Do we really want to have it on the model?
Image manifest schema2 does have this in the json file, meanwhile image manifest schema 1 does not. The only way to figure out what's the mediaType of the schema 1( signed or not) is to check the response headers.

https://docs.docker.com/registry/spec/manifest-v2-1/#manifest-field-descriptions
https://docs.docker.com/registry/spec/manifest-v2-2/#image-manifest-field-descriptions

In case we'd decide to remove this field from the model, i would do the same for ManifestList, for consistency.

Actions #14

Updated by bizhang about 6 years ago

  • Description updated (diff)

s/layer/blob/

Actions #15

Updated by ipanova@redhat.com about 6 years ago

  • Sprint Candidate changed from No to Yes
Actions #16

Updated by amacdona@redhat.com about 6 years ago

  • Subject changed from Create Manifest Models for pulp_docker 4.0 to Create Manifest Model, ViewSet, Serializer for pulp_docker 4.0
  • Description updated (diff)
Actions #17

Updated by amacdona@redhat.com about 6 years ago

  • Description updated (diff)
Actions #18

Updated by amacdona@redhat.com about 6 years ago

We only need schema 2 for MVP. But please namespace this appropriately to later add schema 1.

Namespace meaning model and vs endpoint?

Actions #19

Updated by dkliban@redhat.com over 5 years ago

  • Groomed changed from No to Yes
Actions #20

Updated by ipanova@redhat.com over 5 years ago

  • Sprint set to Sprint 41
Actions #21

Updated by rchan over 5 years ago

  • Sprint changed from Sprint 41 to Sprint 42
Actions #22

Updated by jortel@redhat.com over 5 years ago

  • Blocked by Task #3398: Make pulp_docker/4.0-dev branch ready for plugin development added
Actions #23

Updated by jortel@redhat.com over 5 years ago

  • Description updated (diff)
Actions #24

Updated by jortel@redhat.com over 5 years ago

Actions #25

Updated by jortel@redhat.com over 5 years ago

  • Description updated (diff)
Actions #26

Updated by jortel@redhat.com over 5 years ago

  • Description updated (diff)
Actions #27

Updated by jortel@redhat.com over 5 years ago

  • Description updated (diff)
Actions #28

Updated by jortel@redhat.com over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com
Actions #29

Updated by jortel@redhat.com over 5 years ago

  • Description updated (diff)
Actions #30

Updated by jortel@redhat.com over 5 years ago

  • Description updated (diff)
Actions #31

Updated by jortel@redhat.com over 5 years ago

  • Description updated (diff)
Actions #32

Updated by jortel@redhat.com over 5 years ago

  • Description updated (diff)
Actions #33

Updated by jortel@redhat.com over 5 years ago

  • Description updated (diff)
Actions #34

Updated by jortel@redhat.com over 5 years ago

  • Description updated (diff)
Actions #35

Updated by jortel@redhat.com over 5 years ago

  • Description updated (diff)
Actions #36

Updated by jortel@redhat.com over 5 years ago

  • Subject changed from Create Manifest Model, ViewSet, Serializer for pulp_docker 4.0 to Create content models for pulp_docker 4.0
  • Description updated (diff)
Actions #37

Updated by jortel@redhat.com over 5 years ago

  • Description updated (diff)
Actions #38

Updated by jortel@redhat.com over 5 years ago

  • Description updated (diff)
Actions #39

Updated by jortel@redhat.com over 5 years ago

  • Status changed from ASSIGNED to POST
Actions #40

Updated by jortel@redhat.com over 5 years ago

  • Blocks Story #3989: Create content model serializers and viewsets for pulp_docker 4.0 added
Actions #41

Updated by jortel@redhat.com over 5 years ago

Do we really want to use choices= in the CharField for media_type? Bounding the permitted value seems like a better fit for the ViewSet. Or, was the intent to store encoded values to be translated by the field?

Thoughts?

Actions #42

Updated by amacdona@redhat.com over 5 years ago

A choice field seems appropriate to me, because there are a finite set of acceptable values. @jortel, I might be missing something, why would we want to restrict these choices on the ViewSet and allow invalid choices in the db?

Actions #43

Updated by jortel@redhat.com over 5 years ago

Certainly I agree that it would be best for the field to perform the validation.

However, the choices= parameter seems to be geared for value translation rather than field validation. I get this impression mainly because choices takes a list of tuples (actual-value, human-readable). Also, it's my understanding that field validation only get's invoked by one of the clean() methods which are not called on Model.save(). Best I can tell, the clean() methods only get called from ModelForm which I don't think we're using.

I'm likely wrong about how django works here. The docs are a little unclear. :)

Can someone confirm that using the choices= in this way (in pulp) will result in enforced field validation?

https://docs.djangoproject.com/en/2.1/ref/models/fields/#choices
https://docs.djangoproject.com/en/2.1/ref/models/instances/#validating-objects

Added by jortel@redhat.com over 5 years ago

Revision 4c5cd707 | View on GitHub

Add docker content models. closes #3401

Added by jortel@redhat.com over 5 years ago

Revision 4c5cd707 | View on GitHub

Add docker content models. closes #3401

Added by jortel@redhat.com over 5 years ago

Revision 4c5cd707 | View on GitHub

Add docker content models. closes #3401

Added by jortel@redhat.com over 5 years ago

Revision 4c5cd707 | View on GitHub

Add docker content models. closes #3401

Actions #44

Updated by jortel@redhat.com over 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #45

Updated by amacdona@redhat.com over 5 years ago

Actions #46

Updated by amacdona@redhat.com over 5 years ago

Actions #47

Updated by bmbouter about 5 years ago

  • Tags deleted (Pulp 3)
Actions #48

Updated by ipanova@redhat.com over 4 years ago

  • Project changed from Docker Support to Container Support
Actions #49

Updated by ipanova@redhat.com over 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions #50

Updated by ipanova@redhat.com over 4 years ago

  • Sprint/Milestone set to 1.0.0

Also available in: Atom PDF