Project

Profile

Help

Story #3401

Create content models for pulp_docker 4.0

Added by bizhang over 1 year ago. Updated 7 months ago.

Status:
MODIFIED
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Platform Release:
Blocks Release:
Target Release - Docker:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 42

Description

Manifest(Content) models a docker manifest:

  • digest: CharField(255, db_index=True, unique=True)
  • schema_version: IntegerField()
  • media_type: CharField(choices=)
media_type choices:
  • application/vnd.docker.distribution.manifest.v1+json
  • application/vnd.docker.distribution.manifest.v2+json

ManifestBlob(Content) models the relationship between Manifests and Blobs. Each blob is stored as an Artifact and is related by ContentArtifact.

  • digest: CharField(max_length=255, db_index=True, unique=True)
  • manifest: ForeignKey(Manifest, related_name='blobs', on_delete=CASCADE)
  • media_type: CharField(choices=)

media_type choices:

  • application/vnd.docker.container.image.v1+json - Container config json represented as a blob
  • application/vnd.docker.image.rootfs.diff.tar.gzip - Regular blob
  • application/vnd.docker.image.rootfs.foreign.diff.tar.gzip - Foreign blob ( we do not intend to store foreign blobs. Docker client during pull will fetch it from the remote location)

ManifestList(Content) models a docker manifest-list.
  • digest: CharField(max_length=255, db_index=True, unique=True)
  • schema_version: IntegerField()
  • manifests: ManyToManyField('Manifest', through='ManifestListManifest')
  • media_type: CharField(choices=)
media_type choices:
  • application/vnd.docker.distribution.manifest.list.v2+json

ManifestListManifest - models the many-to-many relationship between Manifest and ManifestList.
  • manifest: ForeignKey(Manifest, related_name='manifests', on_delete=CASCADE)
  • manifest_list: ForeignKey(ManifestList, related_name='manifest_lists', on_delete=CASCADE)
  • architecture: CharField(255)
  • os: CharField(255)
  • os_version: charField(255)
  • os_features: TextField()
  • variant: CharField(255)
  • features: TextField()

Tag(Content)
  • name CharField(255, db_index=True)
  • manifest ForeignKey(Manifest, null=True, related_name='tags', on_delete=CASCADE)
  • manifest_list ForeignKey(ManifestList, null=True, related_name='tags', on_delete=CASCADE)

unique_together = (name, manifest)
unique_together = (name, manifest_list)

Design assumptions:

  • Pre-save validation on the model ensures that one-of but not both manifest and manifest_list are set.
  • Tag name uniqueness and referential integrity{1} will be enforced by the plugin. During sync, this happens in the task logic. For content add/remove API calls, this happens in the (docker specific) view. Both can re-use the same code.

[1] Tag referential integrity is ensuring that when a manifest or manifest_list is removed from a repository, referencing tags are also removed. Also, when a tag is added to a repository, referenced manifest or manifest_list are also added.


This story will be complete when CRUD is possible for Manifests (for this story, the metadata is not expected to be extracted from the file, but provided by the user in the request body.)


Related issues

Duplicated by Docker Support - Task #3950: Tag model MODIFIED Actions
Blocked by Docker Support - Task #3398: Make pulp_docker/4.0-dev branch ready for plugin development MODIFIED Actions
Blocks Docker Support - Story #3989: Create content model serializers and viewsets for pulp_docker 4.0 MODIFIED Actions

Associated revisions

Revision 4c5cd707 View on GitHub
Added by jortel@redhat.com about 1 year ago

Add docker content models.
closes #3401

Revision 4c5cd707 View on GitHub
Added by jortel@redhat.com about 1 year ago

Add docker content models.
closes #3401

History

#1 Updated by bizhang over 1 year ago

  • Tags Pulp 3 added

#2 Updated by ipanova@redhat.com over 1 year ago

before hardcoding the digest using sha256 hashing algoritm, i'd like to confirm that it is indeed so.
There are the docs, [0] but i don't believe them. I'd look into the code

[0] https://docs.docker.com/registry/spec/api/#content-digests

#4 Updated by bizhang over 1 year ago

  • Description updated (diff)

Updated length of digest field, since sha512 might be supported

#5 Updated by bizhang over 1 year ago

  • Description updated (diff)

#6 Updated by bizhang over 1 year ago

  • Description updated (diff)

#7 Updated by bizhang over 1 year ago

  • Description updated (diff)

#8 Updated by bizhang over 1 year ago

  • Subject changed from Create Image Manifest Models for pulp_docker 4.0 to Create Manifest Models for pulp_docker 4.0
  • Description updated (diff)

#9 Updated by bizhang over 1 year ago

  • Description updated (diff)

#10 Updated by ipanova@redhat.com over 1 year ago

  • Description updated (diff)

#11 Updated by bizhang over 1 year ago

  • Description updated (diff)

#12 Updated by ipanova@redhat.com over 1 year ago

there can be a signed manifest application/vnd.docker.distribution.manifest.v1+prettyjws, we need to investigate on this, and extend the choice field in the Manifest model.

#13 Updated by ipanova@redhat.com over 1 year ago

I gave some more thought to the mediatype field. Do we really want to have it on the model?
Image manifest schema2 does have this in the json file, meanwhile image manifest schema 1 does not. The only way to figure out what's the mediaType of the schema 1( signed or not) is to check the response headers.

https://docs.docker.com/registry/spec/manifest-v2-1/#manifest-field-descriptions
https://docs.docker.com/registry/spec/manifest-v2-2/#image-manifest-field-descriptions

In case we'd decide to remove this field from the model, i would do the same for ManifestList, for consistency.

#14 Updated by bizhang over 1 year ago

  • Description updated (diff)

s/layer/blob/

#15 Updated by ipanova@redhat.com over 1 year ago

  • Sprint Candidate changed from No to Yes

#16 Updated by amacdona@redhat.com over 1 year ago

  • Subject changed from Create Manifest Models for pulp_docker 4.0 to Create Manifest Model, ViewSet, Serializer for pulp_docker 4.0
  • Description updated (diff)

#17 Updated by amacdona@redhat.com over 1 year ago

  • Description updated (diff)

#18 Updated by amacdona@redhat.com over 1 year ago

We only need schema 2 for MVP. But please namespace this appropriately to later add schema 1.

Namespace meaning model and vs endpoint?

#19 Updated by dkliban@redhat.com over 1 year ago

  • Groomed changed from No to Yes

#20 Updated by ipanova@redhat.com about 1 year ago

  • Sprint set to Sprint 41

#21 Updated by rchan about 1 year ago

  • Sprint changed from Sprint 41 to Sprint 42

#22 Updated by jortel@redhat.com about 1 year ago

  • Blocked by Task #3398: Make pulp_docker/4.0-dev branch ready for plugin development added

#23 Updated by jortel@redhat.com about 1 year ago

  • Description updated (diff)

#24 Updated by jortel@redhat.com about 1 year ago

#25 Updated by jortel@redhat.com about 1 year ago

  • Description updated (diff)

#26 Updated by jortel@redhat.com about 1 year ago

  • Description updated (diff)

#27 Updated by jortel@redhat.com about 1 year ago

  • Description updated (diff)

#28 Updated by jortel@redhat.com about 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com

#29 Updated by jortel@redhat.com about 1 year ago

  • Description updated (diff)

#30 Updated by jortel@redhat.com about 1 year ago

  • Description updated (diff)

#31 Updated by jortel@redhat.com about 1 year ago

  • Description updated (diff)

#32 Updated by jortel@redhat.com about 1 year ago

  • Description updated (diff)

#33 Updated by jortel@redhat.com about 1 year ago

  • Description updated (diff)

#34 Updated by jortel@redhat.com about 1 year ago

  • Description updated (diff)

#35 Updated by jortel@redhat.com about 1 year ago

  • Description updated (diff)

#36 Updated by jortel@redhat.com about 1 year ago

  • Checklist item deleted (add fields to Model)
  • Checklist item deleted (add fields and help_text to Serializer)
  • Checklist item deleted (Create a ViewSet)
  • Subject changed from Create Manifest Model, ViewSet, Serializer for pulp_docker 4.0 to Create content models for pulp_docker 4.0
  • Description updated (diff)

#37 Updated by jortel@redhat.com about 1 year ago

  • Description updated (diff)

#38 Updated by jortel@redhat.com about 1 year ago

  • Description updated (diff)

#39 Updated by jortel@redhat.com about 1 year ago

  • Status changed from ASSIGNED to POST

#40 Updated by jortel@redhat.com about 1 year ago

  • Blocks Story #3989: Create content model serializers and viewsets for pulp_docker 4.0 added

#41 Updated by jortel@redhat.com about 1 year ago

Do we really want to use choices= in the CharField for media_type? Bounding the permitted value seems like a better fit for the ViewSet. Or, was the intent to store encoded values to be translated by the field?

Thoughts?

#42 Updated by amacdona@redhat.com about 1 year ago

A choice field seems appropriate to me, because there are a finite set of acceptable values. @jortel, I might be missing something, why would we want to restrict these choices on the ViewSet and allow invalid choices in the db?

#43 Updated by jortel@redhat.com about 1 year ago

Certainly I agree that it would be best for the field to perform the validation.

However, the choices= parameter seems to be geared for value translation rather than field validation. I get this impression mainly because choices takes a list of tuples (actual-value, human-readable). Also, it's my understanding that field validation only get's invoked by one of the clean() methods which are not called on Model.save(). Best I can tell, the clean() methods only get called from ModelForm which I don't think we're using.

I'm likely wrong about how django works here. The docs are a little unclear. :)

Can someone confirm that using the choices= in this way (in pulp) will result in enforced field validation?

https://docs.djangoproject.com/en/2.1/ref/models/fields/#choices
https://docs.djangoproject.com/en/2.1/ref/models/instances/#validating-objects

#44 Updated by jortel@redhat.com about 1 year ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#45 Updated by amacdona@redhat.com 12 months ago

#46 Updated by amacdona@redhat.com 12 months ago

#47 Updated by bmbouter 7 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF