Docker Support

To summarize a discussion on #pulp-dev with @ipanova:

With a Docker registry at registry.example.com, I could create a "myproduct" repository, and store docker images for two containers, "app-nginx" and "app-mysql". Assuming that the image name matches the container name, and we model versioning with tags (as is typical in Docker):

registry.example.com/myproduct/app-nginx:1.0.0
registry.example.com/myproduct/app-mysql:1.2.3

Typically there is a :latest tag that points to the latest tag of each image (but that's just a convention). So those two images can also be addressed as:

registry.example.com/myproduct/app-nginx:latest
registry.example.com/myproduct/app-mysql:latest

which would fetch respectively versions 1.0.0 of app-nginx and 1.2.3 of app-mysql

This scenario cannot be currently implemented in pulp.

The best one can implement (at least based on my understanding) is create a "myproduct" pulp repo, and merge the image name and the tag together. The four tags would become:

app-nginx-1.0.0 and app-nginx-latest (pointing to the same unit)
app-mysql-1.2.3 and app-mysql-latest (pointing to the same unit)

in docker:
you create namespace 'myproduct' under which you push images like "app-nginx" and "app-mysql".
you have an image id 123( which is app-nginx), you tag it 1.0.0 and push to myproduct/app-ngnix( the concatenation of myproduct and app-ngnix make a concept of docker repository)
you tag same image id 123( which is app-nginx) with latest tag and push to myproduct/app-nginx

you have an image id 321( which is app-mysql) you tag it 1.2.3 and push to myproduct/app-mysql
you tag same image id 321( which is app-mysql) with latest tag and push to myproduct/app-mysql

in pulp you create repo A and sync content from myproduct/app-ngnix( because you cannot sync just from myproduclt, there is even no such endpoint in docker registry API). As a result you would have in pulp 2 tags( latest and 1.0.0)
in pulp you create repo B sync content from myproduct/app-mysql. As a result you would have in pulp 2 tags ( latest and 1.2.3)

then you would do docker pull A:latest or docker pull A:1.0.0 and as a result you would get same image id
docker pull B:latest or docker pull 1.2.3 and as a result you would get same image id

You cannot have in pulp under one repo name 'myproduct' both app-nginx and app-mysql
Yes in docker they are under same namespace, but still considered as separate thing, just because when you try to pull repo tags, you don's access 'myproduct' but 'myproduct/appnginx' https://docs.docker.com/registry/spec/api/#tags
https://docs.docker.com/registry/spec/api/#catalog the catalog will return you 2 different repo ---> so as a conclusion, myproduct cannot be considered as as repo because Again there is even no such endpoint in the registry API, a repo is called --> myproduct/foo which can contain multiple tags and images( which are image manifests or manifests list) but all of them will be related to foo, probably different versions of foo, but just foo, not bar. This is how docker works.

https://docs.docker.com/registry/spec/api/#overview

I don't know how to explain in other words, maybe mhrivnak will give me a hand here. Or maybe i am wrong, but the probably my docker world will just collapse :D

I guess there are different ways of looking at the problem.

I understand why you reached the decision to map "image names" at the repo level, when the criterion is that, using a native API, I can enumerate all images in a native repo (e.g. yum has the repomd to accomplish that). Docker's /v2/_catalog API is not quite a match for that.

On the other hand, a Docker image is supporting a Docker container, which in a typical use is one process. A standalone Docker container typically has little value, and you would have more than one container in a deployment. From this perspective, being able to place disparate, but related, Docker images in the same pulp repo is valuable at least to me, because it brings Docker in line with all the other units we (SAS) manage in pulp: rpm, msi, debian, tar (forget I mentioned it), bosh releases.

I think @ipanova is on the right track here. As far as I understand it, a "repository" in a docker registry is referenced by a path with two or more segments. The conceptual difference you are suggesting, where the first segment is a repository name, and the second is an image name within that repository, to my understanding does not line up with how I see registries getting used, nor have I seen such a semantic intent expressed from the docker project. It is certainly a valid idea, and the semantics you suggest could be a reasonable way for you to utilize the path namespace of your repositories.

I see docker repositories as more similar to a github repository than a yum repository. On github, a repository can be referenced as a path with two segments. The first segment is a user or organization, and the second segment identifies the name of an individual source repository. With your examples, I would expect to have github.com/mhrivnak/app-nginx/ and github.com/mhrivnak/app-rest/. Within each repository can be any number of branches and tags to track the lifecycle of that specific app.

In both a github repo and a docker registry repo, you can have tags/branches that contain completely different software and identify that by name, but it's not usually recommended.

In sum, I don't think docker has given us concrete primitives for differentiating a "repository" from an "image name". I believe those two concepts are generally considered to be equivalent within the ecosystem.

That said, if you know of any discussion within the docker ecosystem that supports a semantic differentiation between the path segments that identify a repository, we would love to see it and figure out how to support that.

Hopefully this doesn't sound like arguing for the purpose of arguing...

Let's leave the Docker registry semantics live their painful sad life for a second.

Our pipeline is currently, for all other unit types (rpm/deb/msi/msm/boshrelease)

1. build unit (koji or equivalent)
2. upload unit to existing repository in pulp
3. copy unit to its customer-facing location

Step 1 can produce any unit name, we are not in control of it.

With the current docker implementation, step 2 is impossible to implement, because, in order to not lose the image name, I would need to create one repo per image name, and I cannot create them ahead of time (see previous comment on step 1).

The problems:

• Right now the pulp users performing step 2 are not allowed to create repositories, only to upload to existing repositories. We made that choice some time ago. It can be changed, of course, but it is a change
• it makes Docker conceptually very different from everything else, and for no good reason.

One more data point: the Debian plugin had to implement a "repository in a repository" concept already, because of how Apt repositories are structured. So even with your semantic interpretation of Docker repositories, there is a precedent in making "sub-repos".

i suggest to close this issue:
1) since whole quintessence of the technical discussion moved to #3136
2) there is nothing to clarify here, we use regular docker concepts conform docker specs and terms which could be found on their official docs pages.