Docker Support

Found https://pulp.plan.io/issues/3896, which indeed made getting to an actual error message more complex.

For comparison, here is the s2v2 manifest generated by skopeo from the same container build by docker:

{'config': {'digest': 'sha256:80df321f2ea1b2a5b829a49735755198e7442aa41a4103283e6e4f9ebe9a335e',
            'mediaType': 'application/vnd.docker.container.image.v1+json',
            'size': 3002},
 'layers': [{'digest': 'sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f',
             'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
             'size': 208811520},
            {'digest': 'sha256:b48263b07da508175cef044d90311cde227a8a7330f5c7cac423977c40525043',
             'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
             'size': 5120},
            {'digest': 'sha256:f365f9181415ead2c548882d7d6cca35d3610210ec3b8c970871cc56863fa1e4',
             'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
             'size': 3584},
            {'digest': 'sha256:fc55bc91557a901996323b53397c9e7b823b619e07786032d61b1328c94e7309',
             'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
             'size': 6656}],
 'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
 'schemaVersion': 2}

And this container also caused the same error on upload.

I was not able to reproduce this. I tried skopeo copy same as buildah push.

$ sudo buildah images
IMAGE ID             IMAGE NAME                                               CREATED AT             SIZE
2cc378c061f7         docker.io/library/python:latest                          Oct 25, 2018 00:36     949 MB
0ede40d5477c         localhost/hellofromcontainer:latest                      Oct 30, 2018 12:38     949 MB

[ipanova@ina tmp]$ sudo buildah push --f v2s2 0ede40d5477c dir:/tmp/tada

$ cd /tmp/tada/ && tar -cvf ../tada.tar * && cd ..
pulp-admin -vv docker repo uploads upload --repo-id madness --file ./tada.tar
2018-10-30 12:57:08,243 - INFO - GET request to /pulp/api/v2/tasks/3067d69b-a2b2-465d-bad4-3d87b819d602/ with parameters None
2018-10-30 12:57:08,243 - INFO - Response status : 200 

2018-10-30 12:57:08,243 - INFO - Response body :
 {
  "exception": null, 
  "task_type": "pulp.server.managers.content.upload.import_uploaded_unit", 
  "_href": "/pulp/api/v2/tasks/3067d69b-a2b2-465d-bad4-3d87b819d602/", 
  "task_id": "3067d69b-a2b2-465d-bad4-3d87b819d602", 
  "tags": [
    "pulp:repository:madness", 
    "pulp:action:import_upload"
  ], 
  "finish_time": "2018-10-30T11:57:08Z", 
  "_ns": "task_status", 
  "start_time": "2018-10-30T11:56:39Z", 
  "traceback": null, 
  "spawned_tasks": [], 
  "progress_report": {}, 
  "queue": "reserved_resource_worker-0.dq2", 
  "state": "finished", 
  "worker_name": "reserved_resource_worker-0", 
  "result": {
    "details": {
      "unit": {
        "type_id": "docker_manifest", 
        "unit_key": {
          "digest": "sha256:b6c910a91860f695da46c73ebd784653d6eee3767a4f8f646a04c9b6f084f7f7"
        }, 
        "metadata": {
          "fs_layers": [
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 105529344, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:f715ed19c28b66943ac8bc12dbfb828e8394de2530bbaf1ecce906e748e4fdff"
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 23989248, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:8bb25f9cdc41e7d085033af15a522973b44086d6eedd24c11cc61c9232324f77"
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 7994368, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:08a01612ffca33483a1847c909836610610ce523fb7e1aca880140ee84df23e9"
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 146405888, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:1191b3f5862aa9231858809b7ac8b91c0b727ce85c9b3279932f0baacc92967d"
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 570565632, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:9978d084fd771e0b3d1acd7f3525d1b25288ababe9ad8ed259b36101e4e3addd"
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 17495552, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:d62f0ea9a15e2a05779741bff6654624b7dbd6c12a7292205c81bee0ff53f9a1"
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 70766080, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:25465864233d59ce8667ffe16e09780ea7e38f99f9c39d33a5263f5613f2a3fe"
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 4608, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:a474157cdf60afc9e274c9b529827a06103dab9ba9b15c577da1dc1cd82083d4"
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 6341632, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:7f8f0e833e519f17b9e42f232d4bcd561939a6697f401b1bdc7af26b0657bf6b"
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 2048, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:f9ee991634f497ecf188e9034b3dfb58d98f456459ef2b392ffb9525b4f9d350"
            }
          ], 
          "schema_version": 2, 
          "downloaded": true, 
          "pulp_user_metadata": {}, 
          "config_layer": "sha256:0ede40d5477c25a9b4cd7a9bbe96e629c90da581ae49d1196606d21371642b3a", 
          "id": "076b91f3-e99c-4b97-91e9-e25249e0e73e", 
          "digest": "sha256:b6c910a91860f695da46c73ebd784653d6eee3767a4f8f646a04c9b6f084f7f7"
        }
      }
    }, 
    "success_flag": true, 
    "summary": ""
  }, 
  "error": null, 
  "_id": {
    "$oid": "5bd846f7ce1b25e899d9fcb4"
  }, 
  "id": "5bd846f7ce1b25e899d9fcb4"
[\]
Running...

Task Succeeded

Please provide specific steps to reproduce. I am new buildah so maybe i missed something.

cannot reproduce, followed the mentioned steps above ^

[ipanova@ina pulp_docker]$ pulp-admin docker repo create --repo-id alpine
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

Repository [alpine] successfully created

$ pulp-admin docker repo uploads upload --repo-id alpine --file ./alpine2.tar
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: alpine2.tar
... completed

Creating upload requests on the server...
[==================================================] 100%
Initializing: alpine2.tar
... completed

Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
canceled entirely using the cancel command.

Uploading: alpine2.tar
[==================================================] 100%
2222080/2222080 bytes
... completed

Importing into the repository...
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Succeeded

Deleting the upload request...
... completed

[ipanova@ina pulp_docker]$ pulp-admin docker repo uploads upload --repo-id alpine --file ./alpine1.tar
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: alpine1.tar
... completed

Creating upload requests on the server...
[==================================================] 100%
Initializing: alpine1.tar
... completed

Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
canceled entirely using the cancel command.

Uploading: alpine1.tar
[==================================================] 100%
2222080/2222080 bytes
... completed

Importing into the repository...

This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Succeeded

Deleting the upload request...
... completed
$ pulp-admin docker repo list --repo-id alpine
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

+----------------------------------------------------------------------+
                          Docker Repositories
+----------------------------------------------------------------------+

Id:                   alpine
Display Name:         None
Description:          None
Content Unit Counts:  
  Docker Blob:     3
  Docker Manifest: 2

[ipanova@ina pulp_docker]$ ll alpine*
-rw-rw-r--. 1 ipanova ipanova 2222080 Nov  2 16:11 alpine1.tar
-rw-rw-r--. 1 ipanova ipanova 2222080 Nov  2 16:11 alpine2.tar

I;d wait till the next triage in case the reporter would provide more info, otherwise i'd close this issue as works for me.

I was able to repro this on 2.17.1 and 2.19 nightly. I created a tar with "skopeo copy --format v2s1 docker://busybox dir:/tmp/busybox; cd /tmp/busybox; tar cf busybox.tar *", then tried to upload the tar and it failed with the same message.

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

Applied in changeset 9375ec84064b96b576250f6bc3cbbbb358f1cfae.

Applied in changeset pulp_docker|38fd8eef4e12fd46364ed5382f2d80602087c702.