Project

Profile

Help

Issue #4095

closed

Can't import containers created with Buildah

Added by hc over 5 years ago. Updated about 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version - Docker:
Platform Release:
2.21.1
Target Release - Docker:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 61
Quarter:

Description

Hi!

I tried the tool Buildah (https://www.projectatomic.io/blog/2017/06/introducing-buildah/) to build containers without having docker binary. Containers are built (not sure if they work), but I can't upload them to the pulp repo:

1) I copy container from "container-storage:" to "dir:", then create tar archive per https://docs.pulpproject.org/plugins/pulp_docker/user-guide/recipes.html#upload-v2-schema-2-and-schema-1-images-to-pulp
2) pulp-admin -vvv docker repo uploads upload --repo-id docker_centos7 -f centos7.tar

[2] results in:

  "traceback": "Traceback (most recent call last):\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 529, in __call__\n    return super(Task, self).__call__(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 107, in __call__\n    return super(PulpTask, self).__call__(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in __protected_call__\n    return self.run(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py\", line 223, in import_uploaded_unit\n    unit_type=unit_type_id, summary=result['summary'], details=result['details']\nPulpCodedException: The importer docker_importer indicated a failed response when uploading docker_manifest unit to repository docker_centos7.\n", 

  "error": {
    "code": "PLP0047", 
    "data": {
      "unit_type": "docker_manifest", 
      "importer_id": "docker_importer", 
      "repo_id": "docker_centos7", 
      "details": {}, 
      "summary": [
        "Cannot resolve field \"digest\""
      ]
    }, 
    "description": "The importer docker_importer indicated a failed response when uploading docker_manifest unit to repository docker_centos7.",

I tried both v2s2 and v2s1, both don't work. I tried 2 different Dockerfiles and both produce the same error.

I have the following Pulp versions installed:

pulp-admin-client-2.17.1-1.el7.noarch
pulp-agent-2.17.1-1.el7.noarch
pulp-consumer-client-2.17.1-1.el7.noarch
pulp-deb-admin-extensions-1.8.0-1.el7.noarch
pulp-deb-plugins-1.8.0-1.el7.noarch
pulp-docker-admin-extensions-3.2.1-1.el7.noarch
pulp-docker-plugins-3.2.1-1.el7.noarch
pulp-puppet-consumer-extensions-2.17.1-1.el7.noarch
pulp-puppet-handlers-2.17.1-1.el7.noarch
pulp-python-admin-extensions-2.0.3-1.el7.noarch
pulp-python-plugins-2.0.3-1.el7.noarch
pulp-rpm-admin-extensions-2.17.1-1.el7.noarch
pulp-rpm-consumer-extensions-2.17.1-1.el7.noarch
pulp-rpm-handlers-2.17.1-1.el7.noarch
pulp-rpm-plugins-2.17.1-1.el7.noarch
pulp-rpm-yumplugins-2.17.1-1.el7.noarch
pulp-selinux-2.17.1-1.el7.noarch
pulp-server-2.17.1-1.el7.noarch
python-pulp-agent-lib-2.17.1-1.el7.noarch
python-pulp-bindings-2.17.1-1.el7.noarch
python-pulp-client-lib-2.17.1-1.el7.noarch
python-pulp-common-2.17.1-1.el7.noarch
python-pulp-deb-common-1.8.0-1.el7.noarch
python-pulp-docker-common-3.2.1-1.el7.noarch
python-pulp-oid_validation-2.17.1-1.el7.noarch
python-pulp-puppet-common-2.17.1-1.el7.noarch
python-pulp-python-common-2.0.3-1.el7.noarch
python-pulp-repoauth-2.17.1-1.el7.noarch
python-pulp-rpm-common-2.17.1-1.el7.noarch
buildah-1.2-2.gitbe87762.el7.x86_64

Manifest v2s1 from one of the images created by buildah:

{'architecture': 'amd64',
 'fsLayers': [{'blobSum': 'sha256:4d85b6fa2344380743db8ccf4f0404603f56c8289db0824c13bfc309b2009be8'},
              {'blobSum': 'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'},
              {'blobSum': 'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'},
              {'blobSum': 'sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f'}],
 'history': [{'v1Compatibility': '{"architecture":"amd64","config":{"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Entrypoint":["/bin/bash"],"WorkingDir":"/","Labels":{"org.label-schema.build-date":"20181006","org.label-schema.license":"GPLv2","org.label-schema.name":"CentOS Base Image","org.label-schema.schema-version":"1.0","org.label-schema.vendor":"CentOS"}},"created":"2018-10-20T07:49:56.084883454Z","id":"4591c91f4c223fa61c800a84bcf711647786f3fe71e589606f17b4d11b9c5807","os":"linux","parent":"f871460fb057cfa08e7268df9b884eecedeed72bc6dbc3262deb19a25e66a904"}'},
             {'v1Compatibility': '{"id":"f871460fb057cfa08e7268df9b884eecedeed72bc6dbc3262deb19a25e66a904","parent":"8a4f409d848e6f693d5a4e6c260fbd8ec6a4dc99fc9915e3210bd5ed9c22e700","created":"2018-10-09T18:19:48.447478476Z","container_config":{"Cmd":["/bin/sh -c #(nop)  CMD [\\"/bin/bash\\"]"]},"throwaway":true}'},
             {'v1Compatibility': '{"id":"8a4f409d848e6f693d5a4e6c260fbd8ec6a4dc99fc9915e3210bd5ed9c22e700","parent":"961164371d3af72a151063f656820bc835c695fa44af66c2845d0850f01ce0d9","created":"2018-10-09T18:19:48.258784779Z","container_config":{"Cmd":["/bin/sh -c #(nop)  LABEL org.label-schema.schema-version=1.0 org.label-schema.name=CentOS Base Image org.label-schema.vendor=CentOS org.label-schema.license=GPLv2 org.label-schema.build-date=20181006"]},"throwaway":true}'},
             {'v1Compatibility': '{"id":"961164371d3af72a151063f656820bc835c695fa44af66c2845d0850f01ce0d9","created":"2018-10-09T18:19:47.884079751Z","container_config":{"Cmd":["/bin/sh -c #(nop) ADD file:fbe9badfd2790f0747a25fbe5c94a6daa78969511ca08c8d4ac654f3442570de in / "]}}'}],
 'name': '',
 'schemaVersion': 1,
 'signatures': [{'header': {'alg': 'ES256',
                            'jwk': {'crv': 'P-256',
                                    'kid': '...',
                                    'kty': 'EC',
                                    'x': '...',
                                    'y': '...'}},
                 'protected': '...',
                 'signature': '...'}],
 'tag': ''}

Manifest v2s2 for the same image:

{'config': {'digest': 'sha256:c17d63d23edcbe46b14fbe531e2e9b4411483947669087e90bdba3372eb2684d',
            'mediaType': 'application/vnd.docker.container.image.v1+json',
            'size': 1217},
 'layers': [{'digest': 'sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f',
             'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
             'size': 208811520},
            {'digest': 'sha256:4d85b6fa2344380743db8ccf4f0404603f56c8289db0824c13bfc309b2009be8',
             'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
             'size': 589285888}],
 'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
 'schemaVersion': 2}

Is it an issue in Buildah or in Pulp? Or am I doing something wrong?

Thanks!


Files

docker_upload.log (37.9 KB) docker_upload.log akofink, 11/02/2018 03:17 PM
Actions #1

Updated by hc over 5 years ago

Found https://pulp.plan.io/issues/3896, which indeed made getting to an actual error message more complex.

Actions #2

Updated by hc over 5 years ago

For comparison, here is the s2v2 manifest generated by skopeo from the same container build by docker:

{'config': {'digest': 'sha256:80df321f2ea1b2a5b829a49735755198e7442aa41a4103283e6e4f9ebe9a335e',
            'mediaType': 'application/vnd.docker.container.image.v1+json',
            'size': 3002},
 'layers': [{'digest': 'sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f',
             'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
             'size': 208811520},
            {'digest': 'sha256:b48263b07da508175cef044d90311cde227a8a7330f5c7cac423977c40525043',
             'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
             'size': 5120},
            {'digest': 'sha256:f365f9181415ead2c548882d7d6cca35d3610210ec3b8c970871cc56863fa1e4',
             'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
             'size': 3584},
            {'digest': 'sha256:fc55bc91557a901996323b53397c9e7b823b619e07786032d61b1328c94e7309',
             'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
             'size': 6656}],
 'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
 'schemaVersion': 2}

And this container also caused the same error on upload.

Actions #3

Updated by ipanova@redhat.com over 5 years ago

I was not able to reproduce this. I tried skopeo copy same as buildah push.

$ sudo buildah images
IMAGE ID             IMAGE NAME                                               CREATED AT             SIZE
2cc378c061f7         docker.io/library/python:latest                          Oct 25, 2018 00:36     949 MB
0ede40d5477c         localhost/hellofromcontainer:latest                      Oct 30, 2018 12:38     949 MB

[ipanova@ina tmp]$ sudo buildah push --f v2s2 0ede40d5477c dir:/tmp/tada

$ cd /tmp/tada/ && tar -cvf ../tada.tar * && cd ..
pulp-admin -vv docker repo uploads upload --repo-id madness --file ./tada.tar
2018-10-30 12:57:08,243 - INFO - GET request to /pulp/api/v2/tasks/3067d69b-a2b2-465d-bad4-3d87b819d602/ with parameters None
2018-10-30 12:57:08,243 - INFO - Response status : 200 

2018-10-30 12:57:08,243 - INFO - Response body :
 {
  "exception": null, 
  "task_type": "pulp.server.managers.content.upload.import_uploaded_unit", 
  "_href": "/pulp/api/v2/tasks/3067d69b-a2b2-465d-bad4-3d87b819d602/", 
  "task_id": "3067d69b-a2b2-465d-bad4-3d87b819d602", 
  "tags": [
    "pulp:repository:madness", 
    "pulp:action:import_upload"
  ], 
  "finish_time": "2018-10-30T11:57:08Z", 
  "_ns": "task_status", 
  "start_time": "2018-10-30T11:56:39Z", 
  "traceback": null, 
  "spawned_tasks": [], 
  "progress_report": {}, 
  "queue": "reserved_resource_worker-0.dq2", 
  "state": "finished", 
  "worker_name": "reserved_resource_worker-0", 
  "result": {
    "details": {
      "unit": {
        "type_id": "docker_manifest", 
        "unit_key": {
          "digest": "sha256:b6c910a91860f695da46c73ebd784653d6eee3767a4f8f646a04c9b6f084f7f7"
        }, 
        "metadata": {
          "fs_layers": [
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 105529344, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:f715ed19c28b66943ac8bc12dbfb828e8394de2530bbaf1ecce906e748e4fdff"
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 23989248, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:8bb25f9cdc41e7d085033af15a522973b44086d6eedd24c11cc61c9232324f77"
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 7994368, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:08a01612ffca33483a1847c909836610610ce523fb7e1aca880140ee84df23e9"
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 146405888, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:1191b3f5862aa9231858809b7ac8b91c0b727ce85c9b3279932f0baacc92967d"
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 570565632, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:9978d084fd771e0b3d1acd7f3525d1b25288ababe9ad8ed259b36101e4e3addd"
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 17495552, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:d62f0ea9a15e2a05779741bff6654624b7dbd6c12a7292205c81bee0ff53f9a1"
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 70766080, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:25465864233d59ce8667ffe16e09780ea7e38f99f9c39d33a5263f5613f2a3fe"
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 4608, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:a474157cdf60afc9e274c9b529827a06103dab9ba9b15c577da1dc1cd82083d4"
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 6341632, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:7f8f0e833e519f17b9e42f232d4bcd561939a6697f401b1bdc7af26b0657bf6b"
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 2048, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:f9ee991634f497ecf188e9034b3dfb58d98f456459ef2b392ffb9525b4f9d350"
            }
          ], 
          "schema_version": 2, 
          "downloaded": true, 
          "pulp_user_metadata": {}, 
          "config_layer": "sha256:0ede40d5477c25a9b4cd7a9bbe96e629c90da581ae49d1196606d21371642b3a", 
          "id": "076b91f3-e99c-4b97-91e9-e25249e0e73e", 
          "digest": "sha256:b6c910a91860f695da46c73ebd784653d6eee3767a4f8f646a04c9b6f084f7f7"
        }
      }
    }, 
    "success_flag": true, 
    "summary": ""
  }, 
  "error": null, 
  "_id": {
    "$oid": "5bd846f7ce1b25e899d9fcb4"
  }, 
  "id": "5bd846f7ce1b25e899d9fcb4"
[\]
Running...

Task Succeeded

Please provide specific steps to reproduce. I am new buildah so maybe i missed something.

Actions #4

Updated by akofink over 5 years ago

I can reproduce this with the following steps (see attached log):

  1. Create an empty docker repo in pulp
  2. Use skopeo to copy a v2s2 and a v2s1 image to directories, and tar the contents (following pulp's docs)
  3. Use pulp-admin to upload the v2s2 tar
  4. Use pulp-admin to upload the v2s1 tar. This is when the error occurs.
Actions #5

Updated by CodeHeeler over 5 years ago

  • Triaged changed from No to Yes
Actions #6

Updated by ipanova@redhat.com over 5 years ago

cannot reproduce, followed the mentioned steps above ^

[ipanova@ina pulp_docker]$ pulp-admin docker repo create --repo-id alpine
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

Repository [alpine] successfully created

$ pulp-admin docker repo uploads upload --repo-id alpine --file ./alpine2.tar
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: alpine2.tar
... completed

Creating upload requests on the server...
[==================================================] 100%
Initializing: alpine2.tar
... completed

Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
canceled entirely using the cancel command.

Uploading: alpine2.tar
[==================================================] 100%
2222080/2222080 bytes
... completed

Importing into the repository...
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Succeeded

Deleting the upload request...
... completed

[ipanova@ina pulp_docker]$ pulp-admin docker repo uploads upload --repo-id alpine --file ./alpine1.tar
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: alpine1.tar
... completed

Creating upload requests on the server...
[==================================================] 100%
Initializing: alpine1.tar
... completed

Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
canceled entirely using the cancel command.

Uploading: alpine1.tar
[==================================================] 100%
2222080/2222080 bytes
... completed

Importing into the repository...

This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Succeeded

Deleting the upload request...
... completed
$ pulp-admin docker repo list --repo-id alpine
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

+----------------------------------------------------------------------+
                          Docker Repositories
+----------------------------------------------------------------------+

Id:                   alpine
Display Name:         None
Description:          None
Content Unit Counts:  
  Docker Blob:     3
  Docker Manifest: 2

[ipanova@ina pulp_docker]$ ll alpine*
-rw-rw-r--. 1 ipanova ipanova 2222080 Nov  2 16:11 alpine1.tar
-rw-rw-r--. 1 ipanova ipanova 2222080 Nov  2 16:11 alpine2.tar
Actions #7

Updated by ipanova@redhat.com over 5 years ago

I;d wait till the next triage in case the reporter would provide more info, otherwise i'd close this issue as works for me.

Actions #8

Updated by cduryee over 5 years ago

I was able to repro this on 2.17.1 and 2.19 nightly. I created a tar with "skopeo copy --format v2s1 docker://busybox dir:/tmp/busybox; cd /tmp/busybox; tar cf busybox.tar *", then tried to upload the tar and it failed with the same message.

Actions #9

Updated by bmbouter almost 5 years ago

  • Status changed from NEW to CLOSED - WONTFIX
Actions #10

Updated by bmbouter almost 5 years ago

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

Actions #11

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added
Actions #12

Updated by ipanova@redhat.com over 4 years ago

  • Status changed from CLOSED - WONTFIX to NEW
  • Assignee set to ipanova@redhat.com

steps to reproduce:
1 create a repo and populate it with the content. Make sure there are tags present in the repo
2. Use skopeo to copy a v2s2 or v2s1 image to directories, and tar the content
3. Use pulp-admin to upload the tar. This is when the error occurs.

Actions #13

Updated by ipanova@redhat.com over 4 years ago

  • Status changed from NEW to POST
  • Sprint set to Sprint 59

https://github.com/pulp/pulp_docker/pull/418

Filtering by content_type_id was added, so only blobs are being searched when find_repo_content_units is called. Otherwise other types like tags, are being picked, and since tags do not have 'digest' field defined on the model it leads to the error seen in the bug report.

Actions #14

Updated by rchan over 4 years ago

  • Sprint changed from Sprint 59 to Sprint 60
Actions #15

Updated by rchan over 4 years ago

  • Sprint changed from Sprint 60 to Sprint 61
Actions #16

Updated by ipanova@redhat.com over 4 years ago

  • Status changed from POST to MODIFIED
Actions #18

Updated by ipanova@redhat.com about 4 years ago

  • Platform Release set to 2.21.1

Added by ipanova@redhat.com about 4 years ago

Revision 38fd8eef | View on GitHub

Fix upload of docker image

closes #4095 https://pulp.plan.io/issues/4095

(cherry picked from commit 9375ec84064b96b576250f6bc3cbbbb358f1cfae)

Added by ipanova@redhat.com about 4 years ago

Revision 38fd8eef | View on GitHub

Fix upload of docker image

closes #4095 https://pulp.plan.io/issues/4095

(cherry picked from commit 9375ec84064b96b576250f6bc3cbbbb358f1cfae)

Actions #20

Updated by ipanova@redhat.com about 4 years ago

  • Status changed from MODIFIED to 5
Actions #21

Updated by ipanova@redhat.com about 4 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

Also available in: Atom PDF