Project

Profile

Help

Issue #4095

Can't import containers created with Buildah

Added by hc about 1 year ago. Updated about 12 hours ago.

Status:
MODIFIED
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Severity:
2. Medium
Version - Docker:
Platform Release:
Blocks Release:
Target Release - Docker:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 61

Description

Hi!

I tried the tool Buildah (https://www.projectatomic.io/blog/2017/06/introducing-buildah/) to build containers without having docker binary. Containers are built (not sure if they work), but I can't upload them to the pulp repo:

1) I copy container from "container-storage:" to "dir:", then create tar archive per https://docs.pulpproject.org/plugins/pulp_docker/user-guide/recipes.html#upload-v2-schema-2-and-schema-1-images-to-pulp
2) pulp-admin -vvv docker repo uploads upload --repo-id docker_centos7 -f centos7.tar

[2] results in:

  "traceback": "Traceback (most recent call last):\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 529, in __call__\n    return super(Task, self).__call__(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 107, in __call__\n    return super(PulpTask, self).__call__(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in __protected_call__\n    return self.run(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py\", line 223, in import_uploaded_unit\n    unit_type=unit_type_id, summary=result['summary'], details=result['details']\nPulpCodedException: The importer docker_importer indicated a failed response when uploading docker_manifest unit to repository docker_centos7.\n", 

  "error": {
    "code": "PLP0047", 
    "data": {
      "unit_type": "docker_manifest", 
      "importer_id": "docker_importer", 
      "repo_id": "docker_centos7", 
      "details": {}, 
      "summary": [
        "Cannot resolve field \"digest\"" 
      ]
    }, 
    "description": "The importer docker_importer indicated a failed response when uploading docker_manifest unit to repository docker_centos7.",

I tried both v2s2 and v2s1, both don't work. I tried 2 different Dockerfiles and both produce the same error.

I have the following Pulp versions installed:

pulp-admin-client-2.17.1-1.el7.noarch
pulp-agent-2.17.1-1.el7.noarch
pulp-consumer-client-2.17.1-1.el7.noarch
pulp-deb-admin-extensions-1.8.0-1.el7.noarch
pulp-deb-plugins-1.8.0-1.el7.noarch
pulp-docker-admin-extensions-3.2.1-1.el7.noarch
pulp-docker-plugins-3.2.1-1.el7.noarch
pulp-puppet-consumer-extensions-2.17.1-1.el7.noarch
pulp-puppet-handlers-2.17.1-1.el7.noarch
pulp-python-admin-extensions-2.0.3-1.el7.noarch
pulp-python-plugins-2.0.3-1.el7.noarch
pulp-rpm-admin-extensions-2.17.1-1.el7.noarch
pulp-rpm-consumer-extensions-2.17.1-1.el7.noarch
pulp-rpm-handlers-2.17.1-1.el7.noarch
pulp-rpm-plugins-2.17.1-1.el7.noarch
pulp-rpm-yumplugins-2.17.1-1.el7.noarch
pulp-selinux-2.17.1-1.el7.noarch
pulp-server-2.17.1-1.el7.noarch
python-pulp-agent-lib-2.17.1-1.el7.noarch
python-pulp-bindings-2.17.1-1.el7.noarch
python-pulp-client-lib-2.17.1-1.el7.noarch
python-pulp-common-2.17.1-1.el7.noarch
python-pulp-deb-common-1.8.0-1.el7.noarch
python-pulp-docker-common-3.2.1-1.el7.noarch
python-pulp-oid_validation-2.17.1-1.el7.noarch
python-pulp-puppet-common-2.17.1-1.el7.noarch
python-pulp-python-common-2.0.3-1.el7.noarch
python-pulp-repoauth-2.17.1-1.el7.noarch
python-pulp-rpm-common-2.17.1-1.el7.noarch

buildah-1.2-2.gitbe87762.el7.x86_64

Manifest v2s1 from one of the images created by buildah:

{'architecture': 'amd64',
 'fsLayers': [{'blobSum': 'sha256:4d85b6fa2344380743db8ccf4f0404603f56c8289db0824c13bfc309b2009be8'},
              {'blobSum': 'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'},
              {'blobSum': 'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'},
              {'blobSum': 'sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f'}],
 'history': [{'v1Compatibility': '{"architecture":"amd64","config":{"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Entrypoint":["/bin/bash"],"WorkingDir":"/","Labels":{"org.label-schema.build-date":"20181006","org.label-schema.license":"GPLv2","org.label-schema.name":"CentOS Base Image","org.label-schema.schema-version":"1.0","org.label-schema.vendor":"CentOS"}},"created":"2018-10-20T07:49:56.084883454Z","id":"4591c91f4c223fa61c800a84bcf711647786f3fe71e589606f17b4d11b9c5807","os":"linux","parent":"f871460fb057cfa08e7268df9b884eecedeed72bc6dbc3262deb19a25e66a904"}'},
             {'v1Compatibility': '{"id":"f871460fb057cfa08e7268df9b884eecedeed72bc6dbc3262deb19a25e66a904","parent":"8a4f409d848e6f693d5a4e6c260fbd8ec6a4dc99fc9915e3210bd5ed9c22e700","created":"2018-10-09T18:19:48.447478476Z","container_config":{"Cmd":["/bin/sh -c #(nop)  CMD [\\"/bin/bash\\"]"]},"throwaway":true}'},
             {'v1Compatibility': '{"id":"8a4f409d848e6f693d5a4e6c260fbd8ec6a4dc99fc9915e3210bd5ed9c22e700","parent":"961164371d3af72a151063f656820bc835c695fa44af66c2845d0850f01ce0d9","created":"2018-10-09T18:19:48.258784779Z","container_config":{"Cmd":["/bin/sh -c #(nop)  LABEL org.label-schema.schema-version=1.0 org.label-schema.name=CentOS Base Image org.label-schema.vendor=CentOS org.label-schema.license=GPLv2 org.label-schema.build-date=20181006"]},"throwaway":true}'},
             {'v1Compatibility': '{"id":"961164371d3af72a151063f656820bc835c695fa44af66c2845d0850f01ce0d9","created":"2018-10-09T18:19:47.884079751Z","container_config":{"Cmd":["/bin/sh -c #(nop) ADD file:fbe9badfd2790f0747a25fbe5c94a6daa78969511ca08c8d4ac654f3442570de in / "]}}'}],
 'name': '',
 'schemaVersion': 1,
 'signatures': [{'header': {'alg': 'ES256',
                            'jwk': {'crv': 'P-256',
                                    'kid': '...',
                                    'kty': 'EC',
                                    'x': '...',
                                    'y': '...'}},
                 'protected': '...',
                 'signature': '...'}],
 'tag': ''}

Manifest v2s2 for the same image:

{'config': {'digest': 'sha256:c17d63d23edcbe46b14fbe531e2e9b4411483947669087e90bdba3372eb2684d',
            'mediaType': 'application/vnd.docker.container.image.v1+json',
            'size': 1217},
 'layers': [{'digest': 'sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f',
             'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
             'size': 208811520},
            {'digest': 'sha256:4d85b6fa2344380743db8ccf4f0404603f56c8289db0824c13bfc309b2009be8',
             'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
             'size': 589285888}],
 'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
 'schemaVersion': 2}

Is it an issue in Buildah or in Pulp? Or am I doing something wrong?

Thanks!

docker_upload.log (37.9 KB) akofink, 11/02/2018 03:17 PM docker_upload.log

History

#1 Updated by hc about 1 year ago

Found https://pulp.plan.io/issues/3896, which indeed made getting to an actual error message more complex.

#2 Updated by hc about 1 year ago

For comparison, here is the s2v2 manifest generated by skopeo from the same container build by docker:

{'config': {'digest': 'sha256:80df321f2ea1b2a5b829a49735755198e7442aa41a4103283e6e4f9ebe9a335e',
            'mediaType': 'application/vnd.docker.container.image.v1+json',
            'size': 3002},
 'layers': [{'digest': 'sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f',
             'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
             'size': 208811520},
            {'digest': 'sha256:b48263b07da508175cef044d90311cde227a8a7330f5c7cac423977c40525043',
             'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
             'size': 5120},
            {'digest': 'sha256:f365f9181415ead2c548882d7d6cca35d3610210ec3b8c970871cc56863fa1e4',
             'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
             'size': 3584},
            {'digest': 'sha256:fc55bc91557a901996323b53397c9e7b823b619e07786032d61b1328c94e7309',
             'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',
             'size': 6656}],
 'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',
 'schemaVersion': 2}

And this container also caused the same error on upload.

#3 Updated by ipanova@redhat.com about 1 year ago

I was not able to reproduce this. I tried skopeo copy same as buildah push.

$ sudo buildah images
IMAGE ID             IMAGE NAME                                               CREATED AT             SIZE
2cc378c061f7         docker.io/library/python:latest                          Oct 25, 2018 00:36     949 MB
0ede40d5477c         localhost/hellofromcontainer:latest                      Oct 30, 2018 12:38     949 MB

[ipanova@ina tmp]$ sudo buildah push --f v2s2 0ede40d5477c dir:/tmp/tada

$ cd /tmp/tada/ && tar -cvf ../tada.tar * && cd ..
pulp-admin -vv docker repo uploads upload --repo-id madness --file ./tada.tar
2018-10-30 12:57:08,243 - INFO - GET request to /pulp/api/v2/tasks/3067d69b-a2b2-465d-bad4-3d87b819d602/ with parameters None
2018-10-30 12:57:08,243 - INFO - Response status : 200 

2018-10-30 12:57:08,243 - INFO - Response body :
 {
  "exception": null, 
  "task_type": "pulp.server.managers.content.upload.import_uploaded_unit", 
  "_href": "/pulp/api/v2/tasks/3067d69b-a2b2-465d-bad4-3d87b819d602/", 
  "task_id": "3067d69b-a2b2-465d-bad4-3d87b819d602", 
  "tags": [
    "pulp:repository:madness", 
    "pulp:action:import_upload" 
  ], 
  "finish_time": "2018-10-30T11:57:08Z", 
  "_ns": "task_status", 
  "start_time": "2018-10-30T11:56:39Z", 
  "traceback": null, 
  "spawned_tasks": [], 
  "progress_report": {}, 
  "queue": "reserved_resource_worker-0.dq2", 
  "state": "finished", 
  "worker_name": "reserved_resource_worker-0", 
  "result": {
    "details": {
      "unit": {
        "type_id": "docker_manifest", 
        "unit_key": {
          "digest": "sha256:b6c910a91860f695da46c73ebd784653d6eee3767a4f8f646a04c9b6f084f7f7" 
        }, 
        "metadata": {
          "fs_layers": [
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 105529344, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:f715ed19c28b66943ac8bc12dbfb828e8394de2530bbaf1ecce906e748e4fdff" 
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 23989248, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:8bb25f9cdc41e7d085033af15a522973b44086d6eedd24c11cc61c9232324f77" 
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 7994368, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:08a01612ffca33483a1847c909836610610ce523fb7e1aca880140ee84df23e9" 
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 146405888, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:1191b3f5862aa9231858809b7ac8b91c0b727ce85c9b3279932f0baacc92967d" 
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 570565632, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:9978d084fd771e0b3d1acd7f3525d1b25288ababe9ad8ed259b36101e4e3addd" 
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 17495552, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:d62f0ea9a15e2a05779741bff6654624b7dbd6c12a7292205c81bee0ff53f9a1" 
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 70766080, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:25465864233d59ce8667ffe16e09780ea7e38f99f9c39d33a5263f5613f2a3fe" 
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 4608, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:a474157cdf60afc9e274c9b529827a06103dab9ba9b15c577da1dc1cd82083d4" 
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 6341632, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:7f8f0e833e519f17b9e42f232d4bcd561939a6697f401b1bdc7af26b0657bf6b" 
            }, 
            {
              "layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip", 
              "size": 2048, 
              "_cls": "FSLayer", 
              "blob_sum": "sha256:f9ee991634f497ecf188e9034b3dfb58d98f456459ef2b392ffb9525b4f9d350" 
            }
          ], 
          "schema_version": 2, 
          "downloaded": true, 
          "pulp_user_metadata": {}, 
          "config_layer": "sha256:0ede40d5477c25a9b4cd7a9bbe96e629c90da581ae49d1196606d21371642b3a", 
          "id": "076b91f3-e99c-4b97-91e9-e25249e0e73e", 
          "digest": "sha256:b6c910a91860f695da46c73ebd784653d6eee3767a4f8f646a04c9b6f084f7f7" 
        }
      }
    }, 
    "success_flag": true, 
    "summary": "" 
  }, 
  "error": null, 
  "_id": {
    "$oid": "5bd846f7ce1b25e899d9fcb4" 
  }, 
  "id": "5bd846f7ce1b25e899d9fcb4" 
[\]
Running...

Task Succeeded

Please provide specific steps to reproduce. I am new buildah so maybe i missed something.

#4 Updated by akofink about 1 year ago

I can reproduce this with the following steps (see attached log):

  1. Create an empty docker repo in pulp
  2. Use skopeo to copy a v2s2 and a v2s1 image to directories, and tar the contents (following pulp's docs)
  3. Use pulp-admin to upload the v2s2 tar
  4. Use pulp-admin to upload the v2s1 tar. This is when the error occurs.

#5 Updated by CodeHeeler about 1 year ago

  • Triaged changed from No to Yes

#6 Updated by ipanova@redhat.com about 1 year ago

cannot reproduce, followed the mentioned steps above ^

[ipanova@ina pulp_docker]$ pulp-admin docker repo create --repo-id alpine
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

Repository [alpine] successfully created

$ pulp-admin docker repo uploads upload --repo-id alpine --file ./alpine2.tar
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: alpine2.tar
... completed

Creating upload requests on the server...
[==================================================] 100%
Initializing: alpine2.tar
... completed

Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
canceled entirely using the cancel command.

Uploading: alpine2.tar
[==================================================] 100%
2222080/2222080 bytes
... completed

Importing into the repository...
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Succeeded

Deleting the upload request...
... completed

[ipanova@ina pulp_docker]$ pulp-admin docker repo uploads upload --repo-id alpine --file ./alpine1.tar
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: alpine1.tar
... completed

Creating upload requests on the server...
[==================================================] 100%
Initializing: alpine1.tar
... completed

Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
canceled entirely using the cancel command.

Uploading: alpine1.tar
[==================================================] 100%
2222080/2222080 bytes
... completed

Importing into the repository...

This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Succeeded

Deleting the upload request...
... completed
$ pulp-admin docker repo list --repo-id alpine
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

+----------------------------------------------------------------------+
                          Docker Repositories
+----------------------------------------------------------------------+

Id:                   alpine
Display Name:         None
Description:          None
Content Unit Counts:  
  Docker Blob:     3
  Docker Manifest: 2

[ipanova@ina pulp_docker]$ ll alpine*
-rw-rw-r--. 1 ipanova ipanova 2222080 Nov  2 16:11 alpine1.tar
-rw-rw-r--. 1 ipanova ipanova 2222080 Nov  2 16:11 alpine2.tar

#7 Updated by ipanova@redhat.com about 1 year ago

I;d wait till the next triage in case the reporter would provide more info, otherwise i'd close this issue as works for me.

#8 Updated by cduryee about 1 year ago

I was able to repro this on 2.17.1 and 2.19 nightly. I created a tar with "skopeo copy --format v2s1 docker://busybox dir:/tmp/busybox; cd /tmp/busybox; tar cf busybox.tar *", then tried to upload the tar and it failed with the same message.

#9 Updated by bmbouter 7 months ago

  • Status changed from NEW to CLOSED - WONTFIX

#10 Updated by bmbouter 7 months ago

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

#11 Updated by bmbouter 7 months ago

  • Tags Pulp 2 added

#12 Updated by ipanova@redhat.com about 2 months ago

  • Status changed from CLOSED - WONTFIX to NEW
  • Assignee set to ipanova@redhat.com

steps to reproduce:
1 create a repo and populate it with the content. Make sure there are tags present in the repo
2. Use skopeo to copy a v2s2 or v2s1 image to directories, and tar the content
3. Use pulp-admin to upload the tar. This is when the error occurs.

#13 Updated by ipanova@redhat.com about 2 months ago

  • Status changed from NEW to POST
  • Sprint set to Sprint 59

https://github.com/pulp/pulp_docker/pull/418

Filtering by content_type_id was added, so only blobs are being searched when find_repo_content_units is called. Otherwise other types like tags, are being picked, and since tags do not have 'digest' field defined on the model it leads to the error seen in the bug report.

#14 Updated by rchan about 2 months ago

  • Sprint changed from Sprint 59 to Sprint 60

#15 Updated by rchan 26 days ago

  • Sprint changed from Sprint 60 to Sprint 61

#16 Updated by ipanova@redhat.com 20 days ago

  • Status changed from POST to MODIFIED

Please register to edit this issue

Also available in: Atom PDF