Issue #4328: fsLayers error during sync - Docker Support - Pulp

Actions

Send by e-mail Copy link

Issue #4328

closed

fsLayers error during sync

Added by EppO over 5 years ago. Updated about 5 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

Normal

Assignee:

ipanova@redhat.com

Sprint/Milestone:

Pulp - 2.19.0

Start date:

Due date:

Estimated time:

Severity:

2. Medium

Version - Docker:

Platform Release:

2.19.0

Target Release - Docker:

OS:

CentOS 7

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Pulp 2

Sprint:

Sprint 48

Quarter:

Description

Hello,

trying to mirror gcr.io/google-containers/etcd, I encountered that issue during a sync:

2019-01-15 12:54:13,606 - INFO - Response body :
 {
  "exception": null,
  "task_type": "pulp.server.managers.repo.sync.sync",
  "_href": "/pulp/api/v2/tasks/291a8d9d-4b84-4aeb-af0f-2772e8ade987/",
  "task_id": "291a8d9d-4b84-4aeb-af0f-2772e8ade987",
  "tags": [
    "pulp:repository:etcd-gcr2",
    "pulp:action:sync"
  ],
  "finish_time": "2019-01-15T20:54:12Z",
  "_ns": "task_status",
  "start_time": "2019-01-15T20:54:09Z",
  "traceback": "Traceback (most recent call last):\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 529, in __call__\n    return super(Task, self).__call__(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 107, in __call__\n    return super(PulpTask, self).__call__(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in __protected_call__\n    return self.run(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 770, in sync\n    sync_report = sync_repo(transfer_repo, conduit, call_config)\n  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 737, in wrap_f\n    return f(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py\", line 85, in sync_repo\n    return self.sync_step.process_lifecycle()\n  File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 572, in process_lifecycle\n    super(PluginStep, self).process_lifecycle()\n  File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 163, in process_lifecycle\n    step.process()\n  File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 256, in process\n    self._process_block()\n  File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 303, in _process_block\n    self.process_main()\n  File \"/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py\", line 237, in process_main\n    self._process_manifest_list(manifest, digest, available_blobs, tag)\n  File \"/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py\", line 283, in _process_manifest_list\n    self._process_manifest(manifest, digest, available_blobs, tag=tag)\n  File \"/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py\", line 313, in _process_manifest\n    manifest = models.Manifest.from_json(manifest, digest)\n  File \"/usr/lib/python2.7/site-packages/pulp_docker/plugins/models.py\", line 219, in from_json\n    fs_layers = [FSLayer(blob_sum=layer['blobSum']) for layer in manifest['fsLayers']]\nKeyError: 'fsLayers'\n",
  "spawned_tasks": [],
  "progress_report": {
    "docker_importer": [
      {
        "num_success": 5,
        "description": "Downloading manifests",
        "step_type": "sync_step_metadata",
        "items_total": 1,
        "state": "FAILED",
        "error_details": [
          {
            "traceback": "  File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 256, in process\n    self._process_block()\n\n  File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 303, in _process_block\n    self.process_main()\n\n  File \"/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py\", line 237, in process_main\n    self._process_manifest_list(manifest, digest, available_blobs, tag)\n\n  File \"/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py\", line 283, in _process_manifest_list\n    self._process_manifest(manifest, digest, available_blobs, tag=tag)\n\n  File \"/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py\", line 313, in _process_manifest\n    manifest = models.Manifest.from_json(manifest, digest)\n\n  File \"/usr/lib/python2.7/site-packages/pulp_docker/plugins/models.py\", line 219, in from_json\n    fs_layers = [FSLayer(blob_sum=layer['blobSum']) for layer in manifest['fsLayers']]\n",
            "error": "'fsLayers'"
          }
        ],
        "details": "",
        "num_failures": 1,
        "step_id": "77747e24-bc58-46ed-b5a2-60c21ba5c5ad",
        "num_processed": 6
      },
      {
        "num_success": 0,
        "description": "Copying units already in pulp",
        "step_type": "get_local",
        "items_total": 1,
        "state": "NOT_STARTED",
        "error_details": [],
        "details": "",
        "num_failures": 0,
        "step_id": "695588b1-9004-4bce-ac6a-4f20bb1209c8",
        "num_processed": 0
      },
      {
        "num_success": 0,
        "description": "Copying units already in pulp",
        "step_type": "get_local",
        "items_total": 1,
        "state": "NOT_STARTED",
        "error_details": [],
        "details": "",
        "num_failures": 0,
        "step_id": "d211ca5e-bcf5-49df-a390-0e0016b4246f",
        "num_processed": 0
      },
      {
        "num_success": 0,
        "description": "Downloading remote files",
        "step_type": "sync_step_download",
        "items_total": 1,
        "state": "NOT_STARTED",
        "error_details": [],
        "details": "",
        "num_failures": 0,
        "step_id": "d627e410-d608-4a89-bec1-9709a1172821",
        "num_processed": 0
      },
      {
        "num_success": 0,
        "description": "Saving Manifests and Blobs",
        "step_type": "sync_step_save",
        "items_total": 1,
        "state": "NOT_STARTED",
        "error_details": [],
        "details": "",
        "num_failures": 0,
        "step_id": "23d120bd-01d3-40b8-ac6c-2cd434aed12f",
        "num_processed": 0
      },
      {
        "num_success": 0,
        "description": "Saving Tags",
        "step_type": "sync_step_save",
        "items_total": 1,
        "state": "NOT_STARTED",
        "error_details": [],
        "details": "",
        "num_failures": 0,
        "step_id": "97e64823-ef75-4406-ba71-fe7b8a081e61",
        "num_processed": 0
      }
    ]
  },

Changing the source repository to quay.io/etcd works though.
I tried deleting the repo in pulp and re-create it or whitelisting a single tag, no luck.

Here are the repository details:

Importers:
  Config:
    Feed:          https://gcr.io
    Tags:          3.2.24
    Upstream Name: google-containers/etcd
  Id:                   docker_importer
  Importer Type Id:     docker_importer
  Last Override Config:
  Last Sync:            2019-01-15T20:54:12Z
  Last Updated:         2019-01-15T20:53:58Z
  Repo Id:              etcd-gcr2
  Scratchpad:           None
Distributors:
  Auto Publish:         True
  Config:
    Repo-registry-id: google-containers/etcd2
  Distributor Type Id:  docker_distributor_web
  Id:                   docker_web_distributor_name_cli
  Last Override Config:
  Last Publish:         None
  Last Updated:         2019-01-15T20:53:58Z
  Repo Id:              etcd-gcr2
  Scratchpad:
  Auto Publish:         False
  Config:
    Repo-registry-id: google-containers/etcd2
  Distributor Type Id:  docker_distributor_export
  Id:                   docker_export_distributor_name_cli
  Last Override Config:
  Last Publish:         None
  Last Updated:         2019-01-15T20:53:58Z
  Repo Id:              etcd-gcr2

Actions

Copy link

Updated by ipanova@redhat.com over 5 years ago

can you please share the version of pulp and pulp_docker you have installed?thank you.

if it is older than 2.13 then i advise you to upgrade https://pulp.plan.io/issues/2099
reason why quay works, because it supports schema1

in your particular case, the tag you query goes not support schema1

$ curl -L -H "Accept: application/vnd.docker.distribution.manifest.v1+json"  -X GET 'https://gcr.io/v2/google-containers/etcd/manifests/3.2.24'
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"Manifest with tag '3.2.24' has media type 'application/vnd.docker.distribution.manifest.list.v2+json', but client accepts 'application/vnd.docker.distribution.manifest.v1+json'."}]}

Actions

Copy link

Updated by ttereshc over 5 years ago

Project changed from Pulp to Docker Support
Category changed from 15 to 17

Actions

Copy link

Updated by ttereshc over 5 years ago

Category deleted (17)

Actions

Copy link

Updated by ipanova@redhat.com over 5 years ago

Status changed from NEW to CLOSED - NOTABUG

please re-open in case c#1 does not cover your case.

Actions

Copy link

Updated by EppO about 5 years ago

Here is my pulp stack:

pulp-admin-client-2.18.0-2.el7.noarch
pulp-docker-admin-extensions-3.2.2-1.el7.noarch
pulp-docker-plugins-3.2.2-1.el7.noarch
pulp-nodes-admin-extensions-2.18.0-2.el7.noarch
pulp-nodes-common-2.18.0-2.el7.noarch
pulp-ostree-admin-extensions-1.3.1-1.el7.noarch
pulp-ostree-plugins-1.3.1-1.el7.noarch
pulp-puppet-admin-extensions-2.18.0-1.el7.noarch
pulp-puppet-plugins-2.18.0-1.el7.noarch
pulp-puppet-tools-2.18.0-1.el7.noarch
pulp-python-admin-extensions-2.0.3-1.el7.noarch
pulp-python-plugins-2.0.3-1.el7.noarch
pulp-rpm-admin-extensions-2.18.0-1.el7.noarch
pulp-rpm-plugins-2.18.0-1.el7.noarch
pulp-selinux-2.18.0-2.el7.noarch
pulp-server-2.18.0-2.el7.noarch
python-pulp-bindings-2.18.0-2.el7.noarch
python-pulp-client-lib-2.18.0-2.el7.noarch
python-pulp-common-2.18.0-2.el7.noarch
python-pulp-docker-common-3.2.2-1.el7.noarch
python-pulp-oid_validation-2.18.0-2.el7.noarch
python-pulp-ostree-common-1.3.1-1.el7.noarch
python-pulp-puppet-common-2.18.0-1.el7.noarch
python-pulp-python-common-2.0.3-1.el7.noarch
python-pulp-repoauth-2.18.0-2.el7.noarch
python-pulp-rpm-common-2.18.0-1.el7.noarch
python-pulp-streamer-2.18.0-2.el7.noarch

I find it surprising that gcr.io doesn't support schema v2. It works for other images on gcr.io

Actions

Copy link

Updated by EppO about 5 years ago

Can't reopen myself the issue, but it's still happening and I have latest release of pulp 2.x

Actions

Copy link

Updated by ipanova@redhat.com about 5 years ago

re-opened on request. We need to test and try to reproduce

@EppO
gcr as a registry does support schema1. It is some of the images which are stored only in schema2 format and afiak gcr does not do conversion on the fly of schema2-->schema1

Actions

Copy link

Updated by ipanova@redhat.com about 5 years ago

Status changed from CLOSED - NOTABUG to NEW

 pulp-admin docker repo create --repo-id=gcr --feed=https://gcr.io --upstream-name=google-containers/etcd
pulp-admin -vv docker  repo sync run --repo-id gcr

$ pulp-admin -vv tasks details --task-id 35522fa6-955f-4224-90ac-f7a2c976464a
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

+----------------------------------------------------------------------+
                              Task Details
+----------------------------------------------------------------------+

2019-02-08 15:40:02,767 - DEBUG - sending GET request to /pulp/api/v2/tasks/35522fa6-955f-4224-90ac-f7a2c976464a/
2019-02-08 15:40:02,862 - INFO - GET request to /pulp/api/v2/tasks/35522fa6-955f-4224-90ac-f7a2c976464a/ with parameters None
2019-02-08 15:40:02,862 - INFO - Response status : 200 

2019-02-08 15:40:02,862 - INFO - Response body :
 {
  "exception": null, 
  "task_type": "pulp.server.managers.repo.sync.sync", 
  "_href": "/pulp/api/v2/tasks/35522fa6-955f-4224-90ac-f7a2c976464a/", 
  "task_id": "35522fa6-955f-4224-90ac-f7a2c976464a", 
  "tags": [
    "pulp:repository:gcr", 
    "pulp:action:sync"
  ], 
  "finish_time": "2019-02-08T14:37:03Z", 
  "_ns": "task_status", 
  "start_time": "2019-02-08T14:36:16Z", 
  "traceback": "Traceback (most recent call last):\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File \"/home/ipanova/pulp_development/pulp/server/pulp/server/async/tasks.py\", line 529, in __call__\n    return super(Task, self).__call__(*args, **kwargs)\n  File \"/home/ipanova/pulp_development/pulp/server/pulp/server/async/tasks.py\", line 107, in __call__\n    return super(PulpTask, self).__call__(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in __protected_call__\n    return self.run(*args, **kwargs)\n  File \"/home/ipanova/pulp_development/pulp/server/pulp/server/controllers/repository.py\", line 770, in sync\n    sync_report = sync_repo(transfer_repo, conduit, call_config)\n  File \"/home/ipanova/pulp_development/pulp/server/pulp/server/async/tasks.py\", line 737, in wrap_f\n    return f(*args, **kwargs)\n  File \"/home/ipanova/pulp_development/pulp_docker/plugins/pulp_docker/plugins/importers/importer.py\", line 85, in sync_repo\n    return self.sync_step.process_lifecycle()\n  File \"/home/ipanova/pulp_development/pulp/server/pulp/plugins/util/publish_step.py\", line 572, in process_lifecycle\n    super(PluginStep, self).process_lifecycle()\n  File \"/home/ipanova/pulp_development/pulp/server/pulp/plugins/util/publish_step.py\", line 163, in process_lifecycle\n    step.process()\n  File \"/home/ipanova/pulp_development/pulp/server/pulp/plugins/util/publish_step.py\", line 256, in process\n    self._process_block()\n  File \"/home/ipanova/pulp_development/pulp/server/pulp/plugins/util/publish_step.py\", line 303, in _process_block\n    self.process_main()\n  File \"/home/ipanova/pulp_development/pulp_docker/plugins/pulp_docker/plugins/importers/sync.py\", line 237, in process_main\n    self._process_manifest_list(manifest, digest, available_blobs, tag)\n  File \"/home/ipanova/pulp_development/pulp_docker/plugins/pulp_docker/plugins/importers/sync.py\", line 283, in _process_manifest_list\n    self._process_manifest(manifest, digest, available_blobs, tag=tag)\n  File \"/home/ipanova/pulp_development/pulp_docker/plugins/pulp_docker/plugins/importers/sync.py\", line 313, in _process_manifest\n    manifest = models.Manifest.from_json(manifest, digest)\n  File \"/home/ipanova/pulp_development/pulp_docker/plugins/pulp_docker/plugins/models.py\", line 219, in from_json\n    fs_layers = [FSLayer(blob_sum=layer['blobSum']) for layer in manifest['fsLayers']]\nKeyError: 'fsLayers'\n", 
  "spawned_tasks": [], 
  "progress_report": {
    "docker_importer": [
      {
        "num_success": 100, 
        "description": "Downloading manifests", 
        "step_type": "sync_step_metadata", 
        "items_total": 71, 
        "state": "FAILED", 
        "error_details": [
          {
            "traceback": "  File \"/home/ipanova/pulp_development/pulp/server/pulp/plugins/util/publish_step.py\", line 256, in process\n    self._process_block()\n\n  File \"/home/ipanova/pulp_development/pulp/server/pulp/plugins/util/publish_step.py\", line 303, in _process_block\n    self.process_main()\n\n  File \"/home/ipanova/pulp_development/pulp_docker/plugins/pulp_docker/plugins/importers/sync.py\", line 237, in process_main\n    self._process_manifest_list(manifest, digest, available_blobs, tag)\n\n  File \"/home/ipanova/pulp_development/pulp_docker/plugins/pulp_docker/plugins/importers/sync.py\", line 283, in _process_manifest_list\n    self._process_manifest(manifest, digest, available_blobs, tag=tag)\n\n  File \"/home/ipanova/pulp_development/pulp_docker/plugins/pulp_docker/plugins/importers/sync.py\", line 313, in _process_manifest\n    manifest = models.Manifest.from_json(manifest, digest)\n\n  File \"/home/ipanova/pulp_development/pulp_docker/plugins/pulp_docker/plugins/models.py\", line 219, in from_json\n    fs_layers = [FSLayer(blob_sum=layer['blobSum']) for layer in manifest['fsLayers']]\n", 
            "error": "'fsLayers'"
          }
        ], 
        "details": "", 
        "num_failures": 1, 
        "step_id": "3358cefd-f518-44ac-b9e3-c62504a36bc4", 
        "num_processed": 101
      }, 
      {
        "num_success": 0, 
        "description": "Copying units already in pulp", 
        "step_type": "get_local", 
        "items_total": 1, 
        "state": "NOT_STARTED", 
        "error_details": [], 
        "details": "", 
        "num_failures": 0, 
        "step_id": "174fc073-2fe5-43d1-ab17-8aacc0edc617", 
        "num_processed": 0
      }, 
      {
        "num_success": 0, 
        "description": "Copying units already in pulp", 
        "step_type": "get_local", 
        "items_total": 1, 
        "state": "NOT_STARTED", 
        "error_details": [], 
        "details": "", 
        "num_failures": 0, 
        "step_id": "94ea98c6-e665-4b37-8a3c-e88a6cc9ae07", 
        "num_processed": 0
      }, 
      {
        "num_success": 0, 
        "description": "Downloading remote files", 
        "step_type": "sync_step_download", 
        "items_total": 1, 
        "state": "NOT_STARTED", 
        "error_details": [], 
        "details": "", 
        "num_failures": 0, 
        "step_id": "796fff61-a4b4-492c-9dca-e899f6265251", 
        "num_processed": 0
      }, 
      {
        "num_success": 0, 
        "description": "Saving Manifests and Blobs", 
        "step_type": "sync_step_save", 
        "items_total": 1, 
        "state": "NOT_STARTED", 
        "error_details": [], 
        "details": "", 
        "num_failures": 0, 
        "step_id": "b0ae0015-53f9-4315-8302-cc26ac6a93fd", 
        "num_processed": 0
      }, 
      {
        "num_success": 0, 
        "description": "Saving Tags", 
        "step_type": "sync_step_save", 
        "items_total": 1, 
        "state": "NOT_STARTED", 
        "error_details": [], 
        "details": "", 
        "num_failures": 0, 
        "step_id": "419fcd20-322e-4af6-a518-6199b224a0e5", 
        "num_processed": 0
      }
    ]
  }, 
  "queue": "reserved_resource_worker-0@ina.usersys.redhat.com.dq2", 
  "state": "error", 
  "worker_name": "reserved_resource_worker-0@ina.usersys.redhat.com", 
  "result": null, 
  "error": {
    "code": "PLP0000", 
    "data": {}, 
    "description": "'fsLayers'", 
    "sub_errors": []
  }, 
  "_id": {
    "$oid": "5c5d93e00c8a45468e11f8eb"
  }, 
  "id": "5c5d93e00c8a45468e11f8eb"
}

Operations:       sync
Resources:        gcr (repository)
State:            Failed
Start Time:       2019-02-08T14:36:16Z
Finish Time:      2019-02-08T14:37:03Z
Result:           N/A
Task Id:          35522fa6-955f-4224-90ac-f7a2c976464a
Worker Name:      reserved_resource_worker-0@ina.usersys.redhat.com
Progress Report:  
  Docker Importer: 
    Description:   Downloading manifests
    Details:       
    Error Details: 
      Error:     'fsLayers'
      Traceback:   File
                 "/home/ipanova/pulp_development/pulp/server/pulp/plugins/util/p
                 ublish_step.py", line 256, in process     self._process_block()
                 File
                 "/home/ipanova/pulp_development/pulp/server/pulp/plugins/util/p
                 ublish_step.py", line 303, in _process_block    
                 self.process_main()    File
                 "/home/ipanova/pulp_development/pulp_docker/plugins/pulp_docker
                 /plugins/importers/sync.py", line 237, in process_main    
                 self._process_manifest_list(manifest, digest, available_blobs,
                 tag)    File
                 "/home/ipanova/pulp_development/pulp_docker/plugins/pulp_docker
                 /plugins/importers/sync.py", line 283, in
                 _process_manifest_list     self._process_manifest(manifest,
                 digest, available_blobs, tag=tag)    File
                 "/home/ipanova/pulp_development/pulp_docker/plugins/pulp_docker
                 /plugins/importers/sync.py", line 313, in _process_manifest    
                 manifest = models.Manifest.from_json(manifest, digest)    File
                 "/home/ipanova/pulp_development/pulp_docker/plugins/pulp_docker
                 /plugins/models.py", line 219, in from_json     fs_layers =
                 [FSLayer(blob_sum=layer['blobSum']) for layer in
                 manifest['fsLayers']] 
    Items Total:   71
    Num Failures:  1
    Num Processed: 101
    Num Success:   100
    State:         FAILED
    Step Id:       3358cefd-f518-44ac-b9e3-c62504a36bc4
    Step Type:     sync_step_metadata
    Description:   Copying units already in pulp
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 0
    Num Success:   0
    State:         NOT_STARTED
    Step Id:       174fc073-2fe5-43d1-ab17-8aacc0edc617
    Step Type:     get_local
    Description:   Copying units already in pulp
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 0
    Num Success:   0
    State:         NOT_STARTED
    Step Id:       94ea98c6-e665-4b37-8a3c-e88a6cc9ae07
    Step Type:     get_local
    Description:   Downloading remote files
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 0
    Num Success:   0
    State:         NOT_STARTED
    Step Id:       796fff61-a4b4-492c-9dca-e899f6265251
    Step Type:     sync_step_download
    Description:   Saving Manifests and Blobs
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 0
    Num Success:   0
    State:         NOT_STARTED
    Step Id:       b0ae0015-53f9-4315-8302-cc26ac6a93fd
    Step Type:     sync_step_save
    Description:   Saving Tags
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 0
    Num Success:   0
    State:         NOT_STARTED
    Step Id:       419fcd20-322e-4af6-a518-6199b224a0e5
    Step Type:     sync_step_save
Traceback:        Traceback (most recent call last):   File
                  "/usr/lib/python2.7/site-packages/celery/app/trace.py", line
                  367, in trace_task     R = retval = fun(*args, **kwargs)  
                  File
                  "/home/ipanova/pulp_development/pulp/server/pulp/server/async/
                  tasks.py", line 529, in __call__     return super(Task,
                  self).__call__(*args, **kwargs)   File
                  "/home/ipanova/pulp_development/pulp/server/pulp/server/async/
                  tasks.py", line 107, in __call__     return super(PulpTask,
                  self).__call__(*args, **kwargs)   File
                  "/usr/lib/python2.7/site-packages/celery/app/trace.py", line
                  622, in __protected_call__     return self.run(*args,
                  **kwargs)   File
                  "/home/ipanova/pulp_development/pulp/server/pulp/server/contro
                  llers/repository.py", line 770, in sync     sync_report =
                  sync_repo(transfer_repo, conduit, call_config)   File
                  "/home/ipanova/pulp_development/pulp/server/pulp/server/async/
                  tasks.py", line 737, in wrap_f     return f(*args, **kwargs)  
                  File
                  "/home/ipanova/pulp_development/pulp_docker/plugins/pulp_docke
                  r/plugins/importers/importer.py", line 85, in sync_repo    
                  return self.sync_step.process_lifecycle()   File
                  "/home/ipanova/pulp_development/pulp/server/pulp/plugins/util/
                  publish_step.py", line 572, in process_lifecycle    
                  super(PluginStep, self).process_lifecycle()   File
                  "/home/ipanova/pulp_development/pulp/server/pulp/plugins/util/
                  publish_step.py", line 163, in process_lifecycle    
                  step.process()   File
                  "/home/ipanova/pulp_development/pulp/server/pulp/plugins/util/
                  publish_step.py", line 256, in process    
                  self._process_block()   File
                  "/home/ipanova/pulp_development/pulp/server/pulp/plugins/util/
                  publish_step.py", line 303, in _process_block    
                  self.process_main()   File
                  "/home/ipanova/pulp_development/pulp_docker/plugins/pulp_docke
                  r/plugins/importers/sync.py", line 237, in process_main    
                  self._process_manifest_list(manifest, digest, available_blobs,
                  tag)   File
                  "/home/ipanova/pulp_development/pulp_docker/plugins/pulp_docke
                  r/plugins/importers/sync.py", line 283, in
                  _process_manifest_list     self._process_manifest(manifest,
                  digest, available_blobs, tag=tag)   File
                  "/home/ipanova/pulp_development/pulp_docker/plugins/pulp_docke
                  r/plugins/importers/sync.py", line 313, in _process_manifest  
                  manifest = models.Manifest.from_json(manifest, digest)   File
                  "/home/ipanova/pulp_development/pulp_docker/plugins/pulp_docke
                  r/plugins/models.py", line 219, in from_json     fs_layers =
                  [FSLayer(blob_sum=layer['blobSum']) for layer in
                  manifest['fsLayers']] KeyError: 'fsLayers' 

[ipanova@ina pulp]$

Actions

Copy link

Updated by ipanova@redhat.com about 5 years ago

the issue is here https://github.com/pulp/pulp_docker/blob/2-master/plugins/pulp_docker/plugins/importers/sync.py#L281

Registry is supposed to return the the older format( if available) in case in accept header nothing is specified. GCR instead returns manifest list.

$curl -i -L -X GET 'https://gcr.io/v2/google-containers/etcd/manifests/3.2.24'
HTTP/2 200 
docker-distribution-api-version: registry/2.0
content-type: application/vnd.docker.distribution.manifest.list.v2+json
content-length: 1665
docker-content-digest: sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20
date: Fri, 08 Feb 2019 15:15:19 GMT
server: Docker Registry
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN
alt-svc: quic=":443"; ma=2592000; v="44,43,39"

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 950,
         "digest": "sha256:7b073bdab8c52dc23dfb3e2101597d30304437869ad8c0b425301e96a066c408",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 950,
         "digest": "sha256:4f93210747a2150605fdbc764a86b549824f322f2168b7c762bd4e9a9a264860",
         "platform": {
            "architecture": "arm",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 950,
         "digest": "sha256:f1dcb048bb5a5c6d74e07db03f0a006a8bdcea4dc034722d1d9e24da15901390",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 951,
         "digest": "sha256:bf275fe898f9efda3db428e3287970b348a7e60d1086af36770aa78c9caec886",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 951,
         "digest": "sha256:857376ef4d252c5ce1817902216dc92fedcc133a225144b3a427e5f89c88a17f",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]

check for the reference https://docs.docker.com/registry/spec/manifest-v2-2/#backward-compatibility

If the manifest being requested uses the new format, and the appropriate media type is not present in an Accept header, the registry will assume that the client cannot handle the manifest as-is, and rewrite it on the fly into the old format. If the object that would otherwise be returned is a manifest list, the registry will look up the appropriate manifest for the amd64 platform and linux OS, rewrite that manifest into the old format if necessary, and return the result to the client. If no suitable manifest is found in the manifest list, the registry will return a 404 error.

Actions

Copy link

#10