Project

Profile

Help

Issue #3122

closed

config layer is served gzip-compressed to docker client

Added by mihai.ibanescu@gmail.com over 6 years ago. Updated about 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version - Docker:
3.1.0
Platform Release:
2.15.0
Target Release - Docker:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 28
Quarter:

Description

When I try to import a docker image generated by skopeo, that happens to contain a config layer, pulp will compress it. That seems to make docker unhappy.

Details: In plugins/importers/upload.py, ProcessManifest.get_models() gets passed a manifest object, and manifest.config_layer gets added as a Blob object.

Later, in AddUnits.process_main, blobs are being iterated over and gzip-compressed.

I believe this is what makes docker unhappy when I try to pull the image from crane:

docker pull registry.example.com:443/test-docker:latest
Trying to pull repository registry.example.com:443/test-docker ... 
sha256:c71a68cc5bcd74b2ac0f8418c61230a9f2ee6ac470091f7a10a5bbe26a2fd388: Pulling from registry.example.com:443/test-docker
97ca462ad9ee: Pulling fs layer 
776eeafa9b09: Pulling fs layer 
60058d98fabf: Pulling fs layer 
ce8c08c957d4: Waiting 
885734936e35: Waiting 
ee161b7aa244: Waiting 
da7a176f74a7: Waiting 
2d0f9340ea22: Waiting 
2c0937f8af97: Waiting 
error pulling image configuration: image config verification failed for digest sha256:302b355390efa687ba03ae2bdea881ca2a140da6278a7d948f07c00260aec3ca
Actions #1

Updated by mihai.ibanescu@gmail.com over 6 years ago

Turns out there is no need to compress anything. Skopeo already compresses the layers.

Actions #2

Updated by mihai.ibanescu@gmail.com over 6 years ago

Proposed patch would be a one-liner if I didn't hate the use of functools when shutils has a perfectly good copyfileobj.

diff --git a/plugins/pulp_docker/plugins/importers/upload.py b/plugins/pulp_docker/plugins/importers/upload.py
index 1ce47b0..780de62 100644
--- a/plugins/pulp_docker/plugins/importers/upload.py
+++ b/plugins/pulp_docker/plugins/importers/upload.py
@@ -22,6 +22,7 @@ import functools
 import gzip
 import json
 import os
+import shutil
 import stat
 import tarfile

@@ -340,11 +341,8 @@ class AddUnits(PluginStep):
             blob_src_path = os.path.join(self.get_working_dir(), item.digest.split(':')[1] + '.tar')
             blob_dest_path = os.path.join(self.get_working_dir(), item.digest)
             with open(blob_src_path) as blob_src:
-                with contextlib.closing(gzip.open(blob_dest_path, 'w')) as blob_dest:
-                    # these can be big files, so we chunk them
-                    reader = functools.partial(blob_src.read, 4096)
-                    for chunk in iter(reader, ''):
-                        blob_dest.write(chunk)
+                with contextlib.closing(open(blob_dest_path, 'w')) as blob_dest:
+                    shutil.copyfileobj(blob_src, blob_dest)
             # we don't need the tarfile anymore
             os.remove(blob_src_path)
Actions #3

Updated by ttereshc over 6 years ago

  • Status changed from NEW to POST
  • Assignee set to mihai.ibanescu@gmail.com
Actions #4

Updated by ttereshc over 6 years ago

  • Description updated (diff)
Actions #5

Updated by dalley over 6 years ago

  • Sprint/Milestone set to 47
  • Triaged changed from No to Yes

Added by Mihai Ibanescu over 6 years ago

Revision 3688b6ee | View on GitHub

v2 unit import: do not compress blob layers

Skopeo produces compressed layers already. Double-compressing them invalidates the SHA checksum.

closes #3122

Added by Mihai Ibanescu over 6 years ago

Revision 3688b6ee | View on GitHub

v2 unit import: do not compress blob layers

Skopeo produces compressed layers already. Double-compressing them invalidates the SHA checksum.

closes #3122

Added by Mihai Ibanescu over 6 years ago

Revision 3688b6ee | View on GitHub

v2 unit import: do not compress blob layers

Skopeo produces compressed layers already. Double-compressing them invalidates the SHA checksum.

closes #3122

Added by Mihai Ibanescu over 6 years ago

Revision 3688b6ee | View on GitHub

v2 unit import: do not compress blob layers

Skopeo produces compressed layers already. Double-compressing them invalidates the SHA checksum.

closes #3122

Actions #7

Updated by Anonymous over 6 years ago

  • Status changed from POST to MODIFIED
Actions #8

Updated by pcreech over 6 years ago

  • Platform Release set to 2.15.0
Actions #9

Updated by pcreech over 6 years ago

  • Status changed from MODIFIED to 5
Actions #10

Updated by Ichimonji10 over 6 years ago

@preethi I don't think this issue should be automated at this time. Doing so requires adding a dependency on skopeo, and we have enough other things to do without adding another dependency.

Actions #11

Updated by pcreech over 6 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #12

Updated by bmbouter about 6 years ago

  • Sprint set to Sprint 28
Actions #13

Updated by bmbouter about 6 years ago

  • Sprint/Milestone deleted (47)
Actions #14

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF