Project

Profile

Help

Issue #3122

config layer is served gzip-compressed to docker client

Added by mihai.ibanescu@gmail.com over 2 years ago. Updated 11 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Severity:
2. Medium
Version - Docker:
3.1.0
Platform Release:
2.15.0
Blocks Release:
Target Release - Docker:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 28

Description

When I try to import a docker image generated by skopeo, that happens to contain a config layer, pulp will compress it. That seems to make docker unhappy.

Details: In plugins/importers/upload.py, ProcessManifest.get_models() gets passed a manifest object, and manifest.config_layer gets added as a Blob object.

Later, in AddUnits.process_main, blobs are being iterated over and gzip-compressed.

I believe this is what makes docker unhappy when I try to pull the image from crane:

docker pull registry.example.com:443/test-docker:latest
Trying to pull repository registry.example.com:443/test-docker ... 
sha256:c71a68cc5bcd74b2ac0f8418c61230a9f2ee6ac470091f7a10a5bbe26a2fd388: Pulling from registry.example.com:443/test-docker
97ca462ad9ee: Pulling fs layer 
776eeafa9b09: Pulling fs layer 
60058d98fabf: Pulling fs layer 
ce8c08c957d4: Waiting 
885734936e35: Waiting 
ee161b7aa244: Waiting 
da7a176f74a7: Waiting 
2d0f9340ea22: Waiting 
2c0937f8af97: Waiting 
error pulling image configuration: image config verification failed for digest sha256:302b355390efa687ba03ae2bdea881ca2a140da6278a7d948f07c00260aec3ca

Associated revisions

Revision 3688b6ee View on GitHub
Added by Mihai Ibanescu over 2 years ago

v2 unit import: do not compress blob layers

Skopeo produces compressed layers already. Double-compressing them invalidates the SHA checksum.

closes #3122

Revision 3688b6ee View on GitHub
Added by Mihai Ibanescu over 2 years ago

v2 unit import: do not compress blob layers

Skopeo produces compressed layers already. Double-compressing them invalidates the SHA checksum.

closes #3122

Revision 3688b6ee View on GitHub
Added by Mihai Ibanescu over 2 years ago

v2 unit import: do not compress blob layers

Skopeo produces compressed layers already. Double-compressing them invalidates the SHA checksum.

closes #3122

Revision 3688b6ee View on GitHub
Added by Mihai Ibanescu over 2 years ago

v2 unit import: do not compress blob layers

Skopeo produces compressed layers already. Double-compressing them invalidates the SHA checksum.

closes #3122

History

#1 Updated by mihai.ibanescu@gmail.com over 2 years ago

Turns out there is no need to compress anything. Skopeo already compresses the layers.

#2 Updated by mihai.ibanescu@gmail.com over 2 years ago

Proposed patch would be a one-liner if I didn't hate the use of functools when shutils has a perfectly good copyfileobj.

diff --git a/plugins/pulp_docker/plugins/importers/upload.py b/plugins/pulp_docker/plugins/importers/upload.py
index 1ce47b0..780de62 100644
--- a/plugins/pulp_docker/plugins/importers/upload.py
+++ b/plugins/pulp_docker/plugins/importers/upload.py
@@ -22,6 +22,7 @@ import functools
 import gzip
 import json
 import os
+import shutil
 import stat
 import tarfile

@@ -340,11 +341,8 @@ class AddUnits(PluginStep):
             blob_src_path = os.path.join(self.get_working_dir(), item.digest.split(':')[1] + '.tar')
             blob_dest_path = os.path.join(self.get_working_dir(), item.digest)
             with open(blob_src_path) as blob_src:
-                with contextlib.closing(gzip.open(blob_dest_path, 'w')) as blob_dest:
-                    # these can be big files, so we chunk them
-                    reader = functools.partial(blob_src.read, 4096)
-                    for chunk in iter(reader, ''):
-                        blob_dest.write(chunk)
+                with contextlib.closing(open(blob_dest_path, 'w')) as blob_dest:
+                    shutil.copyfileobj(blob_src, blob_dest)
             # we don't need the tarfile anymore
             os.remove(blob_src_path)

#3 Updated by ttereshc over 2 years ago

  • Status changed from NEW to POST
  • Assignee set to mihai.ibanescu@gmail.com

#4 Updated by ttereshc over 2 years ago

  • Description updated (diff)

#5 Updated by dalley over 2 years ago

  • Sprint/Milestone set to 47
  • Triaged changed from No to Yes

#7 Updated by Anonymous over 2 years ago

  • Status changed from POST to MODIFIED

#8 Updated by pcreech about 2 years ago

  • Platform Release set to 2.15.0

#9 Updated by pcreech about 2 years ago

  • Status changed from MODIFIED to ON_QA

#10 Updated by Ichimonji10 about 2 years ago

@preethi I don't think this issue should be automated at this time. Doing so requires adding a dependency on skopeo, and we have enough other things to do without adding another dependency.

#11 Updated by pcreech about 2 years ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#12 Updated by bmbouter almost 2 years ago

  • Sprint set to Sprint 28

#13 Updated by bmbouter almost 2 years ago

  • Sprint/Milestone deleted (47)

#14 Updated by bmbouter 11 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF