Issue #3122
closedconfig layer is served gzip-compressed to docker client
Description
When I try to import a docker image generated by skopeo, that happens to contain a config layer, pulp will compress it. That seems to make docker unhappy.
Details: In plugins/importers/upload.py, ProcessManifest.get_models()
gets passed a manifest object, and manifest.config_layer
gets added as a Blob object.
Later, in AddUnits.process_main
, blobs are being iterated over and gzip-compressed.
I believe this is what makes docker unhappy when I try to pull the image from crane:
docker pull registry.example.com:443/test-docker:latest
Trying to pull repository registry.example.com:443/test-docker ...
sha256:c71a68cc5bcd74b2ac0f8418c61230a9f2ee6ac470091f7a10a5bbe26a2fd388: Pulling from registry.example.com:443/test-docker
97ca462ad9ee: Pulling fs layer
776eeafa9b09: Pulling fs layer
60058d98fabf: Pulling fs layer
ce8c08c957d4: Waiting
885734936e35: Waiting
ee161b7aa244: Waiting
da7a176f74a7: Waiting
2d0f9340ea22: Waiting
2c0937f8af97: Waiting
error pulling image configuration: image config verification failed for digest sha256:302b355390efa687ba03ae2bdea881ca2a140da6278a7d948f07c00260aec3ca
Updated by mihai.ibanescu@gmail.com about 7 years ago
Turns out there is no need to compress anything. Skopeo already compresses the layers.
Updated by mihai.ibanescu@gmail.com about 7 years ago
Proposed patch would be a one-liner if I didn't hate the use of functools when shutils has a perfectly good copyfileobj.
diff --git a/plugins/pulp_docker/plugins/importers/upload.py b/plugins/pulp_docker/plugins/importers/upload.py
index 1ce47b0..780de62 100644
--- a/plugins/pulp_docker/plugins/importers/upload.py
+++ b/plugins/pulp_docker/plugins/importers/upload.py
@@ -22,6 +22,7 @@ import functools
import gzip
import json
import os
+import shutil
import stat
import tarfile
@@ -340,11 +341,8 @@ class AddUnits(PluginStep):
blob_src_path = os.path.join(self.get_working_dir(), item.digest.split(':')[1] + '.tar')
blob_dest_path = os.path.join(self.get_working_dir(), item.digest)
with open(blob_src_path) as blob_src:
- with contextlib.closing(gzip.open(blob_dest_path, 'w')) as blob_dest:
- # these can be big files, so we chunk them
- reader = functools.partial(blob_src.read, 4096)
- for chunk in iter(reader, ''):
- blob_dest.write(chunk)
+ with contextlib.closing(open(blob_dest_path, 'w')) as blob_dest:
+ shutil.copyfileobj(blob_src, blob_dest)
# we don't need the tarfile anymore
os.remove(blob_src_path)
Updated by ttereshc about 7 years ago
- Status changed from NEW to POST
- Assignee set to mihai.ibanescu@gmail.com
Updated by dalley about 7 years ago
- Sprint/Milestone set to 47
- Triaged changed from No to Yes
Added by Mihai Ibanescu about 7 years ago
Added by Mihai Ibanescu about 7 years ago
Revision 3688b6ee | View on GitHub
v2 unit import: do not compress blob layers
Skopeo produces compressed layers already. Double-compressing them invalidates the SHA checksum.
closes #3122
Added by Mihai Ibanescu about 7 years ago
Revision 3688b6ee | View on GitHub
v2 unit import: do not compress blob layers
Skopeo produces compressed layers already. Double-compressing them invalidates the SHA checksum.
closes #3122
Added by Mihai Ibanescu about 7 years ago
Revision 3688b6ee | View on GitHub
v2 unit import: do not compress blob layers
Skopeo produces compressed layers already. Double-compressing them invalidates the SHA checksum.
closes #3122
Updated by ipanova@redhat.com about 7 years ago
Steps to reproduce and verify:
https://github.com/pulp/pulp_docker/pull/211#issuecomment-344270012
Updated by Anonymous about 7 years ago
- Status changed from POST to MODIFIED
Applied in changeset 3688b6eeb9261bd11ad1023efb2c6d5facd6c073.
Updated by Ichimonji10 almost 7 years ago
@preethi I don't think this issue should be automated at this time. Doing so requires adding a dependency on skopeo, and we have enough other things to do without adding another dependency.
Updated by pcreech almost 7 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
v2 unit import: do not compress blob layers
Skopeo produces compressed layers already. Double-compressing them invalidates the SHA checksum.
closes #3122