Project

Profile

Help

Issue #3010

Can't sync Docker images

Added by Ichimonji10 13 days ago. Updated about 5 hours ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Severity:
2. Medium
Version - Docker:
Platform Release:
2.13.4
Blocks Release:
2.13.z
Target Release - Docker:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
Yes
Verification Required:
No

Description

The Docker plugin is unable to sync Docker images.

You can reproduce the failure manually:

pulp-admin docker repo create --repo-id foo --feed https://registry-1.docker.io --upstream-name library/busybox
pulp-admin docker repo sync run --repo-id foo

Here's the error reported by pulp-admin:

Task Failed

The Manifest digest does not match the expected value. The remote feed announced
a digest of
sha256:6cb02cbabacf59962d0cd05176e68ae4769e998e8d98f5e3f28fe915de341d6c, but the
downloaded digest was
sha256:98f12e5b0243774d2767dee77ad15b7cc5a835061e408c3bae9038ccdfa9d955.

Many of the Pulp Smash tests for Docker fail. Here's one way to reproduce the failure with Pulp Smash:

python -m unittest pulp_smash.tests.docker.api_v2.test_sync_publish.NonNamespacedImageTestCase

The test will fail on this line of code:

utils.sync_repo(cfg, repo)

Here's the journal entry from the Pulp Smash test listed above:

# journalctl -p err..crit
-- Logs begin at Wed 2017-09-13 15:29:00 EDT, end at Wed 2017-09-13 15:51:39 EDT. --
Sep 13 15:29:03 fedora-25-pulp-2-13 smartd[716]: DEVICESCAN failed: glob(3) aborted matching pattern /dev/discs/disc*
Sep 13 15:29:03 fedora-25-pulp-2-13 smartd[716]: In the system's table of devices NO devices found to scan
Sep 13 15:32:11 fedora-25-pulp-2-13 sshd[993]: pam_lastlog(sshd:session): unable to open /var/log/lastlog: No such file or directory
Sep 13 15:38:08 fedora-25-pulp-2-13 sshd[1093]: pam_lastlog(sshd:session): unable to open /var/log/lastlog: No such file or directory
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) Task pulp.server.managers.repo.sync.sync[5aa76f87-14e1-4dc2-828c-def35e3d6099] raised unexpected: IOError('The Manifest digest does not match the expected value. The remote feed announced a digest of sha256:6cb02cbabacf59962d0cd05176e68ae4769e998e8d98f5e3f28fe915de341d6c, but the downloaded digest was sha256:98f12e5b0243774d2767dee77ad15b7cc5a835061e408c3bae9038ccdfa9d955.',)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) Traceback (most recent call last):
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)     R = retval = fun(*args, **kwargs)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 527, in __call__
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)     return super(Task, self).__call__(*args, **kwargs)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)     return super(PulpTask, self).__call__(*args, **kwargs)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)     return self.run(*args, **kwargs)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 769, in sync
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)     sync_report = sync_repo(transfer_repo, conduit, call_config)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 730, in wrap_f
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)     return f(*args, **kwargs)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)   File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 85, in sync_repo
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)     return self.sync_step.process_lifecycle()
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 572, in process_lifecycle
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)     super(PluginStep, self).process_lifecycle()
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 163, in process_lifecycle
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)     step.process()
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 256, in process
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)     self._process_block()
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 303, in _process_block
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)     self.process_main()
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)   File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py", line 231, in process_main
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)     manifests = self.parent.index_repository.get_manifest(tag)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)   File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/registry.py", line 380, in get_manifest
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)     digest = self._digest_check(response_headers, manifest)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)   File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/registry.py", line 415, in _digest_check
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352)     raise IOError(msg)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) IOError: The Manifest digest does not match the expected value. The remote feed announced a digest of sha256:6cb02cbabacf59962d0cd05176e68ae4769e998e8d98f5e3f28fe915de341d6c, but the downloaded digest was sha256:98f12e5b0243774d2767dee77ad15b7cc5a835061e408c3bae9038ccdfa9d955.
Sep 13 15:49:31 fedora-25-pulp-2-13 sshd[31511]: pam_lastlog(sshd:session): unable to open /var/log/lastlog: No such file or directory

This issue affects Pulp 2.13 for all distributions. Here's the packages installed on an F25 system, as an example:

# rpm -qa | grep -i pulp | sort
pulp-admin-client-2.13.4-0.1.beta.git.19.abf785f.fc25.noarch       
pulp-docker-admin-extensions-2.4.0-2.fc25.noarch                   
pulp-docker-plugins-2.4.0-2.fc25.noarch                            
pulp-ostree-admin-extensions-1.2.1-3.fc25.noarch                   
pulp-ostree-plugins-1.2.1-3.fc25.noarch                            
pulp-puppet-admin-extensions-2.13.4-0.1.beta.git.8.9344c02.fc25.noarch                                                                
pulp-puppet-plugins-2.13.4-0.1.beta.git.8.9344c02.fc25.noarch      
pulp-python-admin-extensions-2.0.0-2.fc25.noarch                   
pulp-python-plugins-2.0.0-2.fc25.noarch                            
pulp-rpm-admin-extensions-2.13.4-0.1.beta.git.10.48cc00d.fc25.noarch                                                                  
pulp-rpm-plugins-2.13.4-0.1.beta.git.10.48cc00d.fc25.noarch        
pulp-selinux-2.13.4-0.1.beta.git.19.abf785f.fc25.noarch            
pulp-server-2.13.4-0.1.beta.git.19.abf785f.fc25.noarch             
python-pulp-bindings-2.13.4-0.1.beta.git.19.abf785f.fc25.noarch    
python-pulp-client-lib-2.13.4-0.1.beta.git.19.abf785f.fc25.noarch  
python-pulp-common-2.13.4-0.1.beta.git.19.abf785f.fc25.noarch      
python-pulp-docker-common-2.4.0-2.fc25.noarch                      
python-pulp-oid_validation-2.13.4-0.1.beta.git.19.abf785f.fc25.noarch                                                                 
python-pulp-ostree-common-1.2.1-3.fc25.noarch                      
python-pulp-puppet-common-2.13.4-0.1.beta.git.8.9344c02.fc25.noarch                                                                   
python-pulp-python-common-2.0.0-2.fc25.noarch                      
python-pulp-repoauth-2.13.4-0.1.beta.git.19.abf785f.fc25.noarch    
python-pulp-rpm-common-2.13.4-0.1.beta.git.10.48cc00d.fc25.noarch  
python-pulp-streamer-2.13.4-0.1.beta.git.19.abf785f.fc25.noarch

Associated revisions

Revision 2565f5d9 View on GitHub
Added by ipanova@redhat.com 12 days ago

Disable digest check for image manifest schema2.

Uppstream docker distribution issue:
https://github.com/docker/distribution/pull/2310

History

#1 Updated by Ichimonji10 13 days ago

  • Description updated (diff)

#2 Updated by Ichimonji10 13 days ago

For what it's worth, one can execute this script:

pulp-admin docker repo create --repo-id bar --feed https://registry-1.docker.io --upstream-name fedora/apache
pulp-admin docker repo sync run --repo-id bar

#3 Updated by pcreech 13 days ago

This exists on 2.13-release branches as well. (using the vagrant environment)

Looks like there is a maybe related issue? https://pulp.plan.io/issues/2517

I failed to sync the following images:

library/amazonlinux
library/busybox
library/fedora

While I was able to sync the following:
library/bash

#4 Updated by ipanova@redhat.com 12 days ago

Because what docker docker registry returns in the headers, unfortunately it is not possible to sync some repos with pulp 2.13

1) why it was possible to sync repos with pulp21.3 previously? Any why now we are able to sync just some repos with pulp2.13?
Because they did not contain manifest lists. Now the repos which contain manifest lists will fail the sync with pulp2.13

2) why the sync fails when in the repo there are manifest lists?
Because docker registry returns in the headers the digest sha256:X, but in the body the content itself does not match the sha256:X.
We do not trust what it is returned in the headers, so what we do is - we calculate the digest on our own based on the content returned, and then compare with the digest which was specified in the response headers from the registry.

Solutions:

1) docker guys should fix that - It is fixed, and even merged https://github.com/docker/distribution/pull/2310 But the problem will still persist with old registries which do not contain the fix.
2) we would remove the check of the digest. ( we already removed that check (partially) in 2.14, when we introduced manifest list support) https://github.com/pulp/pulp_docker/pull/200/files#diff-99c0d67fc4b80f3d190c6b6af1c453b1R385
3) encourage people to upgrade to 2.14

To add more - all sync of repos which contains manifest lists would fail with pulp < 2.14

#5 Updated by ipanova@redhat.com 7 days ago

  • Status changed from NEW to MODIFIED
  • Triaged changed from No to Yes

the fix was implemented for schema2 image manifests https://github.com/pulp/pulp_docker/pull/206

#6 Updated by Ichimonji10 7 days ago

  • Verified changed from No to Yes

This issue is fixed in the latest nightlies of 2.13.

#8 Updated by pcreech about 5 hours ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
  • Platform Release set to 2.13.4

Please register to edit this issue

Also available in: Atom PDF