Issue #3010
closed
Can't sync Docker images
Description
The Docker plugin is unable to sync Docker images.
You can reproduce the failure manually:
pulp-admin docker repo create --repo-id foo --feed https://registry-1.docker.io --upstream-name library/busybox
pulp-admin docker repo sync run --repo-id foo
Here's the error reported by pulp-admin:
Task Failed
The Manifest digest does not match the expected value. The remote feed announced
a digest of
sha256:6cb02cbabacf59962d0cd05176e68ae4769e998e8d98f5e3f28fe915de341d6c, but the
downloaded digest was
sha256:98f12e5b0243774d2767dee77ad15b7cc5a835061e408c3bae9038ccdfa9d955.
Many of the Pulp Smash tests for Docker fail. Here's one way to reproduce the failure with Pulp Smash:
python -m unittest pulp_smash.tests.docker.api_v2.test_sync_publish.NonNamespacedImageTestCase
The test will fail on this line of code:
utils.sync_repo(cfg, repo)
Here's the journal entry from the Pulp Smash test listed above:
# journalctl -p err..crit
-- Logs begin at Wed 2017-09-13 15:29:00 EDT, end at Wed 2017-09-13 15:51:39 EDT. --
Sep 13 15:29:03 fedora-25-pulp-2-13 smartd[716]: DEVICESCAN failed: glob(3) aborted matching pattern /dev/discs/disc*
Sep 13 15:29:03 fedora-25-pulp-2-13 smartd[716]: In the system's table of devices NO devices found to scan
Sep 13 15:32:11 fedora-25-pulp-2-13 sshd[993]: pam_lastlog(sshd:session): unable to open /var/log/lastlog: No such file or directory
Sep 13 15:38:08 fedora-25-pulp-2-13 sshd[1093]: pam_lastlog(sshd:session): unable to open /var/log/lastlog: No such file or directory
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) Task pulp.server.managers.repo.sync.sync[5aa76f87-14e1-4dc2-828c-def35e3d6099] raised unexpected: IOError('The Manifest digest does not match the expected value. The remote feed announced a digest of sha256:6cb02cbabacf59962d0cd05176e68ae4769e998e8d98f5e3f28fe915de341d6c, but the downloaded digest was sha256:98f12e5b0243774d2767dee77ad15b7cc5a835061e408c3bae9038ccdfa9d955.',)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) Traceback (most recent call last):
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) R = retval = fun(*args, **kwargs)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 527, in __call__
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) return super(Task, self).__call__(*args, **kwargs)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) return super(PulpTask, self).__call__(*args, **kwargs)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) return self.run(*args, **kwargs)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 769, in sync
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) sync_report = sync_repo(transfer_repo, conduit, call_config)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 730, in wrap_f
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) return f(*args, **kwargs)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 85, in sync_repo
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) return self.sync_step.process_lifecycle()
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 572, in process_lifecycle
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) super(PluginStep, self).process_lifecycle()
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 163, in process_lifecycle
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) step.process()
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 256, in process
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) self._process_block()
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 303, in _process_block
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) self.process_main()
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py", line 231, in process_main
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) manifests = self.parent.index_repository.get_manifest(tag)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/registry.py", line 380, in get_manifest
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) digest = self._digest_check(response_headers, manifest)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/registry.py", line 415, in _digest_check
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) raise IOError(msg)
Sep 13 15:48:50 fedora-25-pulp-2-13 pulp[30733]: celery.worker.job:ERROR: (30733-16352) IOError: The Manifest digest does not match the expected value. The remote feed announced a digest of sha256:6cb02cbabacf59962d0cd05176e68ae4769e998e8d98f5e3f28fe915de341d6c, but the downloaded digest was sha256:98f12e5b0243774d2767dee77ad15b7cc5a835061e408c3bae9038ccdfa9d955.
Sep 13 15:49:31 fedora-25-pulp-2-13 sshd[31511]: pam_lastlog(sshd:session): unable to open /var/log/lastlog: No such file or directory
This issue affects Pulp 2.13 for all distributions. Here's the packages installed on an F25 system, as an example:
# rpm -qa | grep -i pulp | sort
pulp-admin-client-2.13.4-0.1.beta.git.19.abf785f.fc25.noarch
pulp-docker-admin-extensions-2.4.0-2.fc25.noarch
pulp-docker-plugins-2.4.0-2.fc25.noarch
pulp-ostree-admin-extensions-1.2.1-3.fc25.noarch
pulp-ostree-plugins-1.2.1-3.fc25.noarch
pulp-puppet-admin-extensions-2.13.4-0.1.beta.git.8.9344c02.fc25.noarch
pulp-puppet-plugins-2.13.4-0.1.beta.git.8.9344c02.fc25.noarch
pulp-python-admin-extensions-2.0.0-2.fc25.noarch
pulp-python-plugins-2.0.0-2.fc25.noarch
pulp-rpm-admin-extensions-2.13.4-0.1.beta.git.10.48cc00d.fc25.noarch
pulp-rpm-plugins-2.13.4-0.1.beta.git.10.48cc00d.fc25.noarch
pulp-selinux-2.13.4-0.1.beta.git.19.abf785f.fc25.noarch
pulp-server-2.13.4-0.1.beta.git.19.abf785f.fc25.noarch
python-pulp-bindings-2.13.4-0.1.beta.git.19.abf785f.fc25.noarch
python-pulp-client-lib-2.13.4-0.1.beta.git.19.abf785f.fc25.noarch
python-pulp-common-2.13.4-0.1.beta.git.19.abf785f.fc25.noarch
python-pulp-docker-common-2.4.0-2.fc25.noarch
python-pulp-oid_validation-2.13.4-0.1.beta.git.19.abf785f.fc25.noarch
python-pulp-ostree-common-1.2.1-3.fc25.noarch
python-pulp-puppet-common-2.13.4-0.1.beta.git.8.9344c02.fc25.noarch
python-pulp-python-common-2.0.0-2.fc25.noarch
python-pulp-repoauth-2.13.4-0.1.beta.git.19.abf785f.fc25.noarch
python-pulp-rpm-common-2.13.4-0.1.beta.git.10.48cc00d.fc25.noarch
python-pulp-streamer-2.13.4-0.1.beta.git.19.abf785f.fc25.noarch
Updated by Ichimonji10 over 5 years ago
For what it's worth, one can execute this script:
pulp-admin docker repo create --repo-id bar --feed https://registry-1.docker.io --upstream-name fedora/apache
pulp-admin docker repo sync run --repo-id bar
Updated by pcreech over 5 years ago
This exists on 2.13-release branches as well. (using the vagrant environment)
Looks like there is a maybe related issue? https://pulp.plan.io/issues/2517
I failed to sync the following images:
library/amazonlinux
library/busybox
library/fedora
While I was able to sync the following:
library/bash
Updated by ipanova@redhat.com over 5 years ago
Because what docker docker registry returns in the headers, unfortunately it is not possible to sync some repos with pulp 2.13
1) why it was possible to sync repos with pulp21.3 previously? Any why now we are able to sync just some repos with pulp2.13?
Because they did not contain manifest lists. Now the repos which contain manifest lists will fail the sync with pulp2.13
2) why the sync fails when in the repo there are manifest lists?
Because docker registry returns in the headers the digest sha256:X, but in the body the content itself does not match the sha256:X.
We do not trust what it is returned in the headers, so what we do is - we calculate the digest on our own based on the content returned, and then compare with the digest which was specified in the response headers from the registry.
Solutions:
1) docker guys should fix that - It is fixed, and even merged https://github.com/docker/distribution/pull/2310 But the problem will still persist with old registries which do not contain the fix.
2) we would remove the check of the digest. ( we already removed that check (partially) in 2.14, when we introduced manifest list support) https://github.com/pulp/pulp_docker/pull/200/files#diff-99c0d67fc4b80f3d190c6b6af1c453b1R385
3) encourage people to upgrade to 2.14
To add more - all sync of repos which contains manifest lists would fail with pulp < 2.14
Added by ipanova@redhat.com over 5 years ago
Updated by ipanova@redhat.com over 5 years ago
- Status changed from NEW to MODIFIED
- Triaged changed from No to Yes
the fix was implemented for schema2 image manifests https://github.com/pulp/pulp_docker/pull/206
Updated by pcreech over 5 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
- Platform Release set to 2.13.4
.
Disable digest check for image manifest schema2.
Uppstream docker distribution issue: https://github.com/docker/distribution/pull/2310