Issue #3892
closedCan't copy Docker tags or manifest lists between repositories
Description
It's impossible to copy Docker tags or manifest lists between repositories. Attempting to do triggers an error in Pulp. Here's the relevant journalctl output when attempting to copy manifest lists:
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) Exception from importer [docker_importer] while importing units into repository [14053d5e-0ea9-4b13-9e56-f4c769a
0c0a6]
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) Traceback (most recent call last):
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 273, in associate_from_repo
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) units=transfer_units)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 217, in import_units
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) units_added |= set(unit_importers[type(unit)](import_conduit, unit, dest_repo.repo_obj))
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 369, in _import_manifest_list
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) for manifest in models.Manifest.objects.filter(digest__in=sorted(manifest_digests)):
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 1407, in next
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) raw_doc = self._cursor.next()
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) if len(self.__data) or self._refresh():
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1019, in _refresh
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) self.__read_concern))
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 850, in __send_message
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) **kwargs)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 794, in _send_message_with_response
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) exhaust)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 805, in _reset_on_error
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) return func(*args, **kwargs)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/server.py", line 108, in send_message_with_response
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) set_slave_okay, sock_info.is_mongos, use_find_cmd)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/message.py", line 275, in get_message
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) spec, self.fields, self.codec_options)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) InvalidDocument: Cannot encode object: <EmbeddedManifest: EmbeddedManifest object>
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.async.tasks:INFO: [30e85096] Task failed : [30e85096-c215-4b4e-99e6-682dcbede8b1]
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) Task pulp.server.managers.repo.unit_association.associate_from_repo[30e85096-c215-4b4e-99e6-682dcbede8b1] raised unexpected: InvalidDocument('Cannot encode object: <EmbeddedManifest: EmbeddedManifest object>',)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) Traceback (most recent call last):
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) R = retval = fun(*args, **kwargs)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 529, in __call__
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) return super(Task, self).__call__(*args, **kwargs)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) return super(PulpTask, self).__call__(*args, **kwargs)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) return self.run(*args, **kwargs)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 291, in associate_from_repo
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) raise (e, None, sys.exc_info()[2])
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) InvalidDocument: Cannot encode object: <EmbeddedManifest: EmbeddedManifest object>
...and here's the error when attempting to copy tags between repositories:
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) Exception from importer [docker_importer] while importing units into repository [94d08260-7dc9-4bf0-9b9a-d23ed19
3c2c3]
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) Traceback (most recent call last):
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 273, in associate_from_repo
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) units=transfer_units)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 217, in import_units
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) units_added |= set(unit_importers[type(unit)](import_conduit, unit, dest_repo.repo_obj))
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 299, in _import_tag
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) conduit, manifest, dest_repo))
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 369, in _import_manifest_list
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) for manifest in models.Manifest.objects.filter(digest__in=sorted(manifest_digests)):
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 1407, in next
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) raw_doc = self._cursor.next()
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) if len(self.__data) or self._refresh():
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1019, in _refresh
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) self.__read_concern))
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 850, in __send_message
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) **kwargs)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 794, in _send_message_with_response
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) exhaust)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 805, in _reset_on_error
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) return func(*args, **kwargs)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/server.py", line 108, in send_message_with_response
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) set_slave_okay, sock_info.is_mongos, use_find_cmd)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/message.py", line 275, in get_message
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) spec, self.fields, self.codec_options)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) InvalidDocument: Cannot encode object: <EmbeddedManifest: EmbeddedManifest object>
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.async.tasks:INFO: [7c6d1641] Task failed : [7c6d1641-66f4-4408-8a70-6a7d3d2b36ec]
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) Task pulp.server.managers.repo.unit_association.associate_from_repo[7c6d1641-66f4-4408-8a70-6a7d3d2b36ec] raised unexpected: InvalidDocument('Cannot encode object: <EmbeddedManifest: EmbeddedManifest object>',)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) Traceback (most recent call last):
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) R = retval = fun(*args, **kwargs)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 529, in __call__
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) return super(Task, self).__call__(*args, **kwargs)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) return super(PulpTask, self).__call__(*args, **kwargs)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) return self.run(*args, **kwargs)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 291, in associate_from_repo
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) raise (e, None, sys.exc_info()[2])
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) InvalidDocument: Cannot encode object: <EmbeddedManifest: EmbeddedManifest object>
The error looks nearly identical. This error occurs with the nightly build of Pulp 2 (to-be 2.17) on both RHEL 7 and Fedora 27. Here's some sample RPMs:
[root@rhel-7-pulp-2-17-nightly ~]# rpm -qa | grep pulp | sort
pulp-admin-client-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch
pulp-deb-admin-extensions-1.8.0-0.1.alpha.201807310516git4df9825.el7.noarch
pulp-deb-plugins-1.8.0-0.1.alpha.201807310516git4df9825.el7.noarch
pulp-docker-admin-extensions-3.2.0-0.1.alpha.201807310503git4356228.el7.noarch
pulp-docker-plugins-3.2.0-0.1.alpha.201807310503git4356228.el7.noarch
pulp-ostree-admin-extensions-1.4.0-0.1.alpha.201807310504git2c2d534.el7.noarch
pulp-ostree-plugins-1.4.0-0.1.alpha.201807310504git2c2d534.el7.noarch
pulp-puppet-admin-extensions-2.17.0-0.1.alpha.201807310506git44a02f8.el7.noarch
pulp-puppet-plugins-2.17.0-0.1.alpha.201807310506git44a02f8.el7.noarch
pulp-puppet-tools-2.17.0-0.1.alpha.201807310506git44a02f8.el7.noarch
pulp-python-admin-extensions-2.1.0-0.1.alpha.201807310501git5e2aa35.el7.noarch
pulp-python-plugins-2.1.0-0.1.alpha.201807310501git5e2aa35.el7.noarch
pulp-rpm-admin-extensions-2.17.0-0.1.alpha.201807310508giteaff2ca.el7.noarch
pulp-rpm-plugins-2.17.0-0.1.alpha.201807310508giteaff2ca.el7.noarch
pulp-selinux-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch
pulp-server-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-pulp-bindings-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch
python-pulp-client-lib-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch
python-pulp-common-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch
python-pulp-deb-common-1.8.0-0.1.alpha.201807310516git4df9825.el7.noarch
python-pulp-docker-common-3.2.0-0.1.alpha.201807310503git4356228.el7.noarch
python-pulp-oid_validation-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch
python-pulp-ostree-common-1.4.0-0.1.alpha.201807310504git2c2d534.el7.noarch
python-pulp-puppet-common-2.17.0-0.1.alpha.201807310506git44a02f8.el7.noarch
python-pulp-python-common-2.1.0-0.1.alpha.201807310501git5e2aa35.el7.noarch
python-pulp-repoauth-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch
python-pulp-rpm-common-2.17.0-0.1.alpha.201807310508giteaff2ca.el7.noarch
python-pulp-streamer-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch
The easiest way to reproduce this issue is to run the automated test in Pulp 2 Tests: python -m unittest pulp_2_tests.tests.docker.api_v2.test_copy.CopyV2ContentTestCase
. The error occurs when this line of code is executing, inside test_02_copy_manifest_lists
:
client.post(urljoin(repo['_href'], 'actions/associate/'), {
'criteria': {'filters': {}, 'type_ids': ['docker_manifest_list']},
'source_repo_id': self.repo['id'],
})
It also occurs when this line of code is executing, inside test_02_copy_tags
:
client.post(urljoin(repo['_href'], 'actions/associate/'), {
'source_repo_id': self.repo['id'],
'criteria': {'filters': {}, 'type_ids': ['docker_tag']},
})
Also see:
Related issues
Fix issue with associating embedded manifests in manifest list
https://pulp.plan.io/issues/3892
re #3137 closes #3892