Issue #3892
Updated by Ichimonji10 over 5 years ago
It's impossible to copy Docker tags or manifest lists between repositories. Attempting to do triggers an error in Pulp. Here's the relevant journalctl output when attempting to copy manifest lists: output: <pre>Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) Exception from importer [docker_importer] while importing units into repository [14053d5e-0ea9-4b13-9e56-f4c769a 0c0a6] Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) Traceback (most recent call last): Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 273, in associate_from_repo Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) units=transfer_units) Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 217, in import_units Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) units_added |= set(unit_importers[type(unit)](import_conduit, unit, dest_repo.repo_obj)) Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 369, in _import_manifest_list Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) for manifest in models.Manifest.objects.filter(digest__in=sorted(manifest_digests)): Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 1407, in next Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) raw_doc = self._cursor.next() Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) if len(self.__data) or self._refresh(): Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1019, in _refresh Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) self.__read_concern)) Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 850, in __send_message Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) **kwargs) Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 794, in _send_message_with_response Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) exhaust) Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 805, in _reset_on_error Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) return func(*args, **kwargs) Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/server.py", line 108, in send_message_with_response Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) set_slave_okay, sock_info.is_mongos, use_find_cmd) Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/message.py", line 275, in get_message Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) spec, self.fields, self.codec_options) Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) InvalidDocument: Cannot encode object: <EmbeddedManifest: EmbeddedManifest object> Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.async.tasks:INFO: [30e85096] Task failed : [30e85096-c215-4b4e-99e6-682dcbede8b1] Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) Task pulp.server.managers.repo.unit_association.associate_from_repo[30e85096-c215-4b4e-99e6-682dcbede8b1] raised unexpected: InvalidDocument('Cannot encode object: <EmbeddedManifest: EmbeddedManifest object>',) Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) Traceback (most recent call last): Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) R = retval = fun(*args, **kwargs) Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 529, in __call__ Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) return super(Task, self).__call__(*args, **kwargs) Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__ Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) return super(PulpTask, self).__call__(*args, **kwargs) Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__ Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) return self.run(*args, **kwargs) Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 291, in associate_from_repo Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) raise (e, None, sys.exc_info()[2]) Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) InvalidDocument: Cannot encode object: <EmbeddedManifest: EmbeddedManifest object> </pre> ...and here's the error when attempting to copy tags between repositories: <pre>Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) Exception from importer [docker_importer] while importing units into repository [94d08260-7dc9-4bf0-9b9a-d23ed19 3c2c3] Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) Traceback (most recent call last): Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 273, in associate_from_repo Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) units=transfer_units) Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 217, in import_units Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) units_added |= set(unit_importers[type(unit)](import_conduit, unit, dest_repo.repo_obj)) Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 299, in _import_tag Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) conduit, manifest, dest_repo)) Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 369, in _import_manifest_list Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) for manifest in models.Manifest.objects.filter(digest__in=sorted(manifest_digests)): Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 1407, in next Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) raw_doc = self._cursor.next() Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) if len(self.__data) or self._refresh(): Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1019, in _refresh Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) self.__read_concern)) Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 850, in __send_message Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) **kwargs) Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 794, in _send_message_with_response Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) exhaust) Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 805, in _reset_on_error Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) return func(*args, **kwargs) Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/server.py", line 108, in send_message_with_response Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) set_slave_okay, sock_info.is_mongos, use_find_cmd) Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/message.py", line 275, in get_message Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) spec, self.fields, self.codec_options) Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) InvalidDocument: Cannot encode object: <EmbeddedManifest: EmbeddedManifest object> Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.async.tasks:INFO: [7c6d1641] Task failed : [7c6d1641-66f4-4408-8a70-6a7d3d2b36ec] Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) Task pulp.server.managers.repo.unit_association.associate_from_repo[7c6d1641-66f4-4408-8a70-6a7d3d2b36ec] raised unexpected: InvalidDocument('Cannot encode object: <EmbeddedManifest: EmbeddedManifest object>',) Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) Traceback (most recent call last): Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) R = retval = fun(*args, **kwargs) Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 529, in __call__ Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) return super(Task, self).__call__(*args, **kwargs) Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__ Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) return super(PulpTask, self).__call__(*args, **kwargs) Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__ Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) return self.run(*args, **kwargs) Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 291, in associate_from_repo Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) raise (e, None, sys.exc_info()[2]) Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) InvalidDocument: Cannot encode object: <EmbeddedManifest: EmbeddedManifest object> </pre> The error looks nearly identical. This error occurs with the nightly build of Pulp 2 (to-be 2.17) on both RHEL 7 and Fedora 27. Here's some sample RPMs: <pre>[root@rhel-7-pulp-2-17-nightly ~]# rpm -qa | grep pulp | sort pulp-admin-client-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch pulp-deb-admin-extensions-1.8.0-0.1.alpha.201807310516git4df9825.el7.noarch pulp-deb-plugins-1.8.0-0.1.alpha.201807310516git4df9825.el7.noarch pulp-docker-admin-extensions-3.2.0-0.1.alpha.201807310503git4356228.el7.noarch pulp-docker-plugins-3.2.0-0.1.alpha.201807310503git4356228.el7.noarch pulp-ostree-admin-extensions-1.4.0-0.1.alpha.201807310504git2c2d534.el7.noarch pulp-ostree-plugins-1.4.0-0.1.alpha.201807310504git2c2d534.el7.noarch pulp-puppet-admin-extensions-2.17.0-0.1.alpha.201807310506git44a02f8.el7.noarch pulp-puppet-plugins-2.17.0-0.1.alpha.201807310506git44a02f8.el7.noarch pulp-puppet-tools-2.17.0-0.1.alpha.201807310506git44a02f8.el7.noarch pulp-python-admin-extensions-2.1.0-0.1.alpha.201807310501git5e2aa35.el7.noarch pulp-python-plugins-2.1.0-0.1.alpha.201807310501git5e2aa35.el7.noarch pulp-rpm-admin-extensions-2.17.0-0.1.alpha.201807310508giteaff2ca.el7.noarch pulp-rpm-plugins-2.17.0-0.1.alpha.201807310508giteaff2ca.el7.noarch pulp-selinux-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch pulp-server-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch python-isodate-0.5.0-4.pulp.el7.noarch python-pulp-bindings-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch python-pulp-client-lib-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch python-pulp-common-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch python-pulp-deb-common-1.8.0-0.1.alpha.201807310516git4df9825.el7.noarch python-pulp-docker-common-3.2.0-0.1.alpha.201807310503git4356228.el7.noarch python-pulp-oid_validation-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch python-pulp-ostree-common-1.4.0-0.1.alpha.201807310504git2c2d534.el7.noarch python-pulp-puppet-common-2.17.0-0.1.alpha.201807310506git44a02f8.el7.noarch python-pulp-python-common-2.1.0-0.1.alpha.201807310501git5e2aa35.el7.noarch python-pulp-repoauth-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch python-pulp-rpm-common-2.17.0-0.1.alpha.201807310508giteaff2ca.el7.noarch python-pulp-streamer-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch </pre> The easiest way to reproduce this issue is to run the automated test in Pulp 2 Tests: <code>python -m unittest pulp_2_tests.tests.docker.api_v2.test_copy.CopyV2ContentTestCase</code>. The error occurs when this line of code is executing, inside <code>test_02_copy_manifest_lists</code>: <pre><code class="python">client.post(urljoin(repo['_href'], 'actions/associate/'), { 'criteria': {'filters': {}, 'type_ids': ['docker_manifest_list']}, 'source_repo_id': self.repo['id'], }) </code></pre> It also occurs when this line of code is executing, inside <code>test_02_copy_tags</code>: <pre><code class="python">client.post(urljoin(repo['_href'], 'actions/associate/'), { 'source_repo_id': self.repo['id'], 'criteria': {'filters': {}, 'type_ids': ['docker_tag']}, }) </pre></code> Also see: * http://pulp-2-tests.readthedocs.io/en/latest/tests/pulp_2_tests.tests.docker.api_v2.test_copy.html#pulp_2_tests.tests.docker.api_v2.test_copy.CopyV2ContentTestCase * https://github.com/PulpQE/Pulp-2-Tests/blob/master/pulp_2_tests/tests/docker/api_v2/test_copy.py
.