Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2018-08-08T21:13:08ZPulp
Planio Pulp - Issue #3908 (CLOSED - CURRENTRELEASE): Add pulp-maintenance package https://pulp.plan.io/issues/39082018-08-08T21:13:08Zdkliban@redhat.com
<p>Pulp's monthly maintenance tasks will only run if the process is not restart for more than 30 days. This new package will install a cron config that will execute the following script:</p>
<pre><code>celery -A pulp.server.async.app call pulp.server.maintenance.monthly.queue_monthly_maintenance
</code></pre> Docker Support - Issue #3905 (CLOSED - CURRENTRELEASE): Purge for unlinked manifests is not worki...https://pulp.plan.io/issues/39052018-08-07T14:01:42Zipanova@redhat.comipanova@redhat.com
<p>Steps to reproduce:</p>
<p>1. pulp-admin docker repo create --repo-id=synctest --feed=<a href="https://registry-1.docker.io" class="external">https://registry-1.docker.io</a> --upstream-name=busybox --tags=latest</p>
<p>2) sync repo</p>
<p>[ipanova@ina pulp_docker]$ pulp-admin docker repo list</p>
<p><span><del>--------------------------------------------------------------------</del></span><br>
Docker Repositories<br>
<span><del>--------------------------------------------------------------------</del></span></p>
<p>Id: synctest<br>
Display Name: None<br>
Description: None<br>
Content Unit Counts:<br>
Docker Blob: 17<br>
Docker Manifest: 9<br>
Docker Manifest List: 1<br>
Docker Tag: 2</p>
<p>3) remove 1 manifest list by specifying its digest</p>
<p>pulp-admin -vv docker repo remove manifest-list --repo-id synctest --str-eq='digest=sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd'</p>
<p>4) observe that number of docker manifests did not change, that means manifests where not purged as they should been</p>
<p>[ipanova@ina pulp_docker]$ pulp-admin docker repo list</p>
<p><span><del>--------------------------------------------------------------------</del></span><br>
Docker Repositories<br>
<span><del>--------------------------------------------------------------------</del></span></p>
<p>Id: synctest<br>
Display Name: None<br>
Description: None<br>
Content Unit Counts:<br>
Docker Blob: 17<br>
Docker Manifest: 9<br>
Docker Manifest List: 0<br>
Docker Tag: 1</p>
<p>Expected behaviour:</p>
<p>4) observe that number of docker manifest decreased, because unlinked manifests were removed with manifest list removal</p>
<p>[ipanova@ina pulp_docker]$ pulp-admin docker repo list</p>
<p><span><del>--------------------------------------------------------------------</del></span><br>
Docker Repositories<br>
<span><del>--------------------------------------------------------------------</del></span></p>
<p>Id: synctest<br>
Display Name: None<br>
Description: None<br>
Content Unit Counts:<br>
Docker Blob: 2<br>
Docker Manifest: 1<br>
Docker Manifest List: 0<br>
Docker Tag: 1</p> Docker Support - Issue #3904 (CLOSED - CURRENTRELEASE): Can't upload manifest list to Docker repo...https://pulp.plan.io/issues/39042018-08-06T20:58:15ZIchimonji10jerebear@protonmail.com
<p>It's impossible to upload a manifest to a Docker repository. This is a regression. Attempting to do triggers an error in Pulp. No errors are logged (according to journalctl). Here's the only interesting journal output I could find:</p>
<pre><code>Aug 06 16:46:32 fedora-27-pulp-2-17-nightly pulp[1208]: celery.worker.strategy:INFO: Received task: pulp.server.managers.content.upload.import_uploaded_unit[daaedf02-3573-41c4-95bd-0035d8a05037]
Aug 06 16:46:32 fedora-27-pulp-2-17-nightly pulp[1315]: celery.app.trace:INFO: [034f7848] Task pulp.server.async.tasks._queue_reserved_task[034f7848-ab90-4ac5-964f-ca4566ba148a] succeeded in 0.0262270519997s: None
Aug 06 16:46:32 fedora-27-pulp-2-17-nightly pulp[1321]: pulp.server.async.tasks:INFO: [daaedf02] Task failed : [daaedf02-3573-41c4-95bd-0035d8a05037] : The importer docker_importer indicated a failed response when uploading docker_manifest_list unit to repository f93cc7df-3f53-4fc6-9ad3-df53c5079748.
</code></pre>
<p>Here's the error returned by Pulp:</p>
<pre><code>{'_href': '/pulp/api/v2/tasks/daaedf02-3573-41c4-95bd-0035d8a05037/',
'_id': {'$oid': '5b68b3a811b6e4e34b496802'},
'_ns': 'task_status',
'error': {'code': 'PLP0047',
'data': {'details': {},
'importer_id': 'docker_importer',
'repo_id': 'f93cc7df-3f53-4fc6-9ad3-df53c5079748',
'summary': "unhashable type: 'EmbeddedManifest'",
'unit_type': 'docker_manifest_list'},
'description': 'The importer docker_importer indicated a failed '
'response when uploading docker_manifest_list unit '
'to repository f93cc7df-3f53-4fc6-9ad3-df53c5079748.',
'sub_errors': []},
'exception': None,
'finish_time': '2018-08-06T20:46:32Z',
'id': '5b68b3a811b6e4e34b496802',
'progress_report': {},
'queue': 'reserved_resource_worker-0@fedora-27-pulp-2-17-nightly.dq2',
'result': None,
'spawned_tasks': [],
'start_time': '2018-08-06T20:46:32Z',
'state': 'error',
'tags': ['pulp:repository:f93cc7df-3f53-4fc6-9ad3-df53c5079748',
'pulp:action:import_upload'],
'task_id': 'daaedf02-3573-41c4-95bd-0035d8a05037',
'task_type': 'pulp.server.managers.content.upload.import_uploaded_unit',
'traceback': SNIP!
'worker_name': 'reserved_resource_worker-0@fedora-27-pulp-2-17-nightly'}
</code></pre>
<p>Here's the formatted traceback:</p>
<pre><code>Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 529, in __call__
return super(Task, self).__call__(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
return super(PulpTask, self).__call__(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py", line 223, in import_uploaded_unit
unit_type=unit_type_id, summary=result['summary'], details=result['details']
PulpCodedException: The importer docker_importer indicated a failed response when uploading docker_manifest_list unit to repository f93cc7df-3f53-4fc6-9ad3-df53c5079748.
</code></pre>
<p>This error occurs with the nightly build of Pulp 2 (to-be 2.17) on both RHEL 7 and Fedora 27. Here's some sample RPMs:</p>
<pre><code>[root@fedora-27-pulp-2-17-nightly ~]# rpm -qa | grep -i 'pulp\|docker' | sort
pulp-admin-client-2.17.0-0.1.alpha.201808060500git0697ade.fc27.noarch
pulp-deb-admin-extensions-1.8.0-0.1.alpha.201808060515git4df9825.fc27.noarch
pulp-deb-plugins-1.8.0-0.1.alpha.201808060515git4df9825.fc27.noarch
pulp-docker-admin-extensions-3.2.0-0.1.alpha.201808060507gitfd8d4e7.fc27.noarch
pulp-docker-plugins-3.2.0-0.1.alpha.201808060507gitfd8d4e7.fc27.noarch
pulp-ostree-admin-extensions-1.4.0-0.1.alpha.201808060508git2c2d534.fc27.noarch
pulp-ostree-plugins-1.4.0-0.1.alpha.201808060508git2c2d534.fc27.noarch
pulp-puppet-admin-extensions-2.17.0-0.1.alpha.201808060510git44a02f8.fc27.noarch
pulp-puppet-plugins-2.17.0-0.1.alpha.201808060510git44a02f8.fc27.noarch
pulp-puppet-tools-2.17.0-0.1.alpha.201808060510git44a02f8.fc27.noarch
pulp-python-admin-extensions-2.1.0-0.1.alpha.201808060514git5e2aa35.fc27.noarch
pulp-python-plugins-2.1.0-0.1.alpha.201808060514git5e2aa35.fc27.noarch
pulp-rpm-admin-extensions-2.17.0-0.1.alpha.201808060504git421caf4.fc27.noarch
pulp-rpm-plugins-2.17.0-0.1.alpha.201808060504git421caf4.fc27.noarch
pulp-selinux-2.17.0-0.1.alpha.201808060500git0697ade.fc27.noarch
pulp-server-2.17.0-0.1.alpha.201808060500git0697ade.fc27.noarch
python-pulp-bindings-2.17.0-0.1.alpha.201808060500git0697ade.fc27.noarch
python-pulp-client-lib-2.17.0-0.1.alpha.201808060500git0697ade.fc27.noarch
python-pulp-common-2.17.0-0.1.alpha.201808060500git0697ade.fc27.noarch
python-pulp-deb-common-1.8.0-0.1.alpha.201808060515git4df9825.fc27.noarch
python-pulp-docker-common-3.2.0-0.1.alpha.201808060507gitfd8d4e7.fc27.noarch
python-pulp-oid_validation-2.17.0-0.1.alpha.201808060500git0697ade.fc27.noarch
python-pulp-ostree-common-1.4.0-0.1.alpha.201808060508git2c2d534.fc27.noarch
python-pulp-puppet-common-2.17.0-0.1.alpha.201808060510git44a02f8.fc27.noarch
python-pulp-python-common-2.1.0-0.1.alpha.201808060514git5e2aa35.fc27.noarch
python-pulp-repoauth-2.17.0-0.1.alpha.201808060500git0697ade.fc27.noarch
python-pulp-rpm-common-2.17.0-0.1.alpha.201808060504git421caf4.fc27.noarch
python-pulp-streamer-2.17.0-0.1.alpha.201808060500git0697ade.fc27.noarch
</code></pre>
<p>The easiest way to reproduce this issue is to run the automated test in Pulp 2 Tests: <code>python -m unittest pulp_2_tests.tests.docker.api_v2.test_upload.UploadManifestListV2TestCase</code>. The error occurs when this line of code is executing, inside <code>test_02_copy_manifest_lists</code>:</p>
<pre><code class="python syntaxhl" data-language="python"><span class="n">upload_import_unit</span><span class="p">(</span>
<span class="bp">self</span><span class="p">.</span><span class="n">cfg</span><span class="p">,</span>
<span class="n">json</span><span class="p">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">modified_manifest_list</span><span class="p">).</span><span class="n">encode</span><span class="p">(</span><span class="s">'utf-8'</span><span class="p">),</span>
<span class="p">{</span><span class="s">'unit_type_id'</span><span class="p">:</span> <span class="s">'docker_manifest_list'</span><span class="p">},</span>
<span class="n">repo</span><span class="p">,</span>
<span class="p">)</span>
</code></pre>
<p>This issue is similar to: <a href="https://pulp.plan.io/issues/3892" class="external">https://pulp.plan.io/issues/3892</a></p> Pulp - Issue #3901 (CLOSED - CURRENTRELEASE): Improve upload error message when RPM is invalidhttps://pulp.plan.io/issues/39012018-08-06T16:57:05Zttereshcttereshc@redhat.com
<p>Currently it's <code>Error reading uploaded package header</code>.<br>
The suggestion is to update it to <code>Uploaded file is not a valid RPM</code></p> Docker Support - Issue #3892 (CLOSED - CURRENTRELEASE): Can't copy Docker tags or manifest lists ...https://pulp.plan.io/issues/38922018-07-31T20:58:09ZIchimonji10jerebear@protonmail.com
<p>It's impossible to copy Docker tags or manifest lists between repositories. Attempting to do triggers an error in Pulp. Here's the relevant journalctl output when attempting to copy manifest lists:</p>
<pre><code>Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) Exception from importer [docker_importer] while importing units into repository [14053d5e-0ea9-4b13-9e56-f4c769a
0c0a6]
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) Traceback (most recent call last):
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 273, in associate_from_repo
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) units=transfer_units)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 217, in import_units
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) units_added |= set(unit_importers[type(unit)](import_conduit, unit, dest_repo.repo_obj))
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 369, in _import_manifest_list
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) for manifest in models.Manifest.objects.filter(digest__in=sorted(manifest_digests)):
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 1407, in next
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) raw_doc = self._cursor.next()
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) if len(self.__data) or self._refresh():
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1019, in _refresh
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) self.__read_concern))
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 850, in __send_message
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) **kwargs)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 794, in _send_message_with_response
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) exhaust)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 805, in _reset_on_error
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) return func(*args, **kwargs)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/server.py", line 108, in send_message_with_response
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) set_slave_okay, sock_info.is_mongos, use_find_cmd)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/message.py", line 275, in get_message
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) spec, self.fields, self.codec_options)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [30e85096] (1877-39168) InvalidDocument: Cannot encode object: <EmbeddedManifest: EmbeddedManifest object>
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.async.tasks:INFO: [30e85096] Task failed : [30e85096-c215-4b4e-99e6-682dcbede8b1]
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) Task pulp.server.managers.repo.unit_association.associate_from_repo[30e85096-c215-4b4e-99e6-682dcbede8b1] raised unexpected: InvalidDocument('Cannot encode object: <EmbeddedManifest: EmbeddedManifest object>',)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) Traceback (most recent call last):
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) R = retval = fun(*args, **kwargs)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 529, in __call__
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) return super(Task, self).__call__(*args, **kwargs)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) return super(PulpTask, self).__call__(*args, **kwargs)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) return self.run(*args, **kwargs)
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 291, in associate_from_repo
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) raise (e, None, sys.exc_info()[2])
Jul 31 16:09:17 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [30e85096] (1877-39168) InvalidDocument: Cannot encode object: <EmbeddedManifest: EmbeddedManifest object>
</code></pre>
<p>...and here's the error when attempting to copy tags between repositories:</p>
<pre><code>Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) Exception from importer [docker_importer] while importing units into repository [94d08260-7dc9-4bf0-9b9a-d23ed19
3c2c3]
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) Traceback (most recent call last):
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 273, in associate_from_repo
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) units=transfer_units)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 217, in import_units
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) units_added |= set(unit_importers[type(unit)](import_conduit, unit, dest_repo.repo_obj))
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 299, in _import_tag
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) conduit, manifest, dest_repo))
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 369, in _import_manifest_list
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) for manifest in models.Manifest.objects.filter(digest__in=sorted(manifest_digests)):
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 1407, in next
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) raw_doc = self._cursor.next()
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1097, in next
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) if len(self.__data) or self._refresh():
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 1019, in _refresh
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) self.__read_concern))
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 850, in __send_message
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) **kwargs)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 794, in _send_message_with_response
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) exhaust)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 805, in _reset_on_error
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) return func(*args, **kwargs)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/server.py", line 108, in send_message_with_response
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) set_slave_okay, sock_info.is_mongos, use_find_cmd)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) File "/usr/lib64/python2.7/site-packages/pymongo/message.py", line 275, in get_message
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) spec, self.fields, self.codec_options)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.managers.repo.unit_association:ERROR: [7c6d1641] (1877-39168) InvalidDocument: Cannot encode object: <EmbeddedManifest: EmbeddedManifest object>
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: pulp.server.async.tasks:INFO: [7c6d1641] Task failed : [7c6d1641-66f4-4408-8a70-6a7d3d2b36ec]
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) Task pulp.server.managers.repo.unit_association.associate_from_repo[7c6d1641-66f4-4408-8a70-6a7d3d2b36ec] raised unexpected: InvalidDocument('Cannot encode object: <EmbeddedManifest: EmbeddedManifest object>',)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) Traceback (most recent call last):
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) R = retval = fun(*args, **kwargs)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 529, in __call__
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) return super(Task, self).__call__(*args, **kwargs)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) return super(PulpTask, self).__call__(*args, **kwargs)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) return self.run(*args, **kwargs)
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association.py", line 291, in associate_from_repo
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) raise (e, None, sys.exc_info()[2])
Jul 31 17:02:35 rhel-7-pulp-2-17-nightly pulp[1877]: celery.app.trace:ERROR: [7c6d1641] (1877-39168) InvalidDocument: Cannot encode object: <EmbeddedManifest: EmbeddedManifest object>
</code></pre>
<p>The error looks nearly identical. This error occurs with the nightly build of Pulp 2 (to-be 2.17) on both RHEL 7 and Fedora 27. Here's some sample RPMs:</p>
<pre><code>[root@rhel-7-pulp-2-17-nightly ~]# rpm -qa | grep pulp | sort
pulp-admin-client-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch
pulp-deb-admin-extensions-1.8.0-0.1.alpha.201807310516git4df9825.el7.noarch
pulp-deb-plugins-1.8.0-0.1.alpha.201807310516git4df9825.el7.noarch
pulp-docker-admin-extensions-3.2.0-0.1.alpha.201807310503git4356228.el7.noarch
pulp-docker-plugins-3.2.0-0.1.alpha.201807310503git4356228.el7.noarch
pulp-ostree-admin-extensions-1.4.0-0.1.alpha.201807310504git2c2d534.el7.noarch
pulp-ostree-plugins-1.4.0-0.1.alpha.201807310504git2c2d534.el7.noarch
pulp-puppet-admin-extensions-2.17.0-0.1.alpha.201807310506git44a02f8.el7.noarch
pulp-puppet-plugins-2.17.0-0.1.alpha.201807310506git44a02f8.el7.noarch
pulp-puppet-tools-2.17.0-0.1.alpha.201807310506git44a02f8.el7.noarch
pulp-python-admin-extensions-2.1.0-0.1.alpha.201807310501git5e2aa35.el7.noarch
pulp-python-plugins-2.1.0-0.1.alpha.201807310501git5e2aa35.el7.noarch
pulp-rpm-admin-extensions-2.17.0-0.1.alpha.201807310508giteaff2ca.el7.noarch
pulp-rpm-plugins-2.17.0-0.1.alpha.201807310508giteaff2ca.el7.noarch
pulp-selinux-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch
pulp-server-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-pulp-bindings-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch
python-pulp-client-lib-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch
python-pulp-common-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch
python-pulp-deb-common-1.8.0-0.1.alpha.201807310516git4df9825.el7.noarch
python-pulp-docker-common-3.2.0-0.1.alpha.201807310503git4356228.el7.noarch
python-pulp-oid_validation-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch
python-pulp-ostree-common-1.4.0-0.1.alpha.201807310504git2c2d534.el7.noarch
python-pulp-puppet-common-2.17.0-0.1.alpha.201807310506git44a02f8.el7.noarch
python-pulp-python-common-2.1.0-0.1.alpha.201807310501git5e2aa35.el7.noarch
python-pulp-repoauth-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch
python-pulp-rpm-common-2.17.0-0.1.alpha.201807310508giteaff2ca.el7.noarch
python-pulp-streamer-2.17.0-0.1.alpha.201807310510git0697ade.el7.noarch
</code></pre>
<p>The easiest way to reproduce this issue is to run the automated test in Pulp 2 Tests: <code>python -m unittest pulp_2_tests.tests.docker.api_v2.test_copy.CopyV2ContentTestCase</code>. The error occurs when this line of code is executing, inside <code>test_02_copy_manifest_lists</code>:</p>
<pre><code class="python syntaxhl" data-language="python"><span class="n">client</span><span class="p">.</span><span class="n">post</span><span class="p">(</span><span class="n">urljoin</span><span class="p">(</span><span class="n">repo</span><span class="p">[</span><span class="s">'_href'</span><span class="p">],</span> <span class="s">'actions/associate/'</span><span class="p">),</span> <span class="p">{</span>
<span class="s">'criteria'</span><span class="p">:</span> <span class="p">{</span><span class="s">'filters'</span><span class="p">:</span> <span class="p">{},</span> <span class="s">'type_ids'</span><span class="p">:</span> <span class="p">[</span><span class="s">'docker_manifest_list'</span><span class="p">]},</span>
<span class="s">'source_repo_id'</span><span class="p">:</span> <span class="bp">self</span><span class="p">.</span><span class="n">repo</span><span class="p">[</span><span class="s">'id'</span><span class="p">],</span>
<span class="p">})</span>
</code></pre>
<p>It also occurs when this line of code is executing, inside <code>test_02_copy_tags</code>:</p>
<pre><code class="python syntaxhl" data-language="python"><span class="n">client</span><span class="p">.</span><span class="n">post</span><span class="p">(</span><span class="n">urljoin</span><span class="p">(</span><span class="n">repo</span><span class="p">[</span><span class="s">'_href'</span><span class="p">],</span> <span class="s">'actions/associate/'</span><span class="p">),</span> <span class="p">{</span>
<span class="s">'source_repo_id'</span><span class="p">:</span> <span class="bp">self</span><span class="p">.</span><span class="n">repo</span><span class="p">[</span><span class="s">'id'</span><span class="p">],</span>
<span class="s">'criteria'</span><span class="p">:</span> <span class="p">{</span><span class="s">'filters'</span><span class="p">:</span> <span class="p">{},</span> <span class="s">'type_ids'</span><span class="p">:</span> <span class="p">[</span><span class="s">'docker_tag'</span><span class="p">]},</span>
<span class="p">})</span>
</code></pre>
<p>Also see:</p>
<ul>
<li><a href="http://pulp-2-tests.readthedocs.io/en/latest/tests/pulp_2_tests.tests.docker.api_v2.test_copy.html#pulp_2_tests.tests.docker.api_v2.test_copy.CopyV2ContentTestCase" class="external">http://pulp-2-tests.readthedocs.io/en/latest/tests/pulp_2_tests.tests.docker.api_v2.test_copy.html#pulp_2_tests.tests.docker.api_v2.test_copy.CopyV2ContentTestCase</a></li>
<li><a href="https://github.com/PulpQE/Pulp-2-Tests/blob/master/pulp_2_tests/tests/docker/api_v2/test_copy.py" class="external">https://github.com/PulpQE/Pulp-2-Tests/blob/master/pulp_2_tests/tests/docker/api_v2/test_copy.py</a></li>
</ul> Pulp - Issue #3887 (CLOSED - CURRENTRELEASE): Restarting pulp_celerybeat weekly or so causes pulp...https://pulp.plan.io/issues/38872018-07-30T18:33:58Zttereshcttereshc@redhat.com
<a name="Symptoms"></a>
<h2 >Symptoms<a href="#Symptoms" class="wiki-anchor">¶</a></h2>
<p>No signs of `pulp.server.maintenance.monthly` in the logs for the last 30 days.<br>
If task is triggered manually, it works properly and orphaned applicability profiles got cleaned up.</p>
<a name="Impact"></a>
<h2 >Impact<a href="#Impact" class="wiki-anchor">¶</a></h2>
<p>Having a lot of orphaned applicability profiles triggers a lot of no-op applicability tasks, it can slow down the whole system if there are too many of those tasks. E.g. A repo with many bound consumers (30K unique profiles) will trigger 3K tasks.</p>
<a name="Root-Cause"></a>
<h2 >Root Cause<a href="#Root-Cause" class="wiki-anchor">¶</a></h2>
<p>Pulp users Celery's periodic tasks feature for its dispatching. The monthly maintenance is dispatched every 30 days. Unfortunately Celery measues this as 30 days since the pulp_celerybeat process started. See <a href="http://docs.celeryproject.org/en/v4.0.2/userguide/periodic-tasks.html#beat-entries" class="external">the celery docs</a> This means that if you restart your pulp_celerybeat every now and then, you'll end up with a lot of orphaned applicability profiles.</p>
<a name="Resolution"></a>
<h2 >Resolution<a href="#Resolution" class="wiki-anchor">¶</a></h2>
<p>The recommendation is to use cron to run a script monthly like:</p>
<pre><code>from pulp.server.maintenance.monthly import queue_monthly_maintenance
queue_monthly_maintenance.apply_async()
</code></pre> Packaging - Issue #3786 (CLOSED - CURRENTRELEASE): Incorrect Package Requiremetnshttps://pulp.plan.io/issues/37862018-06-25T09:31:15Zwibbit
<p>Morning All</p>
<p>I'm not sure if this is the correct place to raise this issue, apologies if it is not.</p>
<p>The below link references a requirement on python-qpqid.</p>
<p><a href="https://docs.pulpproject.org/en/2.16/user-guide/installation/f23-.html" class="external">https://docs.pulpproject.org/en/2.16/user-guide/installation/f23-.html</a></p>
<p>python-qpid has now been replaced by python2-qpid in rhel7.5 (possibly other versions, I'm not sure).</p> Pulp - Story #3780 (CLOSED - CURRENTRELEASE): As a user, I can specify a 'ca_path' for the HTTP N...https://pulp.plan.io/issues/37802018-06-21T20:04:39Zdkliban@redhat.com
<p>The HTTP notifier makes a call back to a particular URL when a particular event occurs (e.g. sync or publish completes). A user can unknowingly install packages that change what CA store is used by Python when making requests. As a result the notifier stops working properly.</p>
<p>This feature will allow the user to specify the 'ca_path' when creating a notifier with a POST request[0] to /pulp/api/v2/events/. The CA at the 'ca_path' will then be used by the notifier when making a request to the URL specified in the notifier config[1]. When the 'ca_path' is not specified, the system CA store is used.</p>
<p>When the user retrieves a single notifier via REST API or lists the notifiers via REST API, the 'ca_path' needs to be included in the response.</p>
<p>The user should be able to update the notfier's ca_path.</p>
<p>[0] <a href="https://docs.pulpproject.org/dev-guide/integration/rest-api/event/crud.html#create-an-event-listener" class="external">https://docs.pulpproject.org/dev-guide/integration/rest-api/event/crud.html#create-an-event-listener</a><br>
[1] <a href="https://docs.pulpproject.org/dev-guide/integration/events/http.html#http-notifier" class="external">https://docs.pulpproject.org/dev-guide/integration/events/http.html#http-notifier</a></p> Pulp - Story #3760 (CLOSED - CURRENTRELEASE): As a user of export distributor, I have a file with...https://pulp.plan.io/issues/37602018-06-12T18:49:31Zrabajaj
<p>As a user, I would like to be able to know the checksum of the ISO generated by the export distributor.</p>
<p>After generating an ISO, the export distributor will calculate md5, sha1, sha256, and sha512 checksums of the ISO and write them to a file with the same name as the ISO, but with a '.DIGESTS' extension. The file will have the following format:</p>
<pre><code>bb16e10946c7d24c5bbf989c3877d0dd *repo_export.iso
bd5f9d38232bd31dc7334b70e4259f68ab6327f4 *repo_export.iso
eb93b60b4be097fd618dbdd625f70ee64a6a77d502e50b39d2c259df9ccb8f53 *repo_export.iso
a92d64650c862bf89643dceff172c8762212c24bcb815a33f3356b63538392ee50ba6aa07188b3d1a2a760be7e9a9b0631520768c1c0a6ce60b024a3a8bc766b *repo_export.iso
</code></pre> Docker Support - Issue #3757 (CLOSED - CURRENTRELEASE): If upload fails partial manifest is assoc...https://pulp.plan.io/issues/37572018-06-11T10:37:04Zipanova@redhat.comipanova@redhat.com
<pre><code>[ipanova@ina pulp]$ pulp-admin docker repo create --repo-id 1
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/
Repository [1] successfully created
[ipanova@ina pulp]$ pulp-admin docker repo uploads upload --repo-id 1 --file /home/ipanova/myprojects/tips/image-name-1.tar
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/
+----------------------------------------------------------------------+
Unit Upload
+----------------------------------------------------------------------+
Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: image-name-1.tar
... completed
Creating upload requests on the server...
[==================================================] 100%
Initializing: image-name-1.tar
... completed
Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
canceled entirely using the cancel command.
Uploading: image-name-1.tar
[==================================================] 100%
3696640/3696640 bytes
... completed
Importing into the repository...
This command may be exited via ctrl+c without affecting the request.
[\]
Running...
Task Failed
Layer 7cc9ff24526dfffdb8c188e6bae7e11d75d1ac5731d85e6fdccd5c7382d84207 is not
present in the image
Deleting the upload request...
... completed
[ipanova@ina pulp]$ pulp-admin docker repo list
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/
+----------------------------------------------------------------------+
Docker Repositories
+----------------------------------------------------------------------+
Id: 1
Display Name: None
Description: None
Content Unit Counts:
Docker Manifest: 1
[ipanova@ina pulp]$ pulp-admin docker repo uploads upload --repo-id 1 --file /home/ipanova/myprojects/tips/image-name-1.tar
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/
+----------------------------------------------------------------------+
Unit Upload
+----------------------------------------------------------------------+
Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: image-name-1.tar
... completed
Creating upload requests on the server...
[==================================================] 100%
Initializing: image-name-1.tar
... completed
Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
canceled entirely using the cancel command.
Uploading: image-name-1.tar
[==================================================] 100%
3696640/3696640 bytes
... completed
Importing into the repository...
This command may be exited via ctrl+c without affecting the request.
[\]
Running...
Task Failed
Layer 7cc9ff24526dfffdb8c188e6bae7e11d75d1ac5731d85e6fdccd5c7382d84207 is not
present in the image
Deleting the upload request...
... completed
[ipanova@ina pulp]$ pulp-admin docker repo list
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/
+----------------------------------------------------------------------+
Docker Repositories
+----------------------------------------------------------------------+
Id: 1
Display Name: None
Description: None
Content Unit Counts:
Docker Blob: 3
Docker Manifest: 1
[ipanova@ina pulp]$
</code></pre> Pulp - Issue #3735 (CLOSED - CURRENTRELEASE): Update documentation to use "2-master" branchhttps://pulp.plan.io/issues/37352018-06-04T20:11:53Zamacdona@redhat.comaustin@redhat.com
<p>This is a list of documentation that needs to be updated after "2-master" instead of "master"</p> Docker Support - Story #3497 (CLOSED - CURRENTRELEASE): As a user, I can upload skopeo tar that r...https://pulp.plan.io/issues/34972018-03-16T14:13:14Ztomckay@redhat.comtomckay@redhat.com
<p><strong>Motivation:</strong></p>
<p>To save bandwidth and improve efficiency, layers referred to in a skopeo tar manifest.json that already exist should be allowed to be omitted from the tar.</p>
<p>As I am implementing "docker push" and "skopeo copy docker:", both tools first check if a layer exists before attempting to push it. Since pulp expects all the layers to be part of the upload this leads to having to get the layer from pulp, add it to the tar file, then give tar file to pulp.</p>
<p><strong>Solution:</strong></p>
<p>The upload API will support uploading archives that contain only a subset of layers needed to create the full container image. Archive must contain manifest.json file. When such an archive is uploaded to Pulp, it will be added to the repository only if the missing layers are already present in the repository. If the layers are missing from both the archive being uploaded and repository the upload will fail.</p> Docker Support - Story #3450 (CLOSED - CURRENTRELEASE): As a user I would like to limit the tags ...https://pulp.plan.io/issues/34502018-03-09T18:19:29Zjmontleo@redhat.comjmontleo@redhat.com
<p><strong>Motivation:</strong></p>
<p>We're currently sync'ing some very large openshift repos from brew. In reality we're only interested in the current 3.9 and newerreleases/tags but we're getting everything, which is weighing in at 1.2TB at present and growing with each puddle for each version.</p>
<p>It would be nice to be able to limit the tags we sync.</p>
<p><strong>Solution:</strong></p>
<p>Provide a config on the DockerImporter config called "tags". This config will accept a list of tags to sync.</p>
<p>Provide a CLI option called "--tags". The user will be able to specify "--tags" when creating a docker repository. The user will also be able to override the "--tags" importer config at sync time, by specifying it as part of the override_configs.</p>
<p><span>When the "tags" option is provided:</span></p>
<p>Pulp will sync just those tags + manifests associated with the tags and all the corresponding layers. Sync will only be performed using the immediate download policy. There will be no support for the on_demand download policy.</p>
<p><strong>Note:</strong> If the list of tags was changed, and tag X is not wanted anymore, a manual removal of that tag X should occur in the local repo.<br>
<strong>Note:</strong> If some invalid tags provided are invalid, Pulp will only sync the available ones in the remote repo.</p>
<p><span>When the "tags" option is not provided:</span><br>
Pulp will sync the whole repo.</p> Pulp - Issue #3253 (CLOSED - CURRENTRELEASE): UnicodeDecodeError in case of a non-ASCII character...https://pulp.plan.io/issues/32532018-01-06T20:38:34Zmansari
<pre><code class="python syntaxhl" data-language="python"><span class="n">Nov</span> <span class="mi">16</span> <span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">38</span> <span class="n">ilnxcapl01</span> <span class="n">pulp</span><span class="p">:</span> <span class="n">pulp</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">controllers</span><span class="p">.</span><span class="n">repository</span><span class="p">:</span><span class="n">ERROR</span><span class="p">:</span> <span class="p">(</span><span class="mi">29604</span><span class="o">-</span><span class="mi">74656</span><span class="p">)</span> <span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span>
<span class="n">Nov</span> <span class="mi">16</span> <span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">38</span> <span class="n">ilnxcapl01</span> <span class="n">pulp</span><span class="p">:</span> <span class="n">pulp</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">controllers</span><span class="p">.</span><span class="n">repository</span><span class="p">:</span><span class="n">ERROR</span><span class="p">:</span> <span class="p">(</span><span class="mi">29604</span><span class="o">-</span><span class="mi">74656</span><span class="p">)</span> <span class="n">File</span> <span class="s">"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">411</span><span class="p">,</span> <span class="ow">in</span> <span class="n">create_repo</span>
<span class="n">Nov</span> <span class="mi">16</span> <span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">38</span> <span class="n">ilnxcapl01</span> <span class="n">pulp</span><span class="p">:</span> <span class="n">pulp</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">controllers</span><span class="p">.</span><span class="n">repository</span><span class="p">:</span><span class="n">ERROR</span><span class="p">:</span> <span class="p">(</span><span class="mi">29604</span><span class="o">-</span><span class="mi">74656</span><span class="p">)</span> <span class="n">importer_controller</span><span class="p">.</span><span class="n">set_importer</span><span class="p">(</span><span class="n">repo_id</span><span class="p">,</span> <span class="n">importer_type_id</span><span class="p">,</span> <span class="n">importer_repo_plugin_config</span><span class="p">)</span>
<span class="n">Nov</span> <span class="mi">16</span> <span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">38</span> <span class="n">ilnxcapl01</span> <span class="n">pulp</span><span class="p">:</span> <span class="n">pulp</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">controllers</span><span class="p">.</span><span class="n">repository</span><span class="p">:</span><span class="n">ERROR</span><span class="p">:</span> <span class="p">(</span><span class="mi">29604</span><span class="o">-</span><span class="mi">74656</span><span class="p">)</span> <span class="n">File</span> <span class="s">"/usr/lib/python2.7/site-packages/celery/local.py"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">167</span><span class="p">,</span> <span class="ow">in</span> <span class="o"><</span><span class="k">lambda</span><span class="o">></span>
<span class="n">Nov</span> <span class="mi">16</span> <span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">38</span> <span class="n">ilnxcapl01</span> <span class="n">pulp</span><span class="p">:</span> <span class="n">pulp</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">controllers</span><span class="p">.</span><span class="n">repository</span><span class="p">:</span><span class="n">ERROR</span><span class="p">:</span> <span class="p">(</span><span class="mi">29604</span><span class="o">-</span><span class="mi">74656</span><span class="p">)</span> <span class="n">__call__</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="o">*</span><span class="n">a</span><span class="p">,</span> <span class="o">**</span><span class="n">kw</span><span class="p">:</span> <span class="n">x</span><span class="p">.</span><span class="n">_get_current_object</span><span class="p">()(</span><span class="o">*</span><span class="n">a</span><span class="p">,</span> <span class="o">**</span><span class="n">kw</span><span class="p">)</span>
<span class="n">Nov</span> <span class="mi">16</span> <span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">38</span> <span class="n">ilnxcapl01</span> <span class="n">pulp</span><span class="p">:</span> <span class="n">pulp</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">controllers</span><span class="p">.</span><span class="n">repository</span><span class="p">:</span><span class="n">ERROR</span><span class="p">:</span> <span class="p">(</span><span class="mi">29604</span><span class="o">-</span><span class="mi">74656</span><span class="p">)</span> <span class="n">File</span> <span class="s">"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">473</span><span class="p">,</span> <span class="ow">in</span> <span class="n">__call__</span>
<span class="n">Nov</span> <span class="mi">16</span> <span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">38</span> <span class="n">ilnxcapl01</span> <span class="n">pulp</span><span class="p">:</span> <span class="n">pulp</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">controllers</span><span class="p">.</span><span class="n">repository</span><span class="p">:</span><span class="n">ERROR</span><span class="p">:</span> <span class="p">(</span><span class="mi">29604</span><span class="o">-</span><span class="mi">74656</span><span class="p">)</span> <span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">Task</span><span class="p">,</span> <span class="bp">self</span><span class="p">).</span><span class="n">__call__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="n">Nov</span> <span class="mi">16</span> <span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">38</span> <span class="n">ilnxcapl01</span> <span class="n">pulp</span><span class="p">:</span> <span class="n">pulp</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">controllers</span><span class="p">.</span><span class="n">repository</span><span class="p">:</span><span class="n">ERROR</span><span class="p">:</span> <span class="p">(</span><span class="mi">29604</span><span class="o">-</span><span class="mi">74656</span><span class="p">)</span> <span class="n">File</span> <span class="s">"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">103</span><span class="p">,</span> <span class="ow">in</span> <span class="n">__call__</span>
<span class="n">Nov</span> <span class="mi">16</span> <span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">38</span> <span class="n">ilnxcapl01</span> <span class="n">pulp</span><span class="p">:</span> <span class="n">pulp</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">controllers</span><span class="p">.</span><span class="n">repository</span><span class="p">:</span><span class="n">ERROR</span><span class="p">:</span> <span class="p">(</span><span class="mi">29604</span><span class="o">-</span><span class="mi">74656</span><span class="p">)</span> <span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">PulpTask</span><span class="p">,</span> <span class="bp">self</span><span class="p">).</span><span class="n">__call__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="n">Nov</span> <span class="mi">16</span> <span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">38</span> <span class="n">ilnxcapl01</span> <span class="n">pulp</span><span class="p">:</span> <span class="n">pulp</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">controllers</span><span class="p">.</span><span class="n">repository</span><span class="p">:</span><span class="n">ERROR</span><span class="p">:</span> <span class="p">(</span><span class="mi">29604</span><span class="o">-</span><span class="mi">74656</span><span class="p">)</span> <span class="n">File</span> <span class="s">"/usr/lib/python2.7/site-packages/celery/app/task.py"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">420</span><span class="p">,</span> <span class="ow">in</span> <span class="n">__call__</span>
<span class="n">Nov</span> <span class="mi">16</span> <span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">38</span> <span class="n">ilnxcapl01</span> <span class="n">pulp</span><span class="p">:</span> <span class="n">pulp</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">controllers</span><span class="p">.</span><span class="n">repository</span><span class="p">:</span><span class="n">ERROR</span><span class="p">:</span> <span class="p">(</span><span class="mi">29604</span><span class="o">-</span><span class="mi">74656</span><span class="p">)</span> <span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="n">Nov</span> <span class="mi">16</span> <span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">38</span> <span class="n">ilnxcapl01</span> <span class="n">pulp</span><span class="p">:</span> <span class="n">pulp</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">controllers</span><span class="p">.</span><span class="n">repository</span><span class="p">:</span><span class="n">ERROR</span><span class="p">:</span> <span class="p">(</span><span class="mi">29604</span><span class="o">-</span><span class="mi">74656</span><span class="p">)</span> <span class="n">File</span> <span class="s">"/usr/lib/python2.7/site-packages/pulp/server/controllers/importer.py"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">112</span><span class="p">,</span> <span class="ow">in</span> <span class="n">set_importer</span>
<span class="n">Nov</span> <span class="mi">16</span> <span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">38</span> <span class="n">ilnxcapl01</span> <span class="n">pulp</span><span class="p">:</span> <span class="n">pulp</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">controllers</span><span class="p">.</span><span class="n">repository</span><span class="p">:</span><span class="n">ERROR</span><span class="p">:</span> <span class="p">(</span><span class="mi">29604</span><span class="o">-</span><span class="mi">74656</span><span class="p">)</span> <span class="n">importer</span><span class="p">.</span><span class="n">save</span><span class="p">()</span>
<span class="n">Nov</span> <span class="mi">16</span> <span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">38</span> <span class="n">ilnxcapl01</span> <span class="n">pulp</span><span class="p">:</span> <span class="n">pulp</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">controllers</span><span class="p">.</span><span class="n">repository</span><span class="p">:</span><span class="n">ERROR</span><span class="p">:</span> <span class="p">(</span><span class="mi">29604</span><span class="o">-</span><span class="mi">74656</span><span class="p">)</span> <span class="n">File</span> <span class="s">"/usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">283</span><span class="p">,</span> <span class="ow">in</span> <span class="n">save</span>
<span class="n">Nov</span> <span class="mi">16</span> <span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">38</span> <span class="n">ilnxcapl01</span> <span class="n">pulp</span><span class="p">:</span> <span class="n">pulp</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">controllers</span><span class="p">.</span><span class="n">repository</span><span class="p">:</span><span class="n">ERROR</span><span class="p">:</span> <span class="p">(</span><span class="mi">29604</span><span class="o">-</span><span class="mi">74656</span><span class="p">)</span> <span class="bp">self</span><span class="p">.</span><span class="n">_write_pem_file</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">path</span><span class="p">)</span>
<span class="n">Nov</span> <span class="mi">16</span> <span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">38</span> <span class="n">ilnxcapl01</span> <span class="n">pulp</span><span class="p">:</span> <span class="n">pulp</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">controllers</span><span class="p">.</span><span class="n">repository</span><span class="p">:</span><span class="n">ERROR</span><span class="p">:</span> <span class="p">(</span><span class="mi">29604</span><span class="o">-</span><span class="mi">74656</span><span class="p">)</span> <span class="n">File</span> <span class="s">"/usr/lib/python2.7/site-packages/pulp/server/db/model/__init__.py"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">347</span><span class="p">,</span> <span class="ow">in</span> <span class="n">_write_pem_file</span>
<span class="n">Nov</span> <span class="mi">16</span> <span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">38</span> <span class="n">ilnxcapl01</span> <span class="n">pulp</span><span class="p">:</span> <span class="n">pulp</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">controllers</span><span class="p">.</span><span class="n">repository</span><span class="p">:</span><span class="n">ERROR</span><span class="p">:</span> <span class="p">(</span><span class="mi">29604</span><span class="o">-</span><span class="mi">74656</span><span class="p">)</span> <span class="n">pem_file</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">config</span><span class="p">[</span><span class="n">config_key</span><span class="p">].</span><span class="n">encode</span><span class="p">(</span><span class="s">'utf-8'</span><span class="p">))</span>
<span class="n">Nov</span> <span class="mi">16</span> <span class="mi">08</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">38</span> <span class="n">ilnxcapl01</span> <span class="n">pulp</span><span class="p">:</span> <span class="n">pulp</span><span class="p">.</span><span class="n">server</span><span class="p">.</span><span class="n">controllers</span><span class="p">.</span><span class="n">repository</span><span class="p">:</span><span class="n">ERROR</span><span class="p">:</span> <span class="p">(</span><span class="mi">29604</span><span class="o">-</span><span class="mi">74656</span><span class="p">)</span> <span class="nb">UnicodeDecodeError</span><span class="p">:</span> <span class="s">'ascii'</span> <span class="n">codec</span> <span class="n">can</span><span class="s">'t decode byte 0xc3 in position 58892: ordinal not in range(128)
Nov 16 08:00:38 ilnxcapl01 pulp: pulp.server.controllers.repository:ERROR: (29604-21568) Exception adding importer to repo [belastingdienst-Production-soe-rhel7-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_-_RH_Common_RPMs_x86_64_7Server]; the repo will be deleted
</span></code></pre>
<p>It happens when Pulp is <a href="https://github.com/pulp/pulp/blob/master/server/pulp/server/db/model/__init__.py#L362" class="external">trying to write a provided SSL cert/key/ on a disk</a></p> Docker Support - Story #3137 (CLOSED - CURRENTRELEASE): As a user, I can view docker image Arch a...https://pulp.plan.io/issues/31372017-11-15T15:12:59Zbreilly
<p>As the result of this story it will be possible to see:</p>
<p>- arch and os of each image manifest listed within the manifest list queried trough Pulp API. Information will be added to the ManifestList model.<br>
- size of each blob listed within the image manifest queried trough Pulp API. Information will be added to the Manifest model.</p>
<p>The information ( if available) will be taken from the manifest.json or manifest_list.json file returned from the registry.</p>
<p>Possible example of manifest list representation:</p>
<pre><code> [
{
"updated": "2018-06-22T15:19:30Z",
"repo_id": "ina",
"created": "2018-06-22T15:19:30Z",
"unit_id": "06757e06-1c3c-4c4e-82d7-63fefb184935",
"unit_type_id": "docker_manifest_list",
"_id": {
"$oid": "5b2d1382c46f69d5b7f5258d"
},
"metadata": {
"_storage_path": "/var/lib/pulp/content/units/docker_manifest_list/43/a3683c58bce95e87474617e309bf1182a7a157c8d4616c44a8bf4ccc811e63/sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47",
"_id": "06757e06-1c3c-4c4e-82d7-63fefb184935",
"_ns": "units_docker_manifest_list",
"_last_updated": 1529680770,
"amd64_schema_version": 2,
"schema_version": 2,
"manifests": [
{"digest": "sha256:74f634b1bc1bd74535d5209589734efbd44a25f4e2dc96d78784576a3eb5b335",
"os": "linux",
"arch": "arm64"
},
{"digest": "sha256:35e28b647bd4976b7cacfaa32b7b253817d0881d77b6cda731ad46a29d08c2cb",
"os" :"linux",
"arch": "ppc64le"
},
{"digest: "sha256:420befcb0c197618f0252108d553d8a112e291e2a6a75d8a2b4933f511480ea3",
"os": "linux",
"arch": "s390x"
}
],
"downloaded": true,
"pulp_user_metadata": {},
"_content_type_id": "docker_manifest_list",
"amd64_digest": "sha256:74f634b1bc1bd74535d5209589734efbd44a25f4e2dc96d78784576a3eb5b335",
"digest": "sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47"
}
}
]
</code></pre>
<p>Possible example of image manifest representation:</p>
<pre><code>[
{
"updated": "2018-06-22T15:19:30Z",
"repo_id": "ina",
"created": "2018-06-22T15:19:30Z",
"unit_id": "0c966ae4-320d-43a0-a11c-42b3aa968824",
"unit_type_id": "docker_manifest",
"_id": {
"$oid": "5b2d1382c46f69d5b7f52590"
},
"metadata": {
"_storage_path": "/var/lib/pulp/content/units/docker_manifest/db/a1aa28355611292550bdfe7fcd7bcc793cc85892bffdb1ef1792c466960539/sha256:420befcb0c197618f0252108d553d8a112e291e2a6a75d8a2b4933f511480ea3",
"_ns": "units_docker_manifest",
"_last_updated": 1529680770,
"fs_layers": [
{
"layer_type": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"blob_sum": "sha256:eac115b78508fb249ed57f4e00662f90304fa86496413f393fdc1e2d5d793d83",
"size" : 1234567
}
],
"schema_version": 2,
"downloaded": true,
"pulp_user_metadata": {},
"_content_type_id": "docker_manifest",
"config_layer": "sha256:e6631e66b5519df3fb0ff6e5dc29b168f1d7adb9da3c04092f1894e15c2ca2b3",
"_id": "0c966ae4-320d-43a0-a11c-42b3aa968824",
"digest": "sha256:420befcb0c197618f0252108d553d8a112e291e2a6a75d8a2b4933f511480ea3"
}
},
]
</code></pre>
<p><strong>Note</strong> because none of the above fields( size, os, arch) are required by the registry API specification, they can be missing from the json file, that means - the values in Pulp are populated based on the availability of those in the json file returned from the registry.<br>
For more information on what is defined on manifest list, image manifest schema2, image manifest schema1 check specs [0][1][2]<br>
[0] <a href="https://docs.docker.com/registry/spec/manifest-v2-2/#example-manifest-list" class="external">https://docs.docker.com/registry/spec/manifest-v2-2/#example-manifest-list</a><br>
[1] <a href="https://docs.docker.com/registry/spec/manifest-v2-2/#example-image-manifest" class="external">https://docs.docker.com/registry/spec/manifest-v2-2/#example-image-manifest</a><br>
[2] <a href="https://docs.docker.com/registry/spec/manifest-v2-1/#example-manifest" class="external">https://docs.docker.com/registry/spec/manifest-v2-1/#example-manifest</a></p>