Debian Support

Well that's annoying, I'm not sure how that got created, any way, hopefully the below is the required information.

[dfurlong@snjlv-repo01 ~]$ rpm -qa | grep pulp|sort pulp-admin-client-2.20.0-2.el7.noarch pulp-deb-admin-extensions-1.10.0-1.el7.noarch pulp-deb-plugins-1.10.0-1.el7.noarch pulp-rpm-admin-extensions-2.20.0-2.el7.noarch pulp-rpm-plugins-2.20.0-2.el7.noarch pulp-selinux-2.20.0-2.el7.noarch pulp-server-2.20.0-2.el7.noarch python-isodate-0.5.0-4.pulp.el7.noarch python-pulp-bindings-2.20.0-2.el7.noarch python-pulp-client-lib-2.20.0-2.el7.noarch python-pulp-common-2.20.0-2.el7.noarch python-pulp-deb-common-1.10.0-1.el7.noarch python-pulp-oid_validation-2.20.0-2.el7.noarch python-pulp-repoauth-2.20.0-2.el7.noarch python-pulp-rpm-common-2.20.0-2.el7.noarch

pulp-admin deb repo create --serve-http=true --serve-https=true --publish-default-release=true --relative-url=rasp10/upstream/rasp10_buster --repo-id=upstream_rasp10_buster --feed=http://raspbian.raspberrypi.org/raspbian --releases=buster --components=main,contrib,non-free,rpi,firmware --architectures=armhf

Error received when trying to sync this repository.

Feb 28 05:11:41 snjlv-repo01 pulp: pulp.server.async.tasks:INFO: [dffd14ab] Task failed : [dffd14ab-c5ec-4d10-98cd-318e793da415] Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) Task pulp.server.managers.repo.sync.sync[dffd14ab-c5ec-4d10-98cd-318e793da415] raised unexpected: IOError(2, 'No such file or directory') Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) Traceback (most recent call last): Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) R = retval = fun(*args, **kwargs) Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 529, in call Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) return super(Task, self).call(*args, **kwargs) Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in call Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) return super(PulpTask, self).call(*args, **kwargs) Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in protected_call Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) return self.run(*args, **kwargs) Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 770, in sync Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) sync_report = sync_repo(transfer_repo, conduit, call_config) Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 737, in wrap_f Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) return f(*args, **kwargs) Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/importer.py", line 135, in sync_repo Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) report = self._current_sync.process_lifecycle() Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) super(PluginStep, self).process_lifecycle() Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) step.process() Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 257, in process Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) self._process_block() Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 304, in _process_block Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) self.process_main() Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/sync.py", line 348, in process_main Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) checksums = unit.calculate_deb_checksums(path) Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/db/models.py", line 254, in calculate_deb_checksums Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) with open(input_file_path) as input_file: Feb 28 05:11:45 snjlv-repo01 pulp: celery.app.trace:ERROR: [dffd14ab] (27228-81152) IOError: [Errno 2] No such file or directory: u'/var/cache/pulp/reserved_resource_worker-3@snjlv-repo01.us.cboe.net/dffd14ab-c5ec-4d10-98cd-318e793da415/packages/00/01/librust-gio+v2-40-dev_0.5.1-2_armhf.deb'

This consistently fails against the same package.

My initial findings are that there is nothing wrong with the upstream package. It downloads fine using wget, it is referenced fine from the Packages file in the main component, and all the checksums match. I can see absolutely nothing special about this package.

The next step is to try to reproduce this sync. But to do that I first need to deploy a pulp2 instance with enough disk space to sync all of raspbian.

Possibly more information

After manually placing the librust-gio+v2-40-dev_0.5.1-2_armhf.deb in the expected temporary location in cleanly goes past that file, however it now fails on another, Mar

3 03:55:12 snjlv-repo01 pulp: nectar.downloaders.threaded:INFO: Download succeeded: http://raspbian.raspberrypi.org/raspbian/pool/main/p/python-xstatic-angular-mock/python3-xstatic-angular-mock_1.2.23.1-6_all.deb.

And the same error

Mar 3 03:57:05 snjlv-repo01 pulp: pulp.server.async.tasks:INFO: [3f6853b8] Task failed : [3f6853b8-d6fa-49c6-81be-36dbc07e8d0f] Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) Task pulp.server.managers.repo.sync.sync[3f6853b8-d6fa-49c6-81be-36dbc07e8d0f] raised unexpected: IOError(2, 'No such file or directory') Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) Traceback (most recent call last): Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) R = retval = fun(*args, **kwargs) Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 529, in call Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) return super(Task, self).call(*args, **kwargs) Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in call Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) return super(PulpTask, self).call(*args, **kwargs) Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in protected_call Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) return self.run(*args, **kwargs) Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 770, in sync Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) sync_report = sync_repo(transfer_repo, conduit, call_config) Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 737, in wrap_f Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) return f(*args, **kwargs) Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/importer.py", line 135, in sync_repo Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) report = self._current_sync.process_lifecycle() Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) super(PluginStep, self).process_lifecycle() Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) step.process() Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 257, in process Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) self._process_block() Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 304, in _process_block Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) self.process_main() Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/sync.py", line 348, in process_main Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) checksums = unit.calculate_deb_checksums(path) Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/db/models.py", line 254, in calculate_deb_checksums Mar 3 03:57:08 snjlv-repo01 pulp: celery.app.trace:ERROR: [3f6853b8] (2258-81152) with open(input_file_path) as input_file:

However I continue to see no reference to this file ever appearing to have been downloaded or even attempted.

So I'm not going to be able to invest too much more time into this issue.

This is what I've observed so far.

The temporary workspace grows often to ~15gb with each attempted sync (over 100gb available, just in case).

The sync job can fail between 5 and 50 minutes into the sync (varies).

It isn't always the same package that errors.

I believe we always see the below error just prior to the IOError

ReadTimeoutError, for a slew of packages.

I never see the nectar.downloaders.threaded:INFO message for the missing package (the first you know about it is with an IO error.

A couple of thoughts.

1. I don't believe the RPM based sync requires such a large temporary working area.
2. We appear to have to download the entire distribution, prior to ingesting the individual packages.
3. I've had a look through the code, but my python foo/time limits my ability to assist with this unfortunately.

Doug