Actions
Issue #753
closed
Sync just hangs when running out of diskspace
Status:
CLOSED - WORKSFORME
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:
Description
Version
[root@sparks ~]# rpm -qa pulp-server
pulp-server-2.6.0-0.7.beta.el6.noarch
Description
When there is no space on the disk to complete a sync the sync just hangs.
[root@sparks ~]# pulp-admin rpm repo sync run --repo-id rhel6-ks
Warning: path should have mode 0700 because it may contain sensitive information: /root/.pulp/
+----------------------------------------------------------------------+
Synchronizing Repository [rhel6-ks]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[|]
... completed
Downloading repository content...
[============== ] 28%
RPMs: 1076/3690 items
Delta RPMs: 0/0 items
From the log
Mar 12 11:48:05 sparks pulp: pulp.plugins.conduits.mixins:ERROR: (29235-37056) u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/apr-util-devel-1.3.9-3.el6_0.1.i686.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/ustr-1.0.4-9.1.el6.x86_64.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/taglib-1.6.1-1.1.el6.i686.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/sip-4.9.3-1.el6.x86_64.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/kdebase-workspace-devel-4.3.4-24.el6.i686.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/pixman-devel-0.26.2-5.el6_4.x86_64.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/gvfs-devel-1.4.3-15.el6.i686.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/cyrus-imapd-2.3.16-6.el6_2.5.x86_64.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/PyKDE4-4.3.4-5.el6.x86_64.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/gstreamer-python-0.10.16-1.1.el6.x86_64.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/device-mapper-persistent-data-0.2.8-2.el6.x86_64.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url':
Mar 12 11:48:05 sparks rsyslogd-2177: imuxsock begins to drop messages from pid 29235 due to rate-limiting
Mar 12 11:50:01 sparks auditd[1101]: Audit daemon is low on disk space for logging
Mar 12 11:50:01 sparks auditd[1101]: Audit da
Updated by bmbouter about 9 years ago
- Description updated (diff)
I put pre tags in so its blockquoted
Updated by dkliban@redhat.com about 9 years ago
- Severity set to Medium
- Triaged changed from No to Yes
Updated by mhrivnak over 7 years ago
- Status changed from NEW to CLOSED - WORKSFORME
I tried this just now with a yum repo with pulp master (future 2.11). I created a very small filesystem, mounted it to /var/cache/pulp/, and made sure the apache user had permission to write to it. I then restarted all pulp services and tried to sync a pulp 2.9 repo.
I saw this in the terminal, which happened quickly:
$ pulp-admin rpm repo sync run --repo-id=pulp29
+----------------------------------------------------------------------+
Synchronizing Repository [pulp29]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[|]
... completed
Downloading repository content...
[==================================================] 100%
RPMs: 59/59 items
Delta RPMs: 0/0 items
Task Failed
Importer indicated a failed response
and lots of this is the log:
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) [Errno 28] No space left on device: u'/var/cache/pulp/reserved_resource_worker-0@dev/6b6a4755-6196-4eb1-acf5-c235a81af8e5/tmpKIc5KW/python-pymongo-debuginfo-3.2-1.el7.x86_64.rpm'
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) Traceback (most recent call last):
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 229, in _fetch
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) file_handle = request.initialize_file_handle()
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) File "/usr/lib/python2.7/site-packages/nectar/request.py", line 46, in initialize_file_handle
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) self._file_handle = open(self.destination, 'wb') # cache the handle
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) IOError: [Errno 28] No space left on device: u'/var/cache/pulp/reserved_resource_worker-0@dev/6b6a4755-6196-4eb1-acf5-c235a81af8e5/tmpKIc5KW/python-pymongo-debuginfo-3.2-1.el7.x86_64.rpm'
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) [Errno 28] No space left on device: u'/var/cache/pulp/reserved_resource_worker-0@dev/6b6a4755-6196-4eb1-acf5-c235a81af8e5/tmpKIc5KW/python-pymongo-gridfs-3.2-1.el7.x86_64.rpm'
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) Traceback (most recent call last):
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 229, in _fetch
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) file_handle = request.initialize_file_handle()
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) File "/usr/lib/python2.7/site-packages/nectar/request.py", line 46, in initialize_file_handle
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) self._file_handle = open(self.destination, 'wb') # cache the handle
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) IOError: [Errno 28] No space left on device: u'/var/cache/pulp/reserved_resource_worker-0@dev/6b6a4755-6196-4eb1-acf5-c235a81af8e5/tmpKIc5KW/python-pymongo-gridfs-3.2-1.el7.x86_64.rpm'
Aug 09 13:16:32 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) [Errno 28] No space left on device: u'/var/cache/pulp/reserved_resource_worker-0@dev/6b6a4755-6196-4eb1-acf5-c235a81af8e5/tmpKIc5KW/python-pulp-oid_validation-2.9.1-1.el7.noarch.rpm'
Aug 09 13:16:32 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) Traceback (most recent call last):
Aug 09 13:16:32 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 229, in _fetch
Aug 09 13:16:32 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) file_handle = request.initialize_file_handle()
Aug 09 13:16:32 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) File "/usr/lib/python2.7/site-packages/nectar/request.py", line 46, in initialize_file_handle
Aug 09 13:16:32 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) self._file_handle = open(self.destination, 'wb') # cache the handle
Aug 09 13:16:32 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) IOError: [Errno 28] No space left on device: u'/var/cache/pulp/reserved_resource_worker-0@dev/6b6a4755-6196-4eb1-acf5-c235a81af8e5/tmpKIc5KW/python-pulp-oid_validation-2.9.1-1.el7.noarch.rpm'
Aug 09 13:16:32 dev pulp[19105]: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: The content container reported: {'downloads': {'___/primary/___': {'total_failed': 57, 'total_succeeded': 2}}, 'total_sources': 0} for base URL: https://repos.fedorapeople.org/repos/pulp/pulp/stable/2.9/7/x86_64/
Aug 09 13:16:32 dev pulp[19105]: pulp_rpm.plugins.importers.yum.sync:ERROR: (19105-86400) 'error_code'
Aug 09 13:16:32 dev pulp[19105]: pulp_rpm.plugins.importers.yum.sync:ERROR: (19105-86400) Traceback (most recent call last):
Aug 09 13:16:32 dev pulp[19105]: pulp_rpm.plugins.importers.yum.sync:ERROR: (19105-86400) File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/sync.py", line 224, in run
Aug 09 13:16:32 dev pulp[19105]: pulp_rpm.plugins.importers.yum.sync:ERROR: (19105-86400) self.update_content(metadata_files, url)
Aug 09 13:16:32 dev pulp[19105]: pulp_rpm.plugins.importers.yum.sync:ERROR: (19105-86400) File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/sync.py", line 493, in update_content
Aug 09 13:16:32 dev pulp[19105]: pulp_rpm.plugins.importers.yum.sync:ERROR: (19105-86400) if error[constants.ERROR_CODE] == constants.ERROR_SIGNATURE_VERIFICATION:
Aug 09 13:16:32 dev pulp[19105]: pulp_rpm.plugins.importers.yum.sync:ERROR: (19105-86400) KeyError: 'error_code'
Aug 09 13:16:32 dev pulp[19105]: pulp.server.async.tasks:INFO: Task failed : [6b6a4755-6196-4eb1-acf5-c235a81af8e5]
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400) Task pulp.server.managers.repo.sync.sync[6b6a4755-6196-4eb1-acf5-c235a81af8e5] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400) Traceback (most recent call last):
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400) R = retval = fun(*args, **kwargs)
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400) File "/home/vagrant/devel/pulp/server/pulp/server/async/tasks.py", line 488, in __call__
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400) return super(Task, self).__call__(*args, **kwargs)
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400) File "/home/vagrant/devel/pulp/server/pulp/server/async/tasks.py", line 103, in __call__
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400) return super(PulpTask, self).__call__(*args, **kwargs)
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400) return self.run(*args, **kwargs)
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400) File "/home/vagrant/devel/pulp/server/pulp/server/controllers/repository.py", line 810, in sync
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400) raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400) PulpExecutionException: Importer indicated a failed response
Actions
.