Project

Profile

Help

Issue #753

Sync just hangs when running out of diskspace

Added by pthomas@redhat.com over 6 years ago. Updated over 2 years ago.

Status:
CLOSED - WORKSFORME
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Version
[root@sparks ~]# rpm -qa pulp-server
pulp-server-2.6.0-0.7.beta.el6.noarch

Description
When there is no space on the disk to complete a sync the sync just hangs.

[root@sparks ~]# pulp-admin rpm repo sync run --repo-id rhel6-ks
Warning: path should have mode 0700 because it may contain sensitive information: /root/.pulp/

+----------------------------------------------------------------------+
                  Synchronizing Repository [rhel6-ks]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[|]
... completed

Downloading repository content...
[==============                                    ] 28%
RPMs:       1076/3690 items
Delta RPMs: 0/0 items

From the log

Mar 12 11:48:05 sparks pulp: pulp.plugins.conduits.mixins:ERROR: (29235-37056)  u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/apr-util-devel-1.3.9-3.el6_0.1.i686.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/ustr-1.0.4-9.1.el6.x86_64.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/taglib-1.6.1-1.1.el6.i686.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/sip-4.9.3-1.el6.x86_64.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/kdebase-workspace-devel-4.3.4-24.el6.i686.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/pixman-devel-0.26.2-5.el6_4.x86_64.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/gvfs-devel-1.4.3-15.el6.i686.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/cyrus-imapd-2.3.16-6.el6_2.5.x86_64.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/PyKDE4-4.3.4-5.el6.x86_64.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/gstreamer-python-0.10.16-1.1.el6.x86_64.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url': u'https://cdn.redhat.com/content/dist/rhel/server/6/6.5/x86_64/kickstart/Packages/device-mapper-persistent-data-0.2.8-2.el6.x86_64.rpm', 'errors': ['[Errno 28] No space left on device']}, {'url':
Mar 12 11:48:05 sparks rsyslogd-2177: imuxsock begins to drop messages from pid 29235 due to rate-limiting
Mar 12 11:50:01 sparks auditd[1101]: Audit daemon is low on disk space for logging
Mar 12 11:50:01 sparks auditd[1101]: Audit da

History

#1 Updated by bmbouter over 6 years ago

  • Description updated (diff)

I put pre tags in so its blockquoted

#2 Updated by dkliban@redhat.com over 6 years ago

  • Severity set to Medium
  • Triaged changed from No to Yes

#3 Updated by bmbouter over 6 years ago

  • Severity changed from Medium to 2. Medium

#4 Updated by mhrivnak about 5 years ago

  • Status changed from NEW to CLOSED - WORKSFORME

I tried this just now with a yum repo with pulp master (future 2.11). I created a very small filesystem, mounted it to /var/cache/pulp/, and made sure the apache user had permission to write to it. I then restarted all pulp services and tried to sync a pulp 2.9 repo.

I saw this in the terminal, which happened quickly:

$ pulp-admin rpm repo sync run --repo-id=pulp29
+----------------------------------------------------------------------+
                   Synchronizing Repository [pulp29]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[|]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       59/59 items
Delta RPMs: 0/0 items

Task Failed

Importer indicated a failed response

and lots of this is the log:

Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) [Errno 28] No space left on device: u'/var/cache/pulp/reserved_resource_worker-0@dev/6b6a4755-6196-4eb1-acf5-c235a81af8e5/tmpKIc5KW/python-pymongo-debuginfo-3.2-1.el7.x86_64.rpm'
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) Traceback (most recent call last):
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 229, in _fetch
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656)     file_handle = request.initialize_file_handle()
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656)   File "/usr/lib/python2.7/site-packages/nectar/request.py", line 46, in initialize_file_handle
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656)     self._file_handle = open(self.destination, 'wb')  # cache the handle
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) IOError: [Errno 28] No space left on device: u'/var/cache/pulp/reserved_resource_worker-0@dev/6b6a4755-6196-4eb1-acf5-c235a81af8e5/tmpKIc5KW/python-pymongo-debuginfo-3.2-1.el7.x86_64.rpm'
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) [Errno 28] No space left on device: u'/var/cache/pulp/reserved_resource_worker-0@dev/6b6a4755-6196-4eb1-acf5-c235a81af8e5/tmpKIc5KW/python-pymongo-gridfs-3.2-1.el7.x86_64.rpm'
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) Traceback (most recent call last):
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 229, in _fetch
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224)     file_handle = request.initialize_file_handle()
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224)   File "/usr/lib/python2.7/site-packages/nectar/request.py", line 46, in initialize_file_handle
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224)     self._file_handle = open(self.destination, 'wb')  # cache the handle
Aug 09 13:16:31 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) IOError: [Errno 28] No space left on device: u'/var/cache/pulp/reserved_resource_worker-0@dev/6b6a4755-6196-4eb1-acf5-c235a81af8e5/tmpKIc5KW/python-pymongo-gridfs-3.2-1.el7.x86_64.rpm'
Aug 09 13:16:32 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) [Errno 28] No space left on device: u'/var/cache/pulp/reserved_resource_worker-0@dev/6b6a4755-6196-4eb1-acf5-c235a81af8e5/tmpKIc5KW/python-pulp-oid_validation-2.9.1-1.el7.noarch.rpm'
Aug 09 13:16:32 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) Traceback (most recent call last):
Aug 09 13:16:32 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 229, in _fetch
Aug 09 13:16:32 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)     file_handle = request.initialize_file_handle()
Aug 09 13:16:32 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)   File "/usr/lib/python2.7/site-packages/nectar/request.py", line 46, in initialize_file_handle
Aug 09 13:16:32 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)     self._file_handle = open(self.destination, 'wb')  # cache the handle
Aug 09 13:16:32 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) IOError: [Errno 28] No space left on device: u'/var/cache/pulp/reserved_resource_worker-0@dev/6b6a4755-6196-4eb1-acf5-c235a81af8e5/tmpKIc5KW/python-pulp-oid_validation-2.9.1-1.el7.noarch.rpm'
Aug 09 13:16:32 dev pulp[19105]: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: The content container reported: {'downloads': {'___/primary/___': {'total_failed': 57, 'total_succeeded': 2}}, 'total_sources': 0} for base URL: https://repos.fedorapeople.org/repos/pulp/pulp/stable/2.9/7/x86_64/
Aug 09 13:16:32 dev pulp[19105]: pulp_rpm.plugins.importers.yum.sync:ERROR: (19105-86400) 'error_code'
Aug 09 13:16:32 dev pulp[19105]: pulp_rpm.plugins.importers.yum.sync:ERROR: (19105-86400) Traceback (most recent call last):
Aug 09 13:16:32 dev pulp[19105]: pulp_rpm.plugins.importers.yum.sync:ERROR: (19105-86400)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/sync.py", line 224, in run
Aug 09 13:16:32 dev pulp[19105]: pulp_rpm.plugins.importers.yum.sync:ERROR: (19105-86400)     self.update_content(metadata_files, url)
Aug 09 13:16:32 dev pulp[19105]: pulp_rpm.plugins.importers.yum.sync:ERROR: (19105-86400)   File "/home/vagrant/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/sync.py", line 493, in update_content
Aug 09 13:16:32 dev pulp[19105]: pulp_rpm.plugins.importers.yum.sync:ERROR: (19105-86400)     if error[constants.ERROR_CODE] == constants.ERROR_SIGNATURE_VERIFICATION:
Aug 09 13:16:32 dev pulp[19105]: pulp_rpm.plugins.importers.yum.sync:ERROR: (19105-86400) KeyError: 'error_code'
Aug 09 13:16:32 dev pulp[19105]: pulp.server.async.tasks:INFO: Task failed : [6b6a4755-6196-4eb1-acf5-c235a81af8e5]
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400) Task pulp.server.managers.repo.sync.sync[6b6a4755-6196-4eb1-acf5-c235a81af8e5] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400) Traceback (most recent call last):
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400)     R = retval = fun(*args, **kwargs)
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400)   File "/home/vagrant/devel/pulp/server/pulp/server/async/tasks.py", line 488, in __call__
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400)     return super(Task, self).__call__(*args, **kwargs)
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400)   File "/home/vagrant/devel/pulp/server/pulp/server/async/tasks.py", line 103, in __call__
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400)     return super(PulpTask, self).__call__(*args, **kwargs)
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400)     return self.run(*args, **kwargs)
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400)   File "/home/vagrant/devel/pulp/server/pulp/server/controllers/repository.py", line 810, in sync
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400)     raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
Aug 09 13:16:32 dev pulp[19015]: celery.worker.job:ERROR: (19015-86400) PulpExecutionException: Importer indicated a failed response

#5 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF