Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #1164

closed

pulp sync fails randomly

Added by cristi.falcas@gmail.com over 8 years ago. Updated about 5 years ago.

Status:
CLOSED - WORKSFORME
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.6.2 Beta
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Bellow you can see the same command executed a few times, where the first runs it throws an error. Nothing was changed between runs.

Only one of the pulp servers has this problem.

Packages installed:
[root@v-so-repo-03 sdt_user]# rpm -qa | grep pulp | sort
pulp-admin-client-2.6.2-1.el7.noarch
pulp-agent-2.6.2-1.el7.noarch
pulp-consumer-client-2.6.2-1.el7.noarch
pulp-nodes-admin-extensions-2.6.2-1.el7.noarch
pulp-nodes-common-2.6.2-1.el7.noarch
pulp-nodes-parent-2.6.2-1.el7.noarch
pulp-puppet-admin-extensions-2.6.2-1.el7.noarch
pulp-puppet-plugins-2.6.2-1.el7.noarch
pulp-puppet-tools-2.6.2-1.el7.noarch
pulp-rpm-admin-extensions-2.6.2-1.el7.noarch
pulp-rpm-consumer-extensions-2.6.2-1.el7.noarch
pulp-rpm-handlers-2.6.2-1.el7.noarch
pulp-rpm-plugins-2.6.2-1.el7.noarch
pulp-rpm-yumplugins-2.6.2-1.el7.noarch
pulp-selinux-2.6.2-1.el7.noarch
pulp-server-2.6.2-1.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.24-7.pulp.el7.noarch
python-pulp-agent-lib-2.6.2-1.el7.noarch
python-pulp-bindings-2.6.2-1.el7.noarch
python-pulp-client-lib-2.6.2-1.el7.noarch
python-pulp-common-2.6.2-1.el7.noarch
python-pulp-puppet-common-2.6.2-1.el7.noarch
python-pulp-rpm-common-2.6.2-1.el7.noarch
python-rhsm-1.8.0-2.pulp.el7.x86_64

Output from console:

[root@v-so-repo-03 sdt_user]# pulp-admin rpm repo sync run --repo-id optymyze_el7
+----------------------------------------------------------------------+
                Synchronizing Repository [optymyze_el7]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... failed

[Errno 2] No such file or directory:
u'/var/lib/pulp/working/repos/optymyze_el7/importers/yum_importer/tmpvu0Imj/0b76
f03e2af4156f427ca6e095344c4ab674b68731d9f6470b376a052c01897f-filelists.xml.gz'

Task Failed

Importer indicated a failed response

[root@v-so-repo-03 sdt_user]# pulp-admin rpm repo sync run --repo-id optymyze_el7
+----------------------------------------------------------------------+
                Synchronizing Repository [optymyze_el7]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... failed

[Errno 2] No such file or directory:
u'/var/lib/pulp/working/repos/optymyze_el7/importers/yum_importer/tmp7w7XZL/0b76
f03e2af4156f427ca6e095344c4ab674b68731d9f6470b376a052c01897f-filelists.xml.gz'

Task Failed

Importer indicated a failed response

[root@v-so-repo-03 sdt_user]# pulp-admin rpm repo sync run --repo-id optymyze_el7
+----------------------------------------------------------------------+
                Synchronizing Repository [optymyze_el7]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[/]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       1/1 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Task Succeeded

Copying files
[\]
... completed

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[=                                                 ] 3%
1 of 26 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[-]
... completed

Publishing Comps file
[-]
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

Logs from journald:

Aug 03 17:41:02 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.purge:INFO: Removing old units.
Aug 03 17:41:02 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading additional units.
Aug 03 17:41:02 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:03 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:04 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:INFO: Sync complete.
Aug 03 17:41:04 v-so-repo-03.synygy.net pulp[888]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[f290e4e2-8790-4b61
-a189-86a1216090f1]
Aug 03 17:41:04 v-so-repo-03.synygy.net pulp[1203]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[56b3e9e5-cf91-4f74-a
6cc-24474f78507c]
Aug 03 17:41:04 v-so-repo-03.synygy.net pulp[1203]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[eddbe23e-f243-4fb4-9
861-9d698b19f930]
Aug 03 17:41:04 v-so-repo-03.synygy.net pulp[888]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[f290e4e2-8790-4b61-a189-86a121609
0f1] succeeded in 0.0606332900352s: None
Aug 03 17:41:04 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[e94e0281-43d1-4257-ad46-39f3e0304041] suc
ceeded in 23.458381277s: <pulp.server.async.tasks.TaskResult object at 0x2571a10>
Aug 03 17:41:04 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[b215f20a-b5c5-4f58-81a7-36e4bd7fc5c
5] succeeded in 0.0907369150082s: None
Aug 03 17:41:05 v-so-repo-03.synygy.net hekad[15940]: 2015/08/03 17:41:05 Plugin 'tcpoutput_SendToServer' error: dial tcp 10.220.10.105:5565: connection refus
ed
Aug 03 17:41:06 v-so-repo-03.synygy.net pulp[8277]: pulp.plugins.pulp_rpm.plugins.distributors.yum.metadata.metadata:WARNING: Overwriting existing metadata fi
le [/var/lib/pulp/working/repos/optymyze_el6/distributors/yum_distributor/repodata/repomd.xml]
Aug 03 17:41:06 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:INFO: Task pulp.server.managers.repo.publish.publish[56b3e9e5-cf91-4f74-a6cc-24474f78507
c] succeeded in 2.15032853396s: {'exception': None, 'repo_id': u'optymyze_el6', 'traceback': None, 'started': '2015-08-03T12:11:04Z', '_ns':...
Aug 03 17:41:06 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[eddbe23e-f243-4fb4-9861-9d698b19f93
0] succeeded in 0.0486922189593s: None
Aug 03 17:41:07 v-so-repo-03.synygy.net collectd[20838]: exec plugin: Failed to execute ``/opt/optymyze/collectd/scripts/bash/zk_avail.sh'': No such file or d
irectory
Aug 03 17:41:07 v-so-repo-03.synygy.net collectd[20839]: exec plugin: Failed to execute ``/opt/optymyze/collectd/scripts/bash/indomain.sh'': No such file or d
irectory
Aug 03 17:41:09 v-so-repo-03.synygy.net pulp[888]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[84d59656-d48d-4dca
-948b-edc10fa95d97]
Aug 03 17:41:09 v-so-repo-03.synygy.net pulp[1203]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[cb8ab5f1-37c9-4923-a5b5-6f
8ac9ee0452]
Aug 03 17:41:09 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from http://v-so-repo-01.synygy.net/repos/o
ptymyze/7Server/.
Aug 03 17:41:09 v-so-repo-03.synygy.net pulp[1203]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[4b7233af-d31d-43fd-9
262-54994392349b]
Aug 03 17:41:09 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:09 v-so-repo-03.synygy.net pulp[888]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[84d59656-d48d-4dca-948b-edc10fa95
d97] succeeded in 0.0628154140431s: None
Aug 03 17:41:10 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
Aug 03 17:41:10 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata files.
Aug 03 17:41:10 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:10 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:10 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:10 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:10 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:11 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:INFO: Generating metadata databases.
Aug 03 17:41:11 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312) sync failed
Aug 03 17:41:11 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312) Traceback (most recent call last):
Aug 03 17:41:11 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)   File "/usr/lib/python2.7/site-packages/pulp_rpm/
plugins/importers/yum/sync.py", line 142, in run
Aug 03 17:41:11 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)     metadata_files = self.get_metadata()
Aug 03 17:41:11 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)   File "/usr/lib/python2.7/site-packages/pulp_rpm/
plugins/importers/yum/sync.py", line 247, in get_metadata
Aug 03 17:41:11 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)     metadata_files.generate_dbs()
Aug 03 17:41:11 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)   File "/usr/lib/python2.7/site-packages/pulp_rpm/
plugins/importers/yum/repomd/metadata.py", line 278, in generate_dbs
Aug 03 17:41:11 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)     with contextlib.closing(self.get_metadata_file
_handle(filename)) as xml_file_handle:
Aug 03 17:41:11 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)   File "/usr/lib/python2.7/site-packages/pulp_rpm/
plugins/importers/yum/repomd/metadata.py", line 247, in get_metadata_file_handle
Aug 03 17:41:11 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)     file_handle = gzip.open(file_path, 'r')
Aug 03 17:41:11 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)   File "/usr/lib64/python2.7/gzip.py", line 34, in
 open
Aug 03 17:41:11 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)     return GzipFile(filename, mode, compresslevel)
Aug 03 17:41:11 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)   File "/usr/lib64/python2.7/gzip.py", line 94, in
 __init__
Aug 03 17:41:11 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)     fileobj = self.myfileobj = __builtin__.open(fi
lename, mode or 'rb')
Aug 03 17:41:11 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312) IOError: [Errno 2] No such file or directory: u'/v
ar/lib/pulp/working/repos/optymyze_el7/importers/yum_importer/tmpvu0Imj/0b76f03e2af4156f427ca6e095344c4ab674b68731d9f6470b376a052c01897f-filelists.xml.gz'
Aug 03 17:41:12 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312) Task pulp.server.managers.repo.sync.sync[cb8ab5f1-37c9-4923-a5b5-6f8
ac9ee0452] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Aug 03 17:41:12 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312) Traceback (most recent call last):
Aug 03 17:41:12 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line
240, in trace_task
Aug 03 17:41:12 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312)     R = retval = fun(*args, **kwargs)
Aug 03 17:41:12 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py"
, line 328, in __call__
Aug 03 17:41:12 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312)     return super(Task, self).__call__(*args, **kwargs)
Aug 03 17:41:12 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line
437, in __protected_call__
Aug 03 17:41:12 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312)     return self.run(*args, **kwargs)
Aug 03 17:41:12 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/s
ync.py", line 114, in sync
Aug 03 17:41:12 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312)     raise PulpExecutionException(_('Importer indicated a failed resp
onse'))
Aug 03 17:41:12 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312) PulpExecutionException: Importer indicated a failed response
Aug 03 17:41:12 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[4b7233af-d31d-43fd-9262-54994392349
b] succeeded in 0.0787709239521s: None
Aug 03 17:41:13 v-so-repo-03.synygy.net pulp[888]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[4dc5df91-db79-49e0
-bc8f-ff67620a19f7]
Aug 03 17:41:13 v-so-repo-03.synygy.net pulp[1203]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[9b001ed4-aa45-4b3d-ad94-fb
4aca26fb96]
Aug 03 17:41:13 v-so-repo-03.synygy.net pulp[1203]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[1e7de200-3387-4a40-8
0d4-5dd40018814d]
Aug 03 17:41:13 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from http://v-so-repo-01.synygy.net/repos/o
ptymyze/7Server/.
Aug 03 17:41:13 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:13 v-so-repo-03.synygy.net pulp[888]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[4dc5df91-db79-49e0-bc8f-ff67620a1
9f7] succeeded in 0.0711295710062s: None
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata files.
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81312) ('Connection aborted.', error(104, 'Connection reset by pe
er'))
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81312) Traceback (most recent call last):
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81312)   File "/usr/lib/python2.7/site-packages/nectar/downloader
s/threaded.py", line 190, in _fetch
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81312)     response = session.get(request.url, headers=headers)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81312)   File "/usr/lib/python2.7/site-packages/requests/sessions
.py", line 476, in get
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81312)     return self.request('GET', url, **kwargs)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81312)   File "/usr/lib/python2.7/site-packages/requests/sessions
.py", line 464, in request
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81312)     resp = self.send(prep, **send_kwargs)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81312)   File "/usr/lib/python2.7/site-packages/requests/sessions
.py", line 576, in send
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81312)     r = adapter.send(request, **kwargs)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81312)   File "/usr/lib/python2.7/site-packages/requests/adapters
.py", line 415, in send
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81312)     raise ConnectionError(err, request=request)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81312) ConnectionError: ('Connection aborted.', error(104, 'Conne
ction reset by peer'))
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-74016) ('Connection aborted.', error(104, 'Connection reset by pe
er'))
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-74016) Traceback (most recent call last):
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-74016)   File "/usr/lib/python2.7/site-packages/nectar/downloader
s/threaded.py", line 190, in _fetch
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-74016)     response = session.get(request.url, headers=headers)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-74016)   File "/usr/lib/python2.7/site-packages/requests/sessions
.py", line 476, in get
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-74016)     return self.request('GET', url, **kwargs)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-74016)   File "/usr/lib/python2.7/site-packages/requests/sessions
.py", line 464, in request
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-74016)     resp = self.send(prep, **send_kwargs)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-74016)   File "/usr/lib/python2.7/site-packages/requests/sessions
.py", line 576, in send
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-74016)     r = adapter.send(request, **kwargs)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-74016)   File "/usr/lib/python2.7/site-packages/requests/adapters
.py", line 415, in send
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-74016)     raise ConnectionError(err, request=request)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-74016) ConnectionError: ('Connection aborted.', error(104, 'Conne
ction reset by peer'))
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81664) ('Connection aborted.', error(104, 'Connection reset by pe
er'))
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81664) Traceback (most recent call last):
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81664)   File "/usr/lib/python2.7/site-packages/nectar/downloader
s/threaded.py", line 190, in _fetch
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81664)     response = session.get(request.url, headers=headers)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81664)   File "/usr/lib/python2.7/site-packages/requests/sessions
.py", line 476, in get
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81664)     return self.request('GET', url, **kwargs)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81664)   File "/usr/lib/python2.7/site-packages/requests/sessions
.py", line 464, in request
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81664)     resp = self.send(prep, **send_kwargs)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81664)   File "/usr/lib/python2.7/site-packages/requests/sessions
.py", line 576, in send
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81664)     r = adapter.send(request, **kwargs)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81664)   File "/usr/lib/python2.7/site-packages/requests/adapters
.py", line 415, in send
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81664)     raise ConnectionError(err, request=request)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-81664) ConnectionError: ('Connection aborted.', error(104, 'Conne
ction reset by peer'))
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-88960) ('Connection aborted.', error(104, 'Connection reset by pe
er'))
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-88960) Traceback (most recent call last):
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-88960)   File "/usr/lib/python2.7/site-packages/nectar/downloader
s/threaded.py", line 190, in _fetch
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-88960)     response = session.get(request.url, headers=headers)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-88960)   File "/usr/lib/python2.7/site-packages/requests/sessions
.py", line 476, in get
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-88960)     return self.request('GET', url, **kwargs)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-88960)   File "/usr/lib/python2.7/site-packages/requests/sessions
.py", line 464, in request
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-88960)     resp = self.send(prep, **send_kwargs)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-88960)   File "/usr/lib/python2.7/site-packages/requests/sessions
.py", line 576, in send
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-88960)     r = adapter.send(request, **kwargs)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-88960)   File "/usr/lib/python2.7/site-packages/requests/adapters
.py", line 415, in send
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-88960)     raise ConnectionError(err, request=request)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-88960) ConnectionError: ('Connection aborted.', error(104, 'Conne
ction reset by peer'))
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-03552) ('Connection aborted.', error(104, 'Connection reset by pe
er'))
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-03552) Traceback (most recent call last):
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-03552)   File "/usr/lib/python2.7/site-packages/nectar/downloader
s/threaded.py", line 190, in _fetch
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-03552)     response = session.get(request.url, headers=headers)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-03552)   File "/usr/lib/python2.7/site-packages/requests/sessions
.py", line 476, in get
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-03552)     return self.request('GET', url, **kwargs)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-03552)   File "/usr/lib/python2.7/site-packages/requests/sessions
.py", line 464, in request
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-03552)     resp = self.send(prep, **send_kwargs)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-03552)   File "/usr/lib/python2.7/site-packages/requests/sessions
.py", line 576, in send
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-03552)     r = adapter.send(request, **kwargs)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-03552)   File "/usr/lib/python2.7/site-packages/requests/adapters
.py", line 415, in send
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-03552)     raise ConnectionError(err, request=request)
Aug 03 17:41:14 v-so-repo-03.synygy.net pulp[8277]: nectar.downloaders.threaded:ERROR: (8277-03552) ConnectionError: ('Connection aborted.', error(104, 'Conne
ction reset by peer'))
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:INFO: Generating metadata databases.
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312) sync failed
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312) Traceback (most recent call last):
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)   File "/usr/lib/python2.7/site-packages/pulp_rpm/
plugins/importers/yum/sync.py", line 142, in run
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)     metadata_files = self.get_metadata()
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)   File "/usr/lib/python2.7/site-packages/pulp_rpm/
plugins/importers/yum/sync.py", line 247, in get_metadata
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)     metadata_files.generate_dbs()
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)   File "/usr/lib/python2.7/site-packages/pulp_rpm/
plugins/importers/yum/repomd/metadata.py", line 278, in generate_dbs
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)     with contextlib.closing(self.get_metadata_file
_handle(filename)) as xml_file_handle:
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)   File "/usr/lib/python2.7/site-packages/pulp_rpm/
plugins/importers/yum/repomd/metadata.py", line 247, in get_metadata_file_handle
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)     file_handle = gzip.open(file_path, 'r')
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)   File "/usr/lib64/python2.7/gzip.py", line 34, in
 open
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)     return GzipFile(filename, mode, compresslevel)
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)   File "/usr/lib64/python2.7/gzip.py", line 94, in
 __init__
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312)     fileobj = self.myfileobj = __builtin__.open(fi
lename, mode or 'rb')
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:ERROR: (8277-33312) IOError: [Errno 2] No such file or directory: u'/v
ar/lib/pulp/working/repos/optymyze_el7/importers/yum_importer/tmp7w7XZL/0b76f03e2af4156f427ca6e095344c4ab674b68731d9f6470b376a052c01897f-filelists.xml.gz'
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312) Task pulp.server.managers.repo.sync.sync[9b001ed4-aa45-4b3d-ad94-fb4
aca26fb96] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312) Traceback (most recent call last):
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line
240, in trace_task
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312)     R = retval = fun(*args, **kwargs)
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py"
, line 328, in __call__
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312)     return super(Task, self).__call__(*args, **kwargs)
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line
437, in __protected_call__
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312)     return self.run(*args, **kwargs)
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/s
ync.py", line 114, in sync
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312)     raise PulpExecutionException(_('Importer indicated a failed resp
onse'))
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:ERROR: (1203-33312) PulpExecutionException: Importer indicated a failed response
Aug 03 17:41:15 v-so-repo-03.synygy.net pulp[1203]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[1e7de200-3387-4a40-80d4-5dd40018814
d] succeeded in 0.0543927910039s: None
Aug 03 17:41:16 v-so-repo-03.synygy.net pulp[888]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[2f6aa7c8-1752-4c45
-8070-a8ae5f278150]
Aug 03 17:41:16 v-so-repo-03.synygy.net pulp[1203]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[106a3eb6-36a6-47f4-aa7e-27
476676c163]
Aug 03 17:41:16 v-so-repo-03.synygy.net pulp[1203]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[42293741-2452-4b1e-9
ef6-e06375f3c63b]
Aug 03 17:41:16 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from http://v-so-repo-01.synygy.net/repos/o
ptymyze/7Server/.
Aug 03 17:41:16 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:16 v-so-repo-03.synygy.net pulp[888]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[2f6aa7c8-1752-4c45-8070-a8ae5f278
150] succeeded in 0.0619295460056s: None
Aug 03 17:41:17 v-so-repo-03.synygy.net hekad[15940]: 2015/08/03 17:41:17 Plugin 'tcpoutput_SendToServer' error: dial tcp 10.220.10.105:5565: connection refus
ed
Aug 03 17:41:17 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
Aug 03 17:41:17 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata files.
Aug 03 17:41:17 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:17 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:17 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:17 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:17 v-so-repo-03.synygy.net pulp[8277]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): v-so-repo-01.synygy.net
Aug 03 17:41:21 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:INFO: Generating metadata databases.
Aug 03 17:41:24 v-so-repo-03.synygy.net pulp[8277]: pulp_rpm.plugins.importers.yum.sync:INFO: Determining which units need to be downloaded.
Actions
Copy link
 #1

Updated by mhrivnak over 8 years ago

When you say only one of the pulp servers has this problem, can you elaborate? Do you have multiple servers in a cluster? If so, verify that the filesystem is shared correctly for /var/lib/pulp/

Also, do you have any additional information about the connection errors? Does this failure only occur when there is also a connection error logged?

Actions
Copy link
 #2

Updated by bmbouter over 8 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by cristi.falcas@gmail.com over 8 years ago

We have multiple pulp servers completely independent.

I don't know why the connection errors appear. I didn't notice them until now.

Also, I can't reproduce the error anymore. Probably because I deleted the repo, removed the orphans and recreated the repo again.

I will pay more attention to logs in case it happens again.

Actions
Copy link
 #4

Updated by bcourt over 8 years ago

  • Status changed from NEW to CLOSED - WORKSFORME
  • Triaged changed from No to Yes

Please reopen if this problem re-occurs. Thanks

Actions
Copy link
 #5

Updated by mhrivnak over 8 years ago

  • Triaged changed from Yes to No
Actions
Copy link
 #6

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF