Project

Profile

Help

Issue #983

closed

pulp iso sync failure

Added by Ben.Stanley almost 9 years ago. Updated almost 5 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.6.1 Beta
Platform Release:
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Documentation, Pulp 2
Sprint:
Quarter:

Description

I have observed a repeatable pulp sync failure of an ISO repo, where
curl managed to fetch the PULP_MANIFEST successfully.

The pulp sync failure reason appears to be the following log message:

May 13 17:38:11 bumblebee pulp: pulp_rpm.plugins.importers.iso.sync:ERROR:
Failed to download
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/iso/PULP_MANIFEST:
('Connection aborted.', error(110, 'Connection timed out')).

The timeout occurs 2 min after the previous log message.

Under the same situation, curl fetches the same URL successfully every time.
On the other hand, pulp fails every time.
This result is observed with interleaved tests.

The following list summarises all the details that can be found below.

  • pulp-admin iso repo create
  • pulp-admin iso repo sync
  • /var/log/pulp.log containing failure
  • successful curl command
  • pulp rpm versions

I am situated behind a proxy server, which pulp can use successfully on other repos. The proxy server is configured in /etc/pulp/server/plugins.conf.d/yum_importer.json .

=========== pulp-admin iso repo create ===================================
[root bumblebee conf.d]# pulp-admin iso repo create \
--repo-id=rhel-7-server-x86_64-iso \
--feed=https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/iso
\
--feed-ca-cert=/etc/rhsm/ca/redhat-uep.pem \
--feed-cert=/etc/pki/entitlement/$keyid.pem \
--feed-key=/etc/pki/entitlement/$keyid-key.pem \
--validate true \
--serve-http true
Successfully created repository [rhel-7-server-x86_64-iso]
==========================================================================

=========== pulp-admin iso repo sync run =================================
[root bumblebee conf.d]# date
Wed May 13 17:35:56 AEST 2015
[root bumblebee conf.d]# pulp-admin iso repo sync run
--repo-id=rhel-7-server-x86_64-iso
--------------------------------------------------------------------
Synchronizing Repository [rhel-7-server-x86_64-iso]
--------------------------------------------------------------------

This command may be exited via ctrl+c without affecting the request.

Downloading the Pulp Manifest...
Downloading the Pulp Manifest failed:

{}

Task Failed

Importer indicated a failed response

[root bumblebee conf.d]# date
Wed May 13 17:38:23 AEST 2015
[root bumblebee conf.d]#
==========================================================================

=========== /var/log/pulp.log ============================================
May 13 17:36:00 bumblebee pulp: celery.worker.strategy:INFO: Received
task:
pulp.server.async.tasks._queue_reserved_task[0e90b376-d589-4261-bf8f-5afcb8514c91]
May 13 17:36:00 bumblebee pulp: celery.worker.strategy:INFO: Received
task:
pulp.server.managers.repo.sync.sync[0e0bbc76-43de-4dd4-b45c-edff18a59644]
May 13 17:36:00 bumblebee pulp: celery.worker.strategy:INFO: Received
task:
pulp.server.async.tasks._release_resource[09882136-6b6b-422d-bd55-3d312943cad3]
May 13 17:36:00 bumblebee pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
connection (1): cdn.redhat.com
May 13 17:36:00 bumblebee pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks._queue_reserved_task[0e90b376-d589-4261-bf8f-5afcb8514c91]
succeeded in 0.0365472619887s: None
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) ('Connection aborted.', error(110, 'Connection timed out'))
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) Traceback (most recent call last):
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) File
"/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line
190, in fetch
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) response = session.get(request.url, headers=headers)
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) File
"/usr/lib/python2.7/site-packages/requests/sessions.py", line 469, in get
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) return self.request('GET', url, **kwargs)
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) File
"/usr/lib/python2.7/site-packages/requests/sessions.py", line 457, in
request
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) resp = self.send(prep, **send_kwargs)
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) File
"/usr/lib/python2.7/site-packages/requests/sessions.py", line 569, in send
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) r = adapter.send(request, **kwargs)
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) File
"/usr/lib/python2.7/site-packages/requests/adapters.py", line 407, in send
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) raise ConnectionError(err, request=request)
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) ConnectionError: ('Connection aborted.', error(110,
'Connection timed out'))
May 13 17:38:11 bumblebee pulp: pulp_rpm.plugins.importers.iso.sync:ERROR:
Failed to download
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/iso/PULP_MANIFEST:
('Connection aborted.', error(110, 'Connection timed out')).
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
Task
pulp.server.managers.repo.sync.sync[0e0bbc76-43de-4dd4-b45c-edff18a59644]
raised unexpected: PulpExecutionException('Importer indicated a failed
response',)
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
Traceback (most recent call last):
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in
trace_task
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
R = retval = fun(*args, **kwargs)
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line
328, incall
_
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
return super(Task, self).__call__(*args, **kwargs)
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in
protected_call
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
return self.run(*args, **kwargs)
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py",
line 114, in sync
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
raise PulpExecutionException(_('Importer indicated a failed response'))
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
PulpExecutionException: Importer indicated a failed response
May 13 17:38:12 bumblebee pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks._release_resource[09882136-6b6b-422d-bd55-3d312943cad3]
succeeded in 0.00555812800303s: None
==========================================================================

=========== successful curl command ======================================
[root bumblebee ~]# curl -v \
--proxy $http_proxy \
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/iso/PULP_MANIFEST
\
--insecure \
--cert /etc/pki/entitlement/$keyid.pem \
--key /etc/pki/entitlement/$keyid-key.pem

  • About to connect() to proxy XXX.XXX.XXX.XXX port YY (#0)
  • Trying XXX.XXX.XXX.XXX...
  • Connected to XXX.XXX.XXX.XXX (XXX.XXX.XXX.XXX) port YY (#0)
  • Establish HTTP proxy tunnel to cdn.redhat.com:443
  • Proxy auth using Basic with user 'USER'

CONNECT cdn.redhat.com:443 HTTP/1.1
Host: cdn.redhat.com:443
Proxy-Authorization: Basic XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
User-Agent: curl/7.29.0
Proxy-Connection: Keep-Alive

< HTTP/1.1 200 Connection established
<

  • Proxy replied OK to CONNECT request
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • skipping SSL peer certificate verification
  • NSS: client certificate from file
  • subject: CN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  • start date: Dec 09 05:00:00 2013 GMT
  • expire date: Dec 09 04:59:59 2016 GMT
  • common name: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  • issuer: E=ca-support redhat com,CN=Red Hat Candlepin Authority,OU=Red
    Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US
  • SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
  • Server certificate:
  • subject: CN=cdn.redhat.com,OU=Red Hat Network,O=Red
    Hat,L=Raleigh,ST=North Carolina,C=US
  • start date: May 14 19:48:02 2014 GMT
  • expire date: May 11 19:48:02 2024 GMT
  • common name: cdn.redhat.com
  • issuer: E=ca-support redhat com,CN=Red Hat Entitlement Operations
    Authority,OU=Red Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US

GET /content/dist/rhel/server/7/7Server/x86_64/iso/PULP_MANIFEST HTTP/1.1
User-Agent: curl/7.29.0
Host: cdn.redhat.com
Accept: */*

< HTTP/1.1 200 OK
< Server: Apache
< ETag: "dde1e965a48e56cba8aac0b597f0c7a6:1426504103"
< Last-Modified: Mon, 16 Mar 2015 11:08:16 GMT
< Accept-Ranges: bytes
< Content-Length: 424
< Content-Type: text/plain
< X-Akamai-Request-ID: 5c57dff5.19737949
< Date: Wed, 13 May 2015 07:14:12 GMT
< X-Cache: TCP_MISS from a104-71-131-23.deploy.akamaitechnologies.com
(AkamaiGHost/7.2.0-15182023) (-)
< Connection: keep-alive
< EJ-HOST: edgejavaapp5.prod.a4.vary.redhat.com
<
RedHat-PassSync-1.1.6-i386.msi,0cf9b1e6998f756effc051a193c764e6f9976ac00b506db1381f8649404ec534,3059712
RedHat-PassSync-1.1.6-x86_64.msi,5430ac6ccb26340db4da82235e0bbd70c4a21f7ad7e7cd6d953159ea2f1c80df,3399680
rhel-server-7.1-x86_64-boot.iso,b6c57194130025cecc036676d43235508f4031d99041051ea1c61e259bd95903,358612992
rhel-server-7.1-x86_64-dvd.iso,3685468ec6cdcb70dfc85ebbc164da427dc2d762644c3c2ee1520f4f661c15ce,3890216960

  • Connection #0 to host 10.226.8.34 left intact
    [root bumblebee ~]#
    ==========================================================================

=========== pulp rpm versions ============================================
[root bumblebee conf.d]# rpm -qa | grep pulp
pulp-puppet-admin-extensions-2.6.1-1.el7.noarch
python-pulp-client-lib-2.6.1-1.el7.noarch
pulp-selinux-2.6.1-1.el7.noarch
pulp-rpm-plugins-2.6.1-1.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
pulp-server-2.6.1-1.el7.noarch
python-pulp-rpm-common-2.6.1-1.el7.noarch
pulp-rpm-admin-extensions-2.6.1-1.el7.noarch
python-kombu-3.0.24-5.pulp.el7.noarch
python-pulp-puppet-common-2.6.1-1.el7.noarch
pulp-puppet-plugins-2.6.1-1.el7.noarch
pulp-admin-client-2.6.1-1.el7.noarch
python-pulp-common-2.6.1-1.el7.noarch
python-pulp-bindings-2.6.1-1.el7.noarch
==========================================================================

Also available in: Atom PDF