Project

Profile

Help

Issue #983

closed

pulp iso sync failure

Added by Ben.Stanley almost 9 years ago. Updated almost 5 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.6.1 Beta
Platform Release:
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Documentation, Pulp 2
Sprint:
Quarter:

Description

I have observed a repeatable pulp sync failure of an ISO repo, where
curl managed to fetch the PULP_MANIFEST successfully.

The pulp sync failure reason appears to be the following log message:

May 13 17:38:11 bumblebee pulp: pulp_rpm.plugins.importers.iso.sync:ERROR:
Failed to download
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/iso/PULP_MANIFEST:
('Connection aborted.', error(110, 'Connection timed out')).

The timeout occurs 2 min after the previous log message.

Under the same situation, curl fetches the same URL successfully every time.
On the other hand, pulp fails every time.
This result is observed with interleaved tests.

The following list summarises all the details that can be found below.

  • pulp-admin iso repo create
  • pulp-admin iso repo sync
  • /var/log/pulp.log containing failure
  • successful curl command
  • pulp rpm versions

I am situated behind a proxy server, which pulp can use successfully on other repos. The proxy server is configured in /etc/pulp/server/plugins.conf.d/yum_importer.json .

=========== pulp-admin iso repo create ===================================
[root bumblebee conf.d]# pulp-admin iso repo create \
--repo-id=rhel-7-server-x86_64-iso \
--feed=https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/iso
\
--feed-ca-cert=/etc/rhsm/ca/redhat-uep.pem \
--feed-cert=/etc/pki/entitlement/$keyid.pem \
--feed-key=/etc/pki/entitlement/$keyid-key.pem \
--validate true \
--serve-http true
Successfully created repository [rhel-7-server-x86_64-iso]
==========================================================================

=========== pulp-admin iso repo sync run =================================
[root bumblebee conf.d]# date
Wed May 13 17:35:56 AEST 2015
[root bumblebee conf.d]# pulp-admin iso repo sync run
--repo-id=rhel-7-server-x86_64-iso
--------------------------------------------------------------------
Synchronizing Repository [rhel-7-server-x86_64-iso]
--------------------------------------------------------------------

This command may be exited via ctrl+c without affecting the request.

Downloading the Pulp Manifest...
Downloading the Pulp Manifest failed:

{}

Task Failed

Importer indicated a failed response

[root bumblebee conf.d]# date
Wed May 13 17:38:23 AEST 2015
[root bumblebee conf.d]#
==========================================================================

=========== /var/log/pulp.log ============================================
May 13 17:36:00 bumblebee pulp: celery.worker.strategy:INFO: Received
task:
pulp.server.async.tasks._queue_reserved_task[0e90b376-d589-4261-bf8f-5afcb8514c91]
May 13 17:36:00 bumblebee pulp: celery.worker.strategy:INFO: Received
task:
pulp.server.managers.repo.sync.sync[0e0bbc76-43de-4dd4-b45c-edff18a59644]
May 13 17:36:00 bumblebee pulp: celery.worker.strategy:INFO: Received
task:
pulp.server.async.tasks._release_resource[09882136-6b6b-422d-bd55-3d312943cad3]
May 13 17:36:00 bumblebee pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
connection (1): cdn.redhat.com
May 13 17:36:00 bumblebee pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks._queue_reserved_task[0e90b376-d589-4261-bf8f-5afcb8514c91]
succeeded in 0.0365472619887s: None
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) ('Connection aborted.', error(110, 'Connection timed out'))
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) Traceback (most recent call last):
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) File
"/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line
190, in fetch
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) response = session.get(request.url, headers=headers)
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) File
"/usr/lib/python2.7/site-packages/requests/sessions.py", line 469, in get
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) return self.request('GET', url, **kwargs)
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) File
"/usr/lib/python2.7/site-packages/requests/sessions.py", line 457, in
request
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) resp = self.send(prep, **send_kwargs)
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) File
"/usr/lib/python2.7/site-packages/requests/sessions.py", line 569, in send
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) r = adapter.send(request, **kwargs)
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) File
"/usr/lib/python2.7/site-packages/requests/adapters.py", line 407, in send
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) raise ConnectionError(err, request=request)
May 13 17:38:11 bumblebee pulp: nectar.downloaders.threaded:ERROR:
(17541-47008) ConnectionError: ('Connection aborted.', error(110,
'Connection timed out'))
May 13 17:38:11 bumblebee pulp: pulp_rpm.plugins.importers.iso.sync:ERROR:
Failed to download
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/iso/PULP_MANIFEST:
('Connection aborted.', error(110, 'Connection timed out')).
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
Task
pulp.server.managers.repo.sync.sync[0e0bbc76-43de-4dd4-b45c-edff18a59644]
raised unexpected: PulpExecutionException('Importer indicated a failed
response',)
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
Traceback (most recent call last):
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in
trace_task
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
R = retval = fun(*args, **kwargs)
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line
328, incall
_
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
return super(Task, self).__call__(*args, **kwargs)
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in
protected_call
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
return self.run(*args, **kwargs)
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py",
line 114, in sync
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
raise PulpExecutionException(_('Importer indicated a failed response'))
May 13 17:38:12 bumblebee pulp: celery.worker.job:ERROR: (17431-11936)
PulpExecutionException: Importer indicated a failed response
May 13 17:38:12 bumblebee pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks._release_resource[09882136-6b6b-422d-bd55-3d312943cad3]
succeeded in 0.00555812800303s: None
==========================================================================

=========== successful curl command ======================================
[root bumblebee ~]# curl -v \
--proxy $http_proxy \
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/iso/PULP_MANIFEST
\
--insecure \
--cert /etc/pki/entitlement/$keyid.pem \
--key /etc/pki/entitlement/$keyid-key.pem

  • About to connect() to proxy XXX.XXX.XXX.XXX port YY (#0)
  • Trying XXX.XXX.XXX.XXX...
  • Connected to XXX.XXX.XXX.XXX (XXX.XXX.XXX.XXX) port YY (#0)
  • Establish HTTP proxy tunnel to cdn.redhat.com:443
  • Proxy auth using Basic with user 'USER'

CONNECT cdn.redhat.com:443 HTTP/1.1
Host: cdn.redhat.com:443
Proxy-Authorization: Basic XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
User-Agent: curl/7.29.0
Proxy-Connection: Keep-Alive

< HTTP/1.1 200 Connection established
<

  • Proxy replied OK to CONNECT request
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • skipping SSL peer certificate verification
  • NSS: client certificate from file
  • subject: CN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  • start date: Dec 09 05:00:00 2013 GMT
  • expire date: Dec 09 04:59:59 2016 GMT
  • common name: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  • issuer: E=ca-support redhat com,CN=Red Hat Candlepin Authority,OU=Red
    Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US
  • SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
  • Server certificate:
  • subject: CN=cdn.redhat.com,OU=Red Hat Network,O=Red
    Hat,L=Raleigh,ST=North Carolina,C=US
  • start date: May 14 19:48:02 2014 GMT
  • expire date: May 11 19:48:02 2024 GMT
  • common name: cdn.redhat.com
  • issuer: E=ca-support redhat com,CN=Red Hat Entitlement Operations
    Authority,OU=Red Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US

GET /content/dist/rhel/server/7/7Server/x86_64/iso/PULP_MANIFEST HTTP/1.1
User-Agent: curl/7.29.0
Host: cdn.redhat.com
Accept: */*

< HTTP/1.1 200 OK
< Server: Apache
< ETag: "dde1e965a48e56cba8aac0b597f0c7a6:1426504103"
< Last-Modified: Mon, 16 Mar 2015 11:08:16 GMT
< Accept-Ranges: bytes
< Content-Length: 424
< Content-Type: text/plain
< X-Akamai-Request-ID: 5c57dff5.19737949
< Date: Wed, 13 May 2015 07:14:12 GMT
< X-Cache: TCP_MISS from a104-71-131-23.deploy.akamaitechnologies.com
(AkamaiGHost/7.2.0-15182023) (-)
< Connection: keep-alive
< EJ-HOST: edgejavaapp5.prod.a4.vary.redhat.com
<
RedHat-PassSync-1.1.6-i386.msi,0cf9b1e6998f756effc051a193c764e6f9976ac00b506db1381f8649404ec534,3059712
RedHat-PassSync-1.1.6-x86_64.msi,5430ac6ccb26340db4da82235e0bbd70c4a21f7ad7e7cd6d953159ea2f1c80df,3399680
rhel-server-7.1-x86_64-boot.iso,b6c57194130025cecc036676d43235508f4031d99041051ea1c61e259bd95903,358612992
rhel-server-7.1-x86_64-dvd.iso,3685468ec6cdcb70dfc85ebbc164da427dc2d762644c3c2ee1520f4f661c15ce,3890216960

  • Connection #0 to host 10.226.8.34 left intact
    [root bumblebee ~]#
    ==========================================================================

=========== pulp rpm versions ============================================
[root bumblebee conf.d]# rpm -qa | grep pulp
pulp-puppet-admin-extensions-2.6.1-1.el7.noarch
python-pulp-client-lib-2.6.1-1.el7.noarch
pulp-selinux-2.6.1-1.el7.noarch
pulp-rpm-plugins-2.6.1-1.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
pulp-server-2.6.1-1.el7.noarch
python-pulp-rpm-common-2.6.1-1.el7.noarch
pulp-rpm-admin-extensions-2.6.1-1.el7.noarch
python-kombu-3.0.24-5.pulp.el7.noarch
python-pulp-puppet-common-2.6.1-1.el7.noarch
pulp-puppet-plugins-2.6.1-1.el7.noarch
pulp-admin-client-2.6.1-1.el7.noarch
python-pulp-common-2.6.1-1.el7.noarch
python-pulp-bindings-2.6.1-1.el7.noarch
==========================================================================

Actions #1

Updated by rbarlow almost 9 years ago

  • Project changed from Pulp to RPM Support
  • Category deleted (15)
Actions #2

Updated by mhrivnak almost 9 years ago

Can you tell us more about the proxy? It looks like you used one with curl, but there isn't one in your pulp-admin repo create command. Do you have it globally configured for the iso importer?

Actions #3

Updated by Ben.Stanley almost 9 years ago

As mentioned in the original report:

I am situated behind a proxy server, which pulp can use successfully on other rpm repos. The proxy server is configured in /etc/pulp/server/plugins.conf.d/yum_importer.json .

The proxy server is BlueCoat.

Perhaps I need to configure the iso importer separately?

I looked up the documentation, and I found incomplete information on configuring the proxy.

https://pulp.readthedocs.org/en/latest/user-guide/server.html#plugins

Questions:

  1. If there are separate config files for each importer, what are the config files called? (put these in the documentation).
  2. Where in the documentation is the format of the /etc/pulp/server/plugins.conf.d/yum_importer.json file described? The documentation at https://pulp.readthedocs.org/en/latest/user-guide/server.html#plugins just says that:
    Most of Pulp's importers support these key-value settings in their config files:
    The format of the config files is not described.

The following is the (abstract) contents of my yum_importer.json file:

{
 "proxy_host" : "http://proxy.server",
 "proxy_port" : XX,
 "proxy_username" : "user",
 "proxy_password" : "password"
}
Actions #4

Updated by Ben.Stanley almost 9 years ago

I found some relevant information at
[[https://bugzilla.redhat.com/show_bug_cgi?id=985106]]
It seems I need to configure the proxy settings in the /etc/pulp/server/plugins.conf.d/iso_importer.json file.

I have copied my yum_importer.json to iso_importer.json , and re-started pulp, and I am now able to sync iso repos.

I recommend an improvement to the documentation. At the very least:

  1. The names of the configuration files for each plugin should be named.
  2. An example (perhaps common) should be given, showing the format of the settings.

I am sure I found my yum_importer.json file as an example file somewhere, but I can't find it again now.

Actions #5

Updated by jortel@redhat.com almost 9 years ago

  • Triaged changed from No to Yes
  • Tags Documentation added
Actions #6

Updated by bmbouter almost 5 years ago

  • Status changed from NEW to CLOSED - WONTFIX

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

Actions #7

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF