Project

Profile

Help

Issue #2588

Pulp Gets Stuck On A Particular Mirror

Added by ehelms@redhat.com almost 3 years ago. Updated 9 months ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Severity:
3. High
Version:
Platform Release:
Blocks Release:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:

Description

This is being filed on behalf of Jason Nance who originally opened an issue against the Katello project (http://projects.theforeman.org/issues/18506) and we have since moved it here.

Pulp 2.9.3, Squid 3.5.20

Product repository is configured as on demand, and points to a mirror list:

http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates

Client is trying to install a package which has not previously been downloaded, but Pulp is failing to download the package. Pulp says "Skipping requests to mirror.clarkson.edu due to repeated connection failures", but then retries the same mirror again.

Specific to this instance, http://mirror.clarkson.edu/centos/7.3.1611/os/x86_64/Packages/MySQL-python-1.2.5-1.el7.x86_64.rpm is unavailable (mirror.clarkson.edu is down in my area at this time).

If I run wget 'http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates' I get a different mirror list (nearly every time), but Pulp/Squid seems to not be refreshing its mirror list

Client (yum install MySQL-python) shows the following (10 times):

MySQL-python-1.2.5-1.el7.x86_6 FAILED
https://sl2mmgplsat0001.ipa.lab.gen.zone/pulp/repos/tkc-holdings/pnp-production/c7stable/custom/centos/c7/MySQL-python-1.2.5-1.el7.x86_64.rpm: [Errno 14] HTTPS Error 503 - Service Unavailable
Trying other mirror.

foreman-tail shows the following:

==> /var/log/messages <==
Feb 14 13:42:09 sl2mmgplsat0001 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (21): mirror.clarkson.edu

==> /var/log/httpd/foreman-ssl_access_ssl.log <==
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:42:09 -0600] "GET /pulp/repos/tkc-holdings/pnp-production/c7stable/custom/centos/c7/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 302 - "-" "urlgrabber/3.10 yum/3.4.3" 

==> /var/log/messages <==
Feb 14 13:42:15 sl2mmgplsat0001 pulp_streamer: nectar.downloaders.threaded:ERROR: Skipping requests to mirror.clarkson.edu due to repeated connection failures: HTTPConnectionPool(host='mirror.clarkson.edu', port=80): Max retries exceeded with url: /centos/7.3.1611/os/x86_64/Packages/MySQL-python-1.2.5-1.el7.x86_64.rpm (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x4690890>, 'Connection to mirror.clarkson.edu timed out. (connect timeout=6.05)'))
Feb 14 13:42:15 sl2mmgplsat0001 pulp_streamer: [-] 127.0.0.1 - - [14/Feb/2017:19:42:15 +0000] "GET /var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 
Feb 14 13:42:15 sl2mmgplsat0001 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (22): mirror.clarkson.edu

==> /var/log/httpd/foreman-ssl_access_ssl.log <==
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:42:09 -0600] "GET /streamer/var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm?policy=eyJleHRlbnNpb25zIjogeyJyZW1vdGVfaXAiOiAiMTcyLjE2LjI0Ni4zMyJ9LCAicmVzb3VyY2UiOiAiL3N0cmVhbWVyL3Zhci9saWIvcHVscC9jb250ZW50L3VuaXRzL3JwbS9lNS9mYmJlODFiZTdlYzhhNWM4NjMyNzIwMGIyMzNlOTc1MzNlYzc1NDQwNzBjZWM1NzhiM2VhZTA1MDIzN2E0Yy9NeVNRTC1weXRob24tMS4yLjUtMS5lbDcueDg2XzY0LnJwbSIsICJleHBpcmF0aW9uIjogMTQ4NzEwMTQxOX0%3D;signature=rjPNJ2oWHsEOD7h0WvE1zaCLnkg0Fd8ZLRpmZjBosv7VWgxfCVZSbBLHv_Khvgq9TDPdAYH6GLXu_ljj9vpkc_FFEtMO6PczQT9zGzMWu8Fm21vhGMDwejbVw34Z1K1ERcC53V25_WFczMwikfCYrpDnU83DMor5H60p53l1gC3zpb3pIPQI79WfccbbKveYx31W3mqBXXVSc0WoUS7HOmE3Zg-_zzX53Q24ERrwEJlkHBd0cFCqpYinok-yjniHObhfAEUNaH6BfZiJfAa1VkkIje4Jflyoh6Eta73u_VihZmGfN669svBmMP4VCiACltnSFvieQK6uAY8NwSObDQ%3D%3D HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:42:15 -0600] "GET /pulp/repos/tkc-holdings/pnp-production/c7stable/custom/centos/c7/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 302 - "-" "urlgrabber/3.10 yum/3.4.3" 

==> /var/log/messages <==
Feb 14 13:42:21 sl2mmgplsat0001 pulp_streamer: nectar.downloaders.threaded:ERROR: Skipping requests to mirror.clarkson.edu due to repeated connection failures: HTTPConnectionPool(host='mirror.clarkson.edu', port=80): Max retries exceeded with url: /centos/7.3.1611/os/x86_64/Packages/MySQL-python-1.2.5-1.el7.x86_64.rpm (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x4690890>, 'Connection to mirror.clarkson.edu timed out. (connect timeout=6.05)'))
Feb 14 13:42:21 sl2mmgplsat0001 pulp_streamer: [-] 127.0.0.1 - - [14/Feb/2017:19:42:21 +0000] "GET /var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 
Feb 14 13:42:21 sl2mmgplsat0001 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (23): mirror.clarkson.edu

==> /var/log/httpd/foreman-ssl_access_ssl.log <==
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:42:15 -0600] "GET /streamer/var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm?policy=eyJleHRlbnNpb25zIjogeyJyZW1vdGVfaXAiOiAiMTcyLjE2LjI0Ni4zMyJ9LCAicmVzb3VyY2UiOiAiL3N0cmVhbWVyL3Zhci9saWIvcHVscC9jb250ZW50L3VuaXRzL3JwbS9lNS9mYmJlODFiZTdlYzhhNWM4NjMyNzIwMGIyMzNlOTc1MzNlYzc1NDQwNzBjZWM1NzhiM2VhZTA1MDIzN2E0Yy9NeVNRTC1weXRob24tMS4yLjUtMS5lbDcueDg2XzY0LnJwbSIsICJleHBpcmF0aW9uIjogMTQ4NzEwMTQyNX0%3D;signature=bzUdGI6ZutyR95p8NrTVxYJQrSMz9eqYz4Z-FCuBO-PYlkx3sYksikaygltJV-VE6_ZdQNEeg8ttsVR8EtDktX3VSNU-5X6map6eWRDN2gcj_Wkz0SuQX9Q7kLO-GsWHcf8-mzL6YUzPmOKr9YZYSlnO_VYS8sPqx_FqtTeh2Ry17Jg452Y3D0ggfWxBU-O5xo1ggbM10aLaXjWXuRDDAtqg7PH-eMpbzdnFhyWnGYIpa-6VYXu9Rz2u3J4ZBj4Cejmn1zVvXyv4fcn6x3vauwcNJMan4SZt8cJhNN6Hx2ye_ySUkJ12mnSEkaA4SWUXEeG8Hf0OCOLgsK3gPbYC2w%3D%3D HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:42:21 -0600] "GET /pulp/repos/tkc-holdings/pnp-production/c7stable/custom/centos/c7/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 302 - "-" "urlgrabber/3.10 yum/3.4.3" 

==> /var/log/messages <==
Feb 14 13:42:27 sl2mmgplsat0001 pulp_streamer: nectar.downloaders.threaded:ERROR: Skipping requests to mirror.clarkson.edu due to repeated connection failures: HTTPConnectionPool(host='mirror.clarkson.edu', port=80): Max retries exceeded with url: /centos/7.3.1611/os/x86_64/Packages/MySQL-python-1.2.5-1.el7.x86_64.rpm (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x4690e10>, 'Connection to mirror.clarkson.edu timed out. (connect timeout=6.05)'))
Feb 14 13:42:27 sl2mmgplsat0001 pulp_streamer: [-] 127.0.0.1 - - [14/Feb/2017:19:42:27 +0000] "GET /var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 
Feb 14 13:42:27 sl2mmgplsat0001 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (24): mirror.clarkson.edu

==> /var/log/httpd/foreman-ssl_access_ssl.log <==
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:42:21 -0600] "GET /streamer/var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm?policy=eyJleHRlbnNpb25zIjogeyJyZW1vdGVfaXAiOiAiMTcyLjE2LjI0Ni4zMyJ9LCAicmVzb3VyY2UiOiAiL3N0cmVhbWVyL3Zhci9saWIvcHVscC9jb250ZW50L3VuaXRzL3JwbS9lNS9mYmJlODFiZTdlYzhhNWM4NjMyNzIwMGIyMzNlOTc1MzNlYzc1NDQwNzBjZWM1NzhiM2VhZTA1MDIzN2E0Yy9NeVNRTC1weXRob24tMS4yLjUtMS5lbDcueDg2XzY0LnJwbSIsICJleHBpcmF0aW9uIjogMTQ4NzEwMTQzMX0%3D;signature=tdkNdk4ZivUJ80DkFV0c0hiZodMhNmdQVe8-gbVJBVcoCRvlixFGb2DqhgTKNixnrTv1aV8bOVpJ4NWkkH15jLy6LZaQ9eXuc0GjczSNMpWUIJ_aYHnWbOnC6PN78_Xt3TwWzzD7evBQdMHTXKLCPrKZ2mhSRvQUflbm_4pe08Lf6IIW50-2owndJcfy0lUsMvGIO_pAdvyGkwi2FjlGSFJb_Tdl2qhlqrgS6uIQiQu_f7zbOt_mfZQmxFGi_sGoiCQ4k2_JEL1Xkf9qmCTItIJhkyX6tNcjnTX2hZ0HAiT7riooqFDZzZTw0zmHL1FuccjaDRlzBSShFRxmjlfUbw%3D%3D HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:42:27 -0600] "GET /pulp/repos/tkc-holdings/pnp-production/c7stable/custom/centos/c7/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 302 - "-" "urlgrabber/3.10 yum/3.4.3" 

==> /var/log/messages <==
Feb 14 13:42:33 sl2mmgplsat0001 pulp_streamer: nectar.downloaders.threaded:ERROR: Skipping requests to mirror.clarkson.edu due to repeated connection failures: HTTPConnectionPool(host='mirror.clarkson.edu', port=80): Max retries exceeded with url: /centos/7.3.1611/os/x86_64/Packages/MySQL-python-1.2.5-1.el7.x86_64.rpm (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x4690cd0>, 'Connection to mirror.clarkson.edu timed out. (connect timeout=6.05)'))
Feb 14 13:42:33 sl2mmgplsat0001 pulp_streamer: [-] 127.0.0.1 - - [14/Feb/2017:19:42:33 +0000] "GET /var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 
Feb 14 13:42:34 sl2mmgplsat0001 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (25): mirror.clarkson.edu

==> /var/log/httpd/foreman-ssl_access_ssl.log <==
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:42:27 -0600] "GET /streamer/var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm?policy=eyJleHRlbnNpb25zIjogeyJyZW1vdGVfaXAiOiAiMTcyLjE2LjI0Ni4zMyJ9LCAicmVzb3VyY2UiOiAiL3N0cmVhbWVyL3Zhci9saWIvcHVscC9jb250ZW50L3VuaXRzL3JwbS9lNS9mYmJlODFiZTdlYzhhNWM4NjMyNzIwMGIyMzNlOTc1MzNlYzc1NDQwNzBjZWM1NzhiM2VhZTA1MDIzN2E0Yy9NeVNRTC1weXRob24tMS4yLjUtMS5lbDcueDg2XzY0LnJwbSIsICJleHBpcmF0aW9uIjogMTQ4NzEwMTQzN30%3D;signature=Af0pcqiwj0XDRQrfcjRh6o4h4opZ8hrbn7IsyEoBHMQis2QyX8TqzM79eg_PVPcGxf1h4eY0RsxxeFqQOrTrw7DMyTXd6R1AS9fVKX0kI7GlJlnxqs7BX5QtemFhA1U4Q7ZdO_A16DcxUQW-swlkKzVQffYFMrARIbC4DcrNebJRJYyXKep4XSQ79u8CMtwNGoO_fvnOf9JeclLdguwZI5Zuzbm-CykEGJLDNjIu3fvZQRi6fpI-B6rv3X9ercFT5kSx5xp9_oRCiGkAAa95Y9YIua2iBRgefQCnxk0tp0_4StPKgEUNyagYkED6SRl7ucqbXbEM42ka7MLQaB6EAQ%3D%3D HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:42:33 -0600] "GET /pulp/repos/tkc-holdings/pnp-production/c7stable/custom/centos/c7/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 302 - "-" "urlgrabber/3.10 yum/3.4.3" 

==> /var/log/messages <==
Feb 14 13:42:40 sl2mmgplsat0001 pulp_streamer: nectar.downloaders.threaded:ERROR: Skipping requests to mirror.clarkson.edu due to repeated connection failures: HTTPConnectionPool(host='mirror.clarkson.edu', port=80): Max retries exceeded with url: /centos/7.3.1611/os/x86_64/Packages/MySQL-python-1.2.5-1.el7.x86_64.rpm (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x4690cd0>, 'Connection to mirror.clarkson.edu timed out. (connect timeout=6.05)'))
Feb 14 13:42:40 sl2mmgplsat0001 pulp_streamer: [-] 127.0.0.1 - - [14/Feb/2017:19:42:39 +0000] "GET /var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 
Feb 14 13:42:40 sl2mmgplsat0001 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (26): mirror.clarkson.edu

==> /var/log/httpd/foreman-ssl_access_ssl.log <==
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:42:33 -0600] "GET /streamer/var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm?policy=eyJleHRlbnNpb25zIjogeyJyZW1vdGVfaXAiOiAiMTcyLjE2LjI0Ni4zMyJ9LCAicmVzb3VyY2UiOiAiL3N0cmVhbWVyL3Zhci9saWIvcHVscC9jb250ZW50L3VuaXRzL3JwbS9lNS9mYmJlODFiZTdlYzhhNWM4NjMyNzIwMGIyMzNlOTc1MzNlYzc1NDQwNzBjZWM1NzhiM2VhZTA1MDIzN2E0Yy9NeVNRTC1weXRob24tMS4yLjUtMS5lbDcueDg2XzY0LnJwbSIsICJleHBpcmF0aW9uIjogMTQ4NzEwMTQ0M30%3D;signature=Ux8s1cuKeyzUVpa1spxeEVftblI9n1xC11aInsXzCeCcx9k276MM8KWMKRQUZP5XesAOQuy8PYmfGXVg0YoBFrHdcHEive6O2Ox_lD3ZkA8CcwMeBES-rkxwgQLKYCszm2_avzHJRHoVcSXp9e8Ng0__hhSDyYO6HDX3kRJe4N2BIwrFFnWn8K8fBW_v3Mx6EsiErJq8CqauGES3sRLZCrQxdsUwah_fCKe2nfSvhT__ES2P9jfAdQeUenHDdnwOVOI5AMrld6V9yujGtnRnYZbSnp0xwEngl0xfLHqRyxc9jwiVJ7eD0nSrGXEZ3wVLJ9FGqmbpPv_lBUE6zj538Q%3D%3D HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:42:40 -0600] "GET /pulp/repos/tkc-holdings/pnp-production/c7stable/custom/centos/c7/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 302 - "-" "urlgrabber/3.10 yum/3.4.3" 

==> /var/log/messages <==
Feb 14 13:42:46 sl2mmgplsat0001 pulp_streamer: nectar.downloaders.threaded:ERROR: Skipping requests to mirror.clarkson.edu due to repeated connection failures: HTTPConnectionPool(host='mirror.clarkson.edu', port=80): Max retries exceeded with url: /centos/7.3.1611/os/x86_64/Packages/MySQL-python-1.2.5-1.el7.x86_64.rpm (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x4690cd0>, 'Connection to mirror.clarkson.edu timed out. (connect timeout=6.05)'))
Feb 14 13:42:46 sl2mmgplsat0001 pulp_streamer: [-] 127.0.0.1 - - [14/Feb/2017:19:42:45 +0000] "GET /var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 
Feb 14 13:42:46 sl2mmgplsat0001 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (27): mirror.clarkson.edu

==> /var/log/httpd/foreman-ssl_access_ssl.log <==
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:42:40 -0600] "GET /streamer/var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm?policy=eyJleHRlbnNpb25zIjogeyJyZW1vdGVfaXAiOiAiMTcyLjE2LjI0Ni4zMyJ9LCAicmVzb3VyY2UiOiAiL3N0cmVhbWVyL3Zhci9saWIvcHVscC9jb250ZW50L3VuaXRzL3JwbS9lNS9mYmJlODFiZTdlYzhhNWM4NjMyNzIwMGIyMzNlOTc1MzNlYzc1NDQwNzBjZWM1NzhiM2VhZTA1MDIzN2E0Yy9NeVNRTC1weXRob24tMS4yLjUtMS5lbDcueDg2XzY0LnJwbSIsICJleHBpcmF0aW9uIjogMTQ4NzEwMTQ1MH0%3D;signature=S4A8_oVGDjM1hSBdKlmXFUICm-QOhytN4meGjWI5teMxSNGxuUF99hr__uy_PINxqFpHt5Xu055SkkUHQ5zjBRnmjK1L1A54hge0eXGCSFED9LwGw9X2CRVT7LaETj79F14pWXu5C108gvXDKhfpjOiNcXx0p8piM-w5niuKXsdwQi7zTLpZ41PmamI4qwpiDZvi8TArhfQN2hGuPtCsT1au5o-cgiXda01pk04q59ivt59CjLLboM-cHPzLqZkEEgw1JM9xL3HtRRr3Bzy2Soo8W6QIipbjVjoqak3nB0QDD---Op3o98W4LzuJuPbUJH1bNVd-ovJB1mEEnZiMcw%3D%3D HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:42:46 -0600] "GET /pulp/repos/tkc-holdings/pnp-production/c7stable/custom/centos/c7/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 302 - "-" "urlgrabber/3.10 yum/3.4.3" 

==> /var/log/messages <==
Feb 14 13:42:52 sl2mmgplsat0001 pulp_streamer: nectar.downloaders.threaded:ERROR: Skipping requests to mirror.clarkson.edu due to repeated connection failures: HTTPConnectionPool(host='mirror.clarkson.edu', port=80): Max retries exceeded with url: /centos/7.3.1611/os/x86_64/Packages/MySQL-python-1.2.5-1.el7.x86_64.rpm (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x4690cd0>, 'Connection to mirror.clarkson.edu timed out. (connect timeout=6.05)'))
Feb 14 13:42:52 sl2mmgplsat0001 pulp_streamer: [-] 127.0.0.1 - - [14/Feb/2017:19:42:52 +0000] "GET /var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 
Feb 14 13:42:52 sl2mmgplsat0001 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (28): mirror.clarkson.edu

==> /var/log/httpd/foreman-ssl_access_ssl.log <==
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:42:46 -0600] "GET /streamer/var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm?policy=eyJleHRlbnNpb25zIjogeyJyZW1vdGVfaXAiOiAiMTcyLjE2LjI0Ni4zMyJ9LCAicmVzb3VyY2UiOiAiL3N0cmVhbWVyL3Zhci9saWIvcHVscC9jb250ZW50L3VuaXRzL3JwbS9lNS9mYmJlODFiZTdlYzhhNWM4NjMyNzIwMGIyMzNlOTc1MzNlYzc1NDQwNzBjZWM1NzhiM2VhZTA1MDIzN2E0Yy9NeVNRTC1weXRob24tMS4yLjUtMS5lbDcueDg2XzY0LnJwbSIsICJleHBpcmF0aW9uIjogMTQ4NzEwMTQ1Nn0%3D;signature=JKLf9y6ohBUnUxcVAKZqNUxiyPyhaIpsm2aHdem2Dp9xwmhyRJbBRuhDkSedF6Ae9rMXawKTOSWofX-Fzyk9h3oOZwPwZdpvhnbmb5MLnYP45M-eE8n-FhsI5GFkZcv9TpauTJL_gzpUzKostL8fNuarL9wa_GQdbYVfTIu9SETHMFIRYxKV4Xr0P6DvDaA2v2wjaraXQja2dX8Vh9AkkJ3X0CSHvDdXpeuGe70rknuAj1d9quPpGHncewCbBXqEus2_X1TL7xpGhbZNf60R8wSMZ2Oj7162ZKG_xQJRg4l-8Am3LMPhSxJF7TCdwTxe8plHrvqisI8ofgbnpsEwlw%3D%3D HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:42:52 -0600] "GET /pulp/repos/tkc-holdings/pnp-production/c7stable/custom/centos/c7/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 302 - "-" "urlgrabber/3.10 yum/3.4.3" 

==> /var/log/messages <==
Feb 14 13:42:58 sl2mmgplsat0001 pulp_streamer: nectar.downloaders.threaded:ERROR: Skipping requests to mirror.clarkson.edu due to repeated connection failures: HTTPConnectionPool(host='mirror.clarkson.edu', port=80): Max retries exceeded with url: /centos/7.3.1611/os/x86_64/Packages/MySQL-python-1.2.5-1.el7.x86_64.rpm (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x4690cd0>, 'Connection to mirror.clarkson.edu timed out. (connect timeout=6.05)'))
Feb 14 13:42:58 sl2mmgplsat0001 pulp_streamer: [-] 127.0.0.1 - - [14/Feb/2017:19:42:58 +0000] "GET /var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 
Feb 14 13:42:58 sl2mmgplsat0001 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (29): mirror.clarkson.edu

==> /var/log/httpd/foreman-ssl_access_ssl.log <==
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:42:52 -0600] "GET /streamer/var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm?policy=eyJleHRlbnNpb25zIjogeyJyZW1vdGVfaXAiOiAiMTcyLjE2LjI0Ni4zMyJ9LCAicmVzb3VyY2UiOiAiL3N0cmVhbWVyL3Zhci9saWIvcHVscC9jb250ZW50L3VuaXRzL3JwbS9lNS9mYmJlODFiZTdlYzhhNWM4NjMyNzIwMGIyMzNlOTc1MzNlYzc1NDQwNzBjZWM1NzhiM2VhZTA1MDIzN2E0Yy9NeVNRTC1weXRob24tMS4yLjUtMS5lbDcueDg2XzY0LnJwbSIsICJleHBpcmF0aW9uIjogMTQ4NzEwMTQ2Mn0%3D;signature=pbkwa8nSLs9NEcgFkkgPbLoml54upZ4tZlyCV1wFzkZXHxOmNUjfMpV_Q4FUdWkp7K20i1kmZ4ZYPsxZsWy70qNPTARZREP8lTF1KVq0epsJn0e-fmw7lgz4McVkc_1vRREDhEOP4v6rGBYDw5-t37bIIfQQpjrGaFLe6Zw8VtLxkr_SQDKhKusDqUCyqxsVmkDI_qYN6eFQRw8JtYT8nfTTNpwAhL4kO6prYXs8wQcXmUrFLEuGTV-V1hCbRKEUTaL5xtIp3hdX9c0N5E4TKITXwisYjVZlYmt1owgXKUpKvIHh4j5mFopWjFokv1RAAylSOz8oHlAn9fDasTcPyw%3D%3D HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:42:58 -0600] "GET /pulp/repos/tkc-holdings/pnp-production/c7stable/custom/centos/c7/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 302 - "-" "urlgrabber/3.10 yum/3.4.3" 

==> /var/log/messages <==
Feb 14 13:43:04 sl2mmgplsat0001 pulp_streamer: nectar.downloaders.threaded:ERROR: Skipping requests to mirror.clarkson.edu due to repeated connection failures: HTTPConnectionPool(host='mirror.clarkson.edu', port=80): Max retries exceeded with url: /centos/7.3.1611/os/x86_64/Packages/MySQL-python-1.2.5-1.el7.x86_64.rpm (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x4690cd0>, 'Connection to mirror.clarkson.edu timed out. (connect timeout=6.05)'))
Feb 14 13:43:04 sl2mmgplsat0001 pulp_streamer: [-] 127.0.0.1 - - [14/Feb/2017:19:43:04 +0000] "GET /var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 
Feb 14 13:43:04 sl2mmgplsat0001 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (30): mirror.clarkson.edu

==> /var/log/httpd/foreman-ssl_access_ssl.log <==
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:42:58 -0600] "GET /streamer/var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm?policy=eyJleHRlbnNpb25zIjogeyJyZW1vdGVfaXAiOiAiMTcyLjE2LjI0Ni4zMyJ9LCAicmVzb3VyY2UiOiAiL3N0cmVhbWVyL3Zhci9saWIvcHVscC9jb250ZW50L3VuaXRzL3JwbS9lNS9mYmJlODFiZTdlYzhhNWM4NjMyNzIwMGIyMzNlOTc1MzNlYzc1NDQwNzBjZWM1NzhiM2VhZTA1MDIzN2E0Yy9NeVNRTC1weXRob24tMS4yLjUtMS5lbDcueDg2XzY0LnJwbSIsICJleHBpcmF0aW9uIjogMTQ4NzEwMTQ2OH0%3D;signature=LTIdtE_1BOI8k4lfB6YmUjuqQ-invKsucjBRFrc2xdMTKLBWf4zDyteI-P22DJFk6aj9tzOLY5-FYFnpWefNAW34L31ZXzMaxEIS345zl6tjiE-y36WnsMOSyI1D7lnIvOPyd8QmKUDS-gT4BI2PNCd_CwNFWH4o1eYd0KZGwiU-v8qp_CScsZSJSz_A7GCEAO7JvWS99OyQoukvm7Q3cOY_QtXnbEmyqs8_mofzJ14FN2GZUvaaAJO_u4NMyDK4iAJoHnx939vQNBy1Fa31bDeElu5mONQdJ1RjpqH8L_9qdgVt_sgRVhhjgmE30skL7YQXclUgt_vnFmVF5bhSng%3D%3D HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:43:04 -0600] "GET /pulp/repos/tkc-holdings/pnp-production/c7stable/custom/centos/c7/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 302 - "-" "urlgrabber/3.10 yum/3.4.3" 

==> /var/log/messages <==
Feb 14 13:43:10 sl2mmgplsat0001 pulp_streamer: nectar.downloaders.threaded:ERROR: Skipping requests to mirror.clarkson.edu due to repeated connection failures: HTTPConnectionPool(host='mirror.clarkson.edu', port=80): Max retries exceeded with url: /centos/7.3.1611/os/x86_64/Packages/MySQL-python-1.2.5-1.el7.x86_64.rpm (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x4690cd0>, 'Connection to mirror.clarkson.edu timed out. (connect timeout=6.05)'))
Feb 14 13:43:10 sl2mmgplsat0001 pulp_streamer: [-] 127.0.0.1 - - [14/Feb/2017:19:43:10 +0000] "GET /var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 

==> /var/log/httpd/foreman-ssl_access_ssl.log <==
172.16.246.33 - 2c9076bc5a25cd69015a3dc17f910176 [14/Feb/2017:13:43:04 -0600] "GET /streamer/var/lib/pulp/content/units/rpm/e5/fbbe81be7ec8a5c86327200b233e97533ec7544070cec578b3eae050237a4c/MySQL-python-1.2.5-1.el7.x86_64.rpm?policy=eyJleHRlbnNpb25zIjogeyJyZW1vdGVfaXAiOiAiMTcyLjE2LjI0Ni4zMyJ9LCAicmVzb3VyY2UiOiAiL3N0cmVhbWVyL3Zhci9saWIvcHVscC9jb250ZW50L3VuaXRzL3JwbS9lNS9mYmJlODFiZTdlYzhhNWM4NjMyNzIwMGIyMzNlOTc1MzNlYzc1NDQwNzBjZWM1NzhiM2VhZTA1MDIzN2E0Yy9NeVNRTC1weXRob24tMS4yLjUtMS5lbDcueDg2XzY0LnJwbSIsICJleHBpcmF0aW9uIjogMTQ4NzEwMTQ3NH0%3D;signature=mtpT_N2hwt4gwIgzs51gVBhuikaEDXyoG3MDUPmQOs8NIeh004k1V5OWffJvec-MjKMKi2F8TC7nIiBj3KByqAgCiHiPs1t6JR2M0PYje8RxRFfAIncYVSI1SIRJsFOCNFEcwqd_d8AvWTeIDENPXeiueDRiA0u4HRCmq_ZTs5Ht_fk0s3X0uPRQdclbCLncmdPzjMzt7MItk4T-c0uSpN5-fr04qmj-6vLK6fHGOh4XkvF0WXJJLMr9jvWkSkqF0KDiAzspHrZc36AjHTOjAB6qBWWs7c-0_61fjJXZi4RkFv7lLoqnuS91s7r4Em64G9ROks_VBLttoJdJOSoUPw%3D%3D HTTP/1.1" 503 - "-" "urlgrabber/3.10 yum/3.4.3" 

History

#1 Updated by pthomas@redhat.com almost 3 years ago

  • Smash Test set to 560

#2 Updated by bizhang almost 3 years ago

Jortel will investigate this before we triage.

#3 Updated by bizhang almost 3 years ago

  • Triaged changed from No to Yes
  • Tags Pulp 3 added

#4 Updated by jason@tresgeek.net almost 3 years ago

For those who may run into this issue one way I have found to get around it is to find another mirror which uses the same directory (URI) structure and then add an entry in /etc/hosts on your Pulp server to point the host name of the broken mirror to the IP of a working mirror. Of course, there are lots of caveats here such as "doesn't work with https" and "doesn't work if working mirror uses name-based virtual host".

#5 Updated by dkliban@redhat.com almost 3 years ago

  • Tags deleted (Pulp 3)

#6 Updated by dkliban@redhat.com almost 3 years ago

  • Triaged changed from Yes to No

#7 Updated by dkliban@redhat.com almost 3 years ago

The issue has had it's 'Triaged' flag set to False so that it will come up again when we hold bug triage tomorrow. Most likely we will not have a plan of action by then, but I hope to discuss it in more detail during our triage on Friday. Feel free to join us0 if you want to participate in the discussion.

[0] http://docs.pulpproject.org/dev-guide/contributing/bugs.html#triage-process

#8 Updated by ttereshc almost 3 years ago

@jortel will take a closer look and we will triage it next time.

#9 Updated by jortel@redhat.com over 2 years ago

The way the RPM plugin handles mirror list is as follows:

1. Fetch the list of mirrors.
2. Iterate the mirrors trying to perform the entire sync against the URL.
3. Download (immediate) from the selected mirror.
4. Create catalog entries for the selected mirror.

This approach is flawed. Especially with regard to catalog entries because it populates the catalog with the mirror selected when the sync occurs. If the mirror list changes or the selected mirror goes off line, the sync will fail (perhaps permanently). Changing this for pulp2 would require a significant change in the importer flow. The design for pulp3 is significantly different and will store the mirror list url in the catalog and resolve the mirrors at download time and properly try all the mirrors (round-robin, nearest etc).

I do not recommend we try to fix this for pulp2.

#10 Updated by ttereshc over 2 years ago

  • Severity changed from 2. Medium to 3. High

#11 Updated by ttereshc over 2 years ago

  • Triaged changed from No to Yes

#12 Updated by lzap@redhat.com about 1 year ago

Guys, I am experiencing similar error when syncing repos via my transparent proxy configured as:

https://www.berrange.com/posts/2015/12/09/setting-up-a-local-caching-proxy-for-fedora-yum-repositories/

#13 Updated by bmbouter 9 months ago

  • Status changed from NEW to CLOSED - WONTFIX

#14 Updated by bmbouter 9 months ago

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

#15 Updated by bmbouter 9 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF