Actions
Issue #8130
open
Installer is doubling http on apache snippets
Added by fao89 almost 4 years ago. Updated about 3 years ago.
Status:
POST
Priority:
Normal
Assignee:
Category:
Installer - Moved to GitHub issues
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
(Total: 0:00 h)
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:
Description
Currently, apache and nginx have a scheme inconsistency:
nginx: pulp-api = 127.0.0.1
apache: pulp-api = http://127.0.0.1
which means:
http://pulp-api on nginx = http://127.0.0.1
http://${pulp-api} on apache = http://http://127.0.0.1
It causes proxy error when running plugins on apache.
ProxyPass /token http://${pulp-api}/token
ProxyPassReverse /token http://${pulp-api}/token
Both nginx and apache uses pulp_api_bind, but nginx does use "raw" pulp_api_bind:
While apache injects "http://" on it:
Example of plugin snippets:
Updated by mdepaulo@redhat.com almost 4 years ago
- Triaged changed from No to Yes
Updated by ipanova@redhat.com over 3 years ago
here are some logs from the server
# tail -f /etc/httpd/logs/pulp_error.log
[Wed Apr 14 18:23:36.316989 2021] [proxy:debug] [pid 26158] proxy_util.c(2262): [client 192.168.121.127:48244] AH00944: connecting http://[http:]//127.0.0.1:24817/v2/ocp3-copy/manifests/v3.3.0.35 to http::80
[Wed Apr 14 18:23:36.318505 2021] [proxy:error] [pid 26158] [client 192.168.121.127:48244] AH00898: DNS lookup failure for: http: returned by /v2/ocp3-copy/manifests/v3.3.0.35
[Wed Apr 14 18:23:36.318521 2021] [proxy:debug] [pid 26158] proxy_util.c(2224): AH00943: HTTP: has released connection for (*)
[Wed Apr 14 18:23:36.318589 2021] [http:trace3] [pid 26158] http_filters.c(1129): [client 192.168.121.127:48244] Response sent with status 502, headers:
[Wed Apr 14 18:23:36.318596 2021] [http:trace5] [pid 26158] http_filters.c(1136): [client 192.168.121.127:48244] Date: Wed, 14 Apr 2021 18:23:36 GMT
[Wed Apr 14 18:23:36.318598 2021] [http:trace5] [pid 26158] http_filters.c(1139): [client 192.168.121.127:48244] Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5
[Wed Apr 14 18:23:36.318601 2021] [http:trace4] [pid 26158] http_filters.c(958): [client 192.168.121.127:48244] Content-Length: 440
[Wed Apr 14 18:23:36.318603 2021] [http:trace4] [pid 26158] http_filters.c(958): [client 192.168.121.127:48244] Keep-Alive: timeout=5, max=10000
[Wed Apr 14 18:23:36.318604 2021] [http:trace4] [pid 26158] http_filters.c(958): [client 192.168.121.127:48244] Connection: Keep-Alive
[Wed Apr 14 18:23:36.318606 2021] [http:trace4] [pid 26158] http_filters.c(958): [client 192.168.121.127:48244] Content-Type: text/html; charset=iso-8859-1
[Wed Apr 14 18:23:44.902313 2021] [core:trace5] [pid 26156] protocol.c(656): [client 192.168.121.127:48246] Request received from client: GET /v2/ocp3-copy/manifests/v3.3.0.35 HTTP/1.1
[Wed Apr 14 18:23:44.902447 2021] [http:trace4] [pid 26156] http_request.c(323): [client 192.168.121.127:48246] Headers received from client:
[Wed Apr 14 18:23:44.902467 2021] [http:trace4] [pid 26156] http_request.c(327): [client 192.168.121.127:48246] Host: pulp2-nightly-pulp3-source-centos7.lan
[Wed Apr 14 18:23:44.902472 2021] [http:trace4] [pid 26156] http_request.c(327): [client 192.168.121.127:48246] User-Agent: HTTPie/2.4.0
[Wed Apr 14 18:23:44.902488 2021] [http:trace4] [pid 26156] http_request.c(327): [client 192.168.121.127:48246] Accept-Encoding: gzip, deflate
[Wed Apr 14 18:23:44.902494 2021] [http:trace4] [pid 26156] http_request.c(327): [client 192.168.121.127:48246] Accept: */*
[Wed Apr 14 18:23:44.902510 2021] [http:trace4] [pid 26156] http_request.c(327): [client 192.168.121.127:48246] Connection: keep-alive
[Wed Apr 14 18:23:44.902597 2021] [authz_core:debug] [pid 26156] mod_authz_core.c(835): [client 192.168.121.127:48246] AH01628: authorization result: granted (no directives)
[Wed Apr 14 18:23:44.902612 2021] [core:trace3] [pid 26156] request.c(312): [client 192.168.121.127:48246] request authorized without authentication by access_checker_ex hook: /v2/ocp3-copy/manifests/v3.3.0.35
[Wed Apr 14 18:23:44.902672 2021] [proxy_http:trace1] [pid 26156] mod_proxy_http.c(60): [client 192.168.121.127:48246] HTTP: canonicalising URL //http://127.0.0.1:24817/v2/ocp3-copy/manifests/v3.3.0.35
[Wed Apr 14 18:23:44.902766 2021] [proxy:trace2] [pid 26156] proxy_util.c(2016): [client 192.168.121.127:48246] *: found reverse proxy worker for http://[http:]//127.0.0.1:24817/v2/ocp3-copy/manifests/v3.3.0.35
[Wed Apr 14 18:23:44.902781 2021] [proxy:debug] [pid 26156] mod_proxy.c(1123): [client 192.168.121.127:48246] AH01143: Running scheme http handler (attempt 0)
[Wed Apr 14 18:23:44.902804 2021] [proxy_ajp:debug] [pid 26156] mod_proxy_ajp.c(722): [client 192.168.121.127:48246] AH00894: declining URL http://[http:]//127.0.0.1:24817/v2/ocp3-copy/manifests/v3.3.0.35
[Wed Apr 14 18:23:44.902866 2021] [proxy_fcgi:debug] [pid 26156] mod_proxy_fcgi.c(972): [client 192.168.121.127:48246] AH01076: url: http://[http:]//127.0.0.1:24817/v2/ocp3-copy/manifests/v3.3.0.35 proxyname: (null) proxyport: 0
[Wed Apr 14 18:23:44.902873 2021] [proxy_fcgi:debug] [pid 26156] mod_proxy_fcgi.c(975): [client 192.168.121.127:48246] AH01077: declining URL http://[http:]//127.0.0.1:24817/v2/ocp3-copy/manifests/v3.3.0.35
[Wed Apr 14 18:23:44.902880 2021] [proxy_http:trace1] [pid 26156] mod_proxy_http.c(1939): [client 192.168.121.127:48246] HTTP: serving URL http://[http:]//127.0.0.1:24817/v2/ocp3-copy/manifests/v3.3.0.35
[Wed Apr 14 18:23:44.902892 2021] [proxy:debug] [pid 26156] proxy_util.c(2209): AH00942: HTTP: has acquired connection for (*)
[Wed Apr 14 18:23:44.902923 2021] [proxy:debug] [pid 26156] proxy_util.c(2262): [client 192.168.121.127:48246] AH00944: connecting http://[http:]//127.0.0.1:24817/v2/ocp3-copy/manifests/v3.3.0.35 to http::80
[Wed Apr 14 18:23:44.903914 2021] [proxy:error] [pid 26156] [client 192.168.121.127:48246] AH00898: DNS lookup failure for: http: returned by /v2/ocp3-copy/manifests/v3.3.0.35
[Wed Apr 14 18:23:44.903938 2021] [proxy:debug] [pid 26156] proxy_util.c(2224): AH00943: HTTP: has released connection for (*)
This is was got executed on the host machine
$ http pulp2-nightly-pulp3-source-centos7.lan/v2/ocp3-copy/manifests/v3.3.0.35
HTTP/1.1 502 Proxy Error
Connection: Keep-Alive
Content-Length: 440
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 14 Apr 2021 18:23:44 GMT
Keep-Alive: timeout=5, max=10000
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request <em><a href="/v2/ocp3-copy/manifests/v3.3.0.35">GET /v2/ocp3-copy/manifests/v3.3.0.35</a></em>.<p>
Reason: <strong>DNS lookup failure for: http:</strong></p></p>
</body></html>
After I have changed the snipped config, restarted apache in the server I was able to get a 302 but eventually got to 404 is following the redirect.
$ git diff
diff --git a/pulp_container/app/webserver_snippets/apache.conf b/pulp_container/app/webserver_snippets/apache.conf
index c03008c..dc28ee3 100644
--- a/pulp_container/app/webserver_snippets/apache.conf
+++ b/pulp_container/app/webserver_snippets/apache.conf
@@ -1,8 +1,8 @@
-ProxyPass /v2 http://${pulp-api}/v2
-ProxyPassReverse /v2 http://${pulp-api}/v2
+ProxyPass /v2 ${pulp-api}/v2
+ProxyPassReverse /v2 ${pulp-api}/v2
-ProxyPass /pulp/container http://${pulp-content}/v2
-ProxyPassReverse /pulp/container http://${pulp-content}/v2
+ProxyPass /pulp/container ${pulp-content}/v2 nocanon
+ProxyPassReverse /pulp/container ${pulp-content}/v2
-ProxyPass /token http://${pulp-api}/token
-ProxyPassReverse /token http://${pulp-api}/token
+ProxyPass /token ${pulp-api}/token nocanon
+ProxyPassReverse /token ${pulp-api}/token
[ipanova@fluffy pulp_container]$
Here is what i got on the host
$ http pulp2-nightly-pulp3-source-centos7.lan/v2/ocp3-copy/manifests/v3.3.0.35
HTTP/1.1 302 Found
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, PUT, HEAD, OPTIONS
Connection: Keep-Alive
Content-Length: 0
Content-Type: text/html; charset=utf-8
Correlation-ID: ed620b093f8b4814ac3f0a723f2d44cd
Date: Wed, 14 Apr 2021 18:22:03 GMT
Docker-Distribution-Api-Version: registry/2.0
Keep-Alive: timeout=5, max=10000
Location: http://pulp2-nightly-pulp3-source-centos7.lan/pulp/container/ocp3-copy/manifests/v3.3.0.35
Server: gunicorn/20.0.4
X-Frame-Options: SAMEORIGIN
$ http pulp2-nightly-pulp3-source-centos7.lan/v2/ocp3-copy/manifests/v3.3.0.35 --follow
HTTP/1.1 404 Not Found
Connection: Keep-Alive
Content-Length: 14
Content-Type: text/plain; charset=utf-8
Date: Wed, 14 Apr 2021 18:22:00 GMT
Keep-Alive: timeout=5, max=9999
Server: Python/3.6 aiohttp/3.7.4
404: Not Found
logs:
Wed Apr 14 18:29:15.519534 2021] [proxy:debug] [pid 26536] proxy_util.c(2224): AH00943: http: has released connection for (127.0.0.1)
[Wed Apr 14 18:29:15.519561 2021] [proxy:debug] [pid 26536] proxy_util.c(2926): [remote 127.0.0.1:24817] AH02642: proxy: connection shutdown
[Wed Apr 14 18:29:15.519622 2021] [http:trace3] [pid 26536] http_filters.c(1129): [client 192.168.121.127:48250] Response sent with status 302, headers:
[Wed Apr 14 18:29:15.519632 2021] [http:trace5] [pid 26536] http_filters.c(1136): [client 192.168.121.127:48250] Date: Wed, 14 Apr 2021 18:29:15 GMT
[Wed Apr 14 18:29:15.519634 2021] [http:trace5] [pid 26536] http_filters.c(1139): [client 192.168.121.127:48250] Server: gunicorn/20.0.4
[Wed Apr 14 18:29:15.519638 2021] [http:trace4] [pid 26536] http_filters.c(958): [client 192.168.121.127:48250] Content-Type: text/html; charset=utf-8
[Wed Apr 14 18:29:15.519640 2021] [http:trace4] [pid 26536] http_filters.c(958): [client 192.168.121.127:48250] Location: http://pulp2-nightly-pulp3-source-centos7.lan/pulp/container/ocp3-copy/manifests/v3.3.0.35
[Wed Apr 14 18:29:15.519642 2021] [http:trace4] [pid 26536] http_filters.c(958): [client 192.168.121.127:48250] Allow: GET, PUT, HEAD, OPTIONS
[Wed Apr 14 18:29:15.519644 2021] [http:trace4] [pid 26536] http_filters.c(958): [client 192.168.121.127:48250] Docker-Distribution-Api-Version: registry/2.0
[Wed Apr 14 18:29:15.519646 2021] [http:trace4] [pid 26536] http_filters.c(958): [client 192.168.121.127:48250] X-Frame-Options: SAMEORIGIN
[Wed Apr 14 18:29:15.519647 2021] [http:trace4] [pid 26536] http_filters.c(958): [client 192.168.121.127:48250] Content-Length: 0
[Wed Apr 14 18:29:15.519649 2021] [http:trace4] [pid 26536] http_filters.c(958): [client 192.168.121.127:48250] Correlation-ID: 0be206421e1b4b588faadab6b01c39c6
[Wed Apr 14 18:29:15.519651 2021] [http:trace4] [pid 26536] http_filters.c(958): [client 192.168.121.127:48250] Access-Control-Expose-Headers: Correlation-ID
[Wed Apr 14 18:29:15.519653 2021] [http:trace4] [pid 26536] http_filters.c(958): [client 192.168.121.127:48250] Keep-Alive: timeout=5, max=10000
[Wed Apr 14 18:29:15.519670 2021] [http:trace4] [pid 26536] http_filters.c(958): [client 192.168.121.127:48250] Connection: Keep-Alive
[Wed Apr 14 18:29:15.519859 2021] [proxy_http:trace2] [pid 26536] mod_proxy_http.c(1827): [client 192.168.121.127:48250] end body send
[Wed Apr 14 18:29:15.522497 2021] [core:trace5] [pid 26536] protocol.c(656): [client 192.168.121.127:48250] Request received from client: GET /pulp/container/ocp3-copy/manifests/v3.3.0.35 HTTP/1.1
[Wed Apr 14 18:29:15.522529 2021] [http:trace4] [pid 26536] http_request.c(323): [client 192.168.121.127:48250] Headers received from client:
[Wed Apr 14 18:29:15.522532 2021] [http:trace4] [pid 26536] http_request.c(327): [client 192.168.121.127:48250] Host: pulp2-nightly-pulp3-source-centos7.lan
[Wed Apr 14 18:29:15.522534 2021] [http:trace4] [pid 26536] http_request.c(327): [client 192.168.121.127:48250] User-Agent: HTTPie/2.4.0
[Wed Apr 14 18:29:15.522550 2021] [http:trace4] [pid 26536] http_request.c(327): [client 192.168.121.127:48250] Accept-Encoding: gzip, deflate
[Wed Apr 14 18:29:15.522553 2021] [http:trace4] [pid 26536] http_request.c(327): [client 192.168.121.127:48250] Accept: */*
[Wed Apr 14 18:29:15.522554 2021] [http:trace4] [pid 26536] http_request.c(327): [client 192.168.121.127:48250] Connection: keep-alive
[Wed Apr 14 18:29:15.522589 2021] [authz_core:debug] [pid 26536] mod_authz_core.c(835): [client 192.168.121.127:48250] AH01628: authorization result: granted (no directives)
[Wed Apr 14 18:29:15.522597 2021] [core:trace3] [pid 26536] request.c(312): [client 192.168.121.127:48250] request authorized without authentication by access_checker_ex hook: /pulp/container/ocp3-copy/manifests/v3.3.0.35
[Wed Apr 14 18:29:15.522627 2021] [proxy_http:trace1] [pid 26536] mod_proxy_http.c(60): [client 192.168.121.127:48250] HTTP: canonicalising URL //127.0.0.1:24816/v2/ocp3-copy/manifests/v3.3.0.35
[Wed Apr 14 18:29:15.522645 2021] [proxy:trace2] [pid 26536] proxy_util.c(1991): [client 192.168.121.127:48250] http: found worker http://127.0.0.1:24816/v2 for http://127.0.0.1:24816/v2/ocp3-copy/manifests/v3.3.0.35
[Wed Apr 14 18:29:15.522653 2021] [proxy:debug] [pid 26536] mod_proxy.c(1123): [client 192.168.121.127:48250] AH01143: Running scheme http handler (attempt 0)
[Wed Apr 14 18:29:15.522658 2021] [proxy_ajp:debug] [pid 26536] mod_proxy_ajp.c(722): [client 192.168.121.127:48250] AH00894: declining URL http://127.0.0.1:24816/v2/ocp3-copy/manifests/v3.3.0.35
[Wed Apr 14 18:29:15.522663 2021] [proxy_fcgi:debug] [pid 26536] mod_proxy_fcgi.c(972): [client 192.168.121.127:48250] AH01076: url: http://127.0.0.1:24816/v2/ocp3-copy/manifests/v3.3.0.35 proxyname: (null) proxyport: 0
[Wed Apr 14 18:29:15.522667 2021] [proxy_fcgi:debug] [pid 26536] mod_proxy_fcgi.c(975): [client 192.168.121.127:48250] AH01077: declining URL http://127.0.0.1:24816/v2/ocp3-copy/manifests/v3.3.0.35
[Wed Apr 14 18:29:15.522671 2021] [proxy_http:trace1] [pid 26536] mod_proxy_http.c(1939): [client 192.168.121.127:48250] HTTP: serving URL http://127.0.0.1:24816/v2/ocp3-copy/manifests/v3.3.0.35
[Wed Apr 14 18:29:15.522684 2021] [proxy:debug] [pid 26536] proxy_util.c(2209): AH00942: HTTP: has acquired connection for (127.0.0.1)
[Wed Apr 14 18:29:15.522688 2021] [proxy:debug] [pid 26536] proxy_util.c(2262): [client 192.168.121.127:48250] AH00944: connecting http://127.0.0.1:24816/v2/ocp3-copy/manifests/v3.3.0.35 to 127.0.0.1:24816
[Wed Apr 14 18:29:15.522833 2021] [proxy:debug] [pid 26536] proxy_util.c(2442): [client 192.168.121.127:48250] AH00947: connected /v2/ocp3-copy/manifests/v3.3.0.35 to 127.0.0.1:24816
[Wed Apr 14 18:29:15.522921 2021] [proxy:trace2] [pid 26536] proxy_util.c(2784): HTTP: fam 2 socket created to connect to 127.0.0.1
[Wed Apr 14 18:29:15.523074 2021] [proxy:debug] [pid 26536] proxy_util.c(2818): AH02824: HTTP: connection established with 127.0.0.1:24816 (127.0.0.1)
[Wed Apr 14 18:29:15.523106 2021] [proxy:debug] [pid 26536] proxy_util.c(2985): AH00962: HTTP: connection complete to 127.0.0.1:24816 (127.0.0.1)
[Wed Apr 14 18:29:15.524836 2021] [proxy_http:trace3] [pid 26536] mod_proxy_http.c(1414): [client 192.168.121.127:48250] Status from backend: 404
[Wed Apr 14 18:29:15.524868 2021] [proxy_http:trace4] [pid 26536] mod_proxy_http.c(1089): [client 192.168.121.127:48250] Headers received from backend:
[Wed Apr 14 18:29:15.524874 2021] [proxy_http:trace4] [pid 26536] mod_proxy_http.c(1091): [client 192.168.121.127:48250] Content-Type: text/plain; charset=utf-8
[Wed Apr 14 18:29:15.524889 2021] [proxy_http:trace4] [pid 26536] mod_proxy_http.c(1091): [client 192.168.121.127:48250] Content-Length: 14
[Wed Apr 14 18:29:15.524932 2021] [proxy_http:trace4] [pid 26536] mod_proxy_http.c(1091): [client 192.168.121.127:48250] Date: Wed, 14 Apr 2021 18:29:15 GMT
[Wed Apr 14 18:29:15.524962 2021] [proxy_http:trace4] [pid 26536] mod_proxy_http.c(1091): [client 192.168.121.127:48250] Server: Python/3.6 aiohttp/3.7.4
[Wed Apr 14 18:29:15.524978 2021] [proxy_http:trace3] [pid 26536] mod_proxy_http.c(1682): [client 192.168.121.127:48250] start body send
Updated by fao89 over 3 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to fao89
Updated by fao89 over 3 years ago
I did some tests and ngnix works differently than apache, apache requires setting the scheme while nginx not, the solution is changing apache snippets on plugins https://github.com/fao89/pulp_installer/runs/2355894514?check_suite_focus=true
Updated by pulpbot over 3 years ago
- Status changed from ASSIGNED to POST
Actions
.