Issue #1736
closedNectar fails to handle redirects to FTP gracefully.
Description
When we sync a repository lazily we make entries for each file which include a URL, like this one:
> db.lazy_content_catalog.find({"path": "/var/lib/pulp/content/units/distribution/de/d64602f1a2165af46f6cd4c8c2645a0422607173fb34617458c9f419e2c7bf/images/boot.iso"}).pretty()
{
"_id" : ObjectId("56d727f4e7798940cda81c07"),
"_ns" : "lazy_content_catalog",
"path" : "/var/lib/pulp/content/units/distribution/de/d64602f1a2165af46f6cd4c8c2645a0422607173fb34617458c9f419e2c7bf/images/boot.iso",
"importer_id" : "56d7269ae779893fb9656f55",
"unit_id" : "8335ae60-c6cc-45c3-8c49-2391d533038d",
"unit_type_id" : "distribution",
"url" : "http://ftp.linux.ncsu.edu/pub/CentOS/7/os/x86_64/images/boot.iso",
"revision" : 1,
"data" : {
}
}
>
Sadly, some repositories redirect HTTP requests to FTP:
[vagrant@dev ~]$ curl -v -L -O http://ftp.linux.ncsu.edu/pub/CentOS/7/os/x86_64/images/boot.iso
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 152.1.2.172...
* Connected to ftp.linux.ncsu.edu (152.1.2.172) port 80 (#0)
> GET /pub/CentOS/7/os/x86_64/images/boot.iso HTTP/1.1
> Host: ftp.linux.ncsu.edu
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Date: Wed, 02 Mar 2016 19:20:09 GMT
< Server: Apache/2.2.15 (Red Hat)
< Location: ftp://ftp.linux.ncsu.edu//pub/CentOS/7/os/x86_64/images/boot.iso
< Content-Length: 357
< Content-Type: text/html; charset=iso-8859-1
<
* Ignoring the response-body
{ [357 bytes data]
100 357 100 357 0 0 2752 0 --:--:-- --:--:-- --:--:-- 2767
* Connection #0 to host ftp.linux.ncsu.edu left intact
* Issue another request to this URL: 'ftp://ftp.linux.ncsu.edu//pub/CentOS/7/os/x86_64/images/boot.iso'
* Found bundle for host ftp.linux.ncsu.edu: 0x557ee40cc8f0
* Trying 152.1.2.172...
* Connected to ftp.linux.ncsu.edu (152.1.2.172) port 21 (#1)
< 220 Welcome to NC State Red Hat Linux and Realm Linux mirror.
> USER anonymous
< 331 Please specify the password.
> PASS ftp@example.com
< 230 Login successful.
> PWD
< 257 "/"
* Entry path is '/'
> CWD /
* ftp_perform ends with SECONDARY: 0
< 250 Directory successfully changed.
> CWD pub
< 250 Directory successfully changed.
> CWD CentOS
< 250 Directory successfully changed.
> CWD 7
< 250 Directory successfully changed.
> CWD os
< 250 Directory successfully changed.
> CWD x86_64
< 250 Directory successfully changed.
> CWD images
< 250 Directory successfully changed.
> EPSV
* Connect data stream passively
< 229 Entering Extended Passive Mode (|||46329|).
* Trying 152.1.2.172...
* Connecting to 152.1.2.172 (152.1.2.172) port 46329
* Connected to ftp.linux.ncsu.edu (152.1.2.172) port 21 (#1)
> TYPE I
< 200 Switching to Binary mode.
> SIZE boot.iso
< 213 394264576
> RETR boot.iso
< 150 Opening BINARY mode data connection for boot.iso (394264576 bytes).
* Maxdownload = -1
* Getting file with size: 394264576
{ [14480 bytes data]
This is not handled gracefully by Nectar:
nectar.downloaders.threaded:ERROR: (16301-86976) No connection adapters were found for 'ftp://ftp.linux.ncsu.edu//pub/CentOS/7/os/x86_64/images/boot.iso'
nectar.downloaders.threaded:ERROR: (16301-86976) Traceback (most recent call last):
nectar.downloaders.threaded:ERROR: (16301-86976) File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 229, in _fetch
nectar.downloaders.threaded:ERROR: (16301-86976) self.config.read_timeout))
nectar.downloaders.threaded:ERROR: (16301-86976) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 480, in get
nectar.downloaders.threaded:ERROR: (16301-86976) return self.request('GET', url, **kwargs)
nectar.downloaders.threaded:ERROR: (16301-86976) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
nectar.downloaders.threaded:ERROR: (16301-86976) resp = self.send(prep, **send_kwargs)
nectar.downloaders.threaded:ERROR: (16301-86976) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 597, in send
nectar.downloaders.threaded:ERROR: (16301-86976) history = [resp for resp in gen] if allow_redirects else []
nectar.downloaders.threaded:ERROR: (16301-86976) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 195, in resolve_redirects
nectar.downloaders.threaded:ERROR: (16301-86976) **adapter_kwargs
nectar.downloaders.threaded:ERROR: (16301-86976) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 570, in send
nectar.downloaders.threaded:ERROR: (16301-86976) adapter = self.get_adapter(url=request.url)
nectar.downloaders.threaded:ERROR: (16301-86976) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 644, in get_adapter
nectar.downloaders.threaded:ERROR: (16301-86976) raise InvalidSchema("No connection adapters were found for '%%s'" %% url)
nectar.downloaders.threaded:ERROR: (16301-86976) InvalidSchema: No connection adapters were found for 'ftp://ftp.linux.ncsu.edu//pub/CentOS/7/os/x86_64/images/boot.iso'
[-] "127.0.0.1" - - [02/Mar/2016:18:16:54 +0000] "GET /var/lib/pulp/content/units/distribution/de/d64602f1a2165af46f6cd4c8c2645a0422607173fb34617458c9f419e2c7bf/images/boot.iso HTTP/1.1" 200 - "-" "python-requests"
Related issues
Updated by mhrivnak over 8 years ago
I don't think this is specific to redirects. Neither nectar nor python-requests supports ftp at all.
Updated by jcline@redhat.com over 8 years ago
Well no, it's not specific to redirects, but it is what occurred when I encountered the problem.
It may well be the case that neither Pulp nor Nectar ever checks URLs to make sure they are only HTTP and HTTPS, but even if they did do that, it wouldn't be enough. That's really what I wanted to capture in my bug report. Perhaps I could have made it more clear.
Updated by mhrivnak over 8 years ago
- Priority changed from Normal to High
- Severity changed from 2. Medium to 3. High
- Triaged changed from No to Yes
Updated by mhrivnak over 8 years ago
- Related to Issue #1170: rpm importer does not support switching protocols from http to ftp mid download using a redirect- fails downloading distribution files added
Updated by semyers over 8 years ago
We talked about this in a triage meeting today, and came up with some more clarification:
When nectar/requests gets a redirect to FTP, it explodes and breaks the sync. This is bad, thus High priority and severity. This issue is about preventing that error from breaking a sync. A related issue, #1170, exists to handle potentially adding FTP support to nectar/requests, and therefore to rpm sync.
Updated by pthomas@redhat.com over 7 years ago
- Related to Issue #2474: CentOs repo that was synced in 2.8 results in "Task Failed Importer indicated a failed response" after upgrade to 2.11 added
Updated by bmbouter over 7 years ago
- Related to deleted (Issue #2474: CentOs repo that was synced in 2.8 results in "Task Failed Importer indicated a failed response" after upgrade to 2.11)
Updated by bmbouter over 7 years ago
- Has duplicate Issue #2474: CentOs repo that was synced in 2.8 results in "Task Failed Importer indicated a failed response" after upgrade to 2.11 added
Updated by bmbouter over 5 years ago
- Status changed from NEW to CLOSED - WONTFIX
Updated by bmbouter over 5 years ago
Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.