Nectar

When we sync a repository lazily we make entries for each file which include a URL, like this one:

> db.lazy_content_catalog.find({"path": "/var/lib/pulp/content/units/distribution/de/d64602f1a2165af46f6cd4c8c2645a0422607173fb34617458c9f419e2c7bf/images/boot.iso"}).pretty()
{
        "_id" : ObjectId("56d727f4e7798940cda81c07"),
        "_ns" : "lazy_content_catalog",
        "path" : "/var/lib/pulp/content/units/distribution/de/d64602f1a2165af46f6cd4c8c2645a0422607173fb34617458c9f419e2c7bf/images/boot.iso",
        "importer_id" : "56d7269ae779893fb9656f55",
        "unit_id" : "8335ae60-c6cc-45c3-8c49-2391d533038d",
        "unit_type_id" : "distribution",
        "url" : "http://ftp.linux.ncsu.edu/pub/CentOS/7/os/x86_64/images/boot.iso",
        "revision" : 1,
        "data" : {

        }
}
>

Sadly, some repositories redirect HTTP requests to FTP:

[vagrant@dev ~]$ curl -v -L -O http://ftp.linux.ncsu.edu/pub/CentOS/7/os/x86_64/images/boot.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 152.1.2.172...
* Connected to ftp.linux.ncsu.edu (152.1.2.172) port 80 (#0)
> GET /pub/CentOS/7/os/x86_64/images/boot.iso HTTP/1.1
> Host: ftp.linux.ncsu.edu
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Date: Wed, 02 Mar 2016 19:20:09 GMT
< Server: Apache/2.2.15 (Red Hat)
< Location: ftp://ftp.linux.ncsu.edu//pub/CentOS/7/os/x86_64/images/boot.iso
< Content-Length: 357
< Content-Type: text/html; charset=iso-8859-1
< 
* Ignoring the response-body
{ [357 bytes data]
100   357  100   357    0     0   2752      0 --:--:-- --:--:-- --:--:--  2767
* Connection #0 to host ftp.linux.ncsu.edu left intact
* Issue another request to this URL: 'ftp://ftp.linux.ncsu.edu//pub/CentOS/7/os/x86_64/images/boot.iso'
* Found bundle for host ftp.linux.ncsu.edu: 0x557ee40cc8f0
*   Trying 152.1.2.172...
* Connected to ftp.linux.ncsu.edu (152.1.2.172) port 21 (#1)
< 220 Welcome to NC State Red Hat Linux and Realm Linux mirror.
> USER anonymous
< 331 Please specify the password.
> PASS ftp@example.com
< 230 Login successful.
> PWD
< 257 "/"
* Entry path is '/'
> CWD /
* ftp_perform ends with SECONDARY: 0
< 250 Directory successfully changed.
> CWD pub
< 250 Directory successfully changed.
> CWD CentOS
< 250 Directory successfully changed.
> CWD 7
< 250 Directory successfully changed.
> CWD os
< 250 Directory successfully changed.
> CWD x86_64
< 250 Directory successfully changed.
> CWD images
< 250 Directory successfully changed.
> EPSV
* Connect data stream passively
< 229 Entering Extended Passive Mode (|||46329|).
*   Trying 152.1.2.172...
* Connecting to 152.1.2.172 (152.1.2.172) port 46329
* Connected to ftp.linux.ncsu.edu (152.1.2.172) port 21 (#1)
> TYPE I
< 200 Switching to Binary mode.
> SIZE boot.iso
< 213 394264576
> RETR boot.iso
< 150 Opening BINARY mode data connection for boot.iso (394264576 bytes).
* Maxdownload = -1
* Getting file with size: 394264576
{ [14480 bytes data]

This is not handled gracefully by Nectar:

nectar.downloaders.threaded:ERROR: (16301-86976) No connection adapters were found for 'ftp://ftp.linux.ncsu.edu//pub/CentOS/7/os/x86_64/images/boot.iso'
nectar.downloaders.threaded:ERROR: (16301-86976) Traceback (most recent call last):
nectar.downloaders.threaded:ERROR: (16301-86976)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 229, in _fetch
nectar.downloaders.threaded:ERROR: (16301-86976)     self.config.read_timeout))
nectar.downloaders.threaded:ERROR: (16301-86976)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 480, in get
nectar.downloaders.threaded:ERROR: (16301-86976)     return self.request('GET', url, **kwargs)
nectar.downloaders.threaded:ERROR: (16301-86976)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
nectar.downloaders.threaded:ERROR: (16301-86976)     resp = self.send(prep, **send_kwargs)
nectar.downloaders.threaded:ERROR: (16301-86976)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 597, in send
nectar.downloaders.threaded:ERROR: (16301-86976)     history = [resp for resp in gen] if allow_redirects else []
nectar.downloaders.threaded:ERROR: (16301-86976)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 195, in resolve_redirects
nectar.downloaders.threaded:ERROR: (16301-86976)     **adapter_kwargs
nectar.downloaders.threaded:ERROR: (16301-86976)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 570, in send
nectar.downloaders.threaded:ERROR: (16301-86976)     adapter = self.get_adapter(url=request.url)
nectar.downloaders.threaded:ERROR: (16301-86976)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 644, in get_adapter
nectar.downloaders.threaded:ERROR: (16301-86976)     raise InvalidSchema("No connection adapters were found for '%%s'" %% url)
nectar.downloaders.threaded:ERROR: (16301-86976) InvalidSchema: No connection adapters were found for 'ftp://ftp.linux.ncsu.edu//pub/CentOS/7/os/x86_64/images/boot.iso'
[-] "127.0.0.1" - - [02/Mar/2016:18:16:54 +0000] "GET /var/lib/pulp/content/units/distribution/de/d64602f1a2165af46f6cd4c8c2645a0422607173fb34617458c9f419e2c7bf/images/boot.iso HTTP/1.1" 200 - "-" "python-requests"