Project

Profile

Help

Issue #1736

Nectar fails to handle redirects to FTP gracefully.

Added by jcline@redhat.com over 4 years ago. Updated over 1 year ago.

Status:
CLOSED - WONTFIX
Priority:
High
Assignee:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version - Nectar:
Platform Release:
Target Release - Nectar:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

When we sync a repository lazily we make entries for each file which include a URL, like this one:

> db.lazy_content_catalog.find({"path": "/var/lib/pulp/content/units/distribution/de/d64602f1a2165af46f6cd4c8c2645a0422607173fb34617458c9f419e2c7bf/images/boot.iso"}).pretty()
{
        "_id" : ObjectId("56d727f4e7798940cda81c07"),
        "_ns" : "lazy_content_catalog",
        "path" : "/var/lib/pulp/content/units/distribution/de/d64602f1a2165af46f6cd4c8c2645a0422607173fb34617458c9f419e2c7bf/images/boot.iso",
        "importer_id" : "56d7269ae779893fb9656f55",
        "unit_id" : "8335ae60-c6cc-45c3-8c49-2391d533038d",
        "unit_type_id" : "distribution",
        "url" : "http://ftp.linux.ncsu.edu/pub/CentOS/7/os/x86_64/images/boot.iso",
        "revision" : 1,
        "data" : {

        }
}
>

Sadly, some repositories redirect HTTP requests to FTP:

[vagrant@dev ~]$ curl -v -L -O http://ftp.linux.ncsu.edu/pub/CentOS/7/os/x86_64/images/boot.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 152.1.2.172...
* Connected to ftp.linux.ncsu.edu (152.1.2.172) port 80 (#0)
> GET /pub/CentOS/7/os/x86_64/images/boot.iso HTTP/1.1
> Host: ftp.linux.ncsu.edu
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Date: Wed, 02 Mar 2016 19:20:09 GMT
< Server: Apache/2.2.15 (Red Hat)
< Location: ftp://ftp.linux.ncsu.edu//pub/CentOS/7/os/x86_64/images/boot.iso
< Content-Length: 357
< Content-Type: text/html; charset=iso-8859-1
< 
* Ignoring the response-body
{ [357 bytes data]
100   357  100   357    0     0   2752      0 --:--:-- --:--:-- --:--:--  2767
* Connection #0 to host ftp.linux.ncsu.edu left intact
* Issue another request to this URL: 'ftp://ftp.linux.ncsu.edu//pub/CentOS/7/os/x86_64/images/boot.iso'
* Found bundle for host ftp.linux.ncsu.edu: 0x557ee40cc8f0
*   Trying 152.1.2.172...
* Connected to ftp.linux.ncsu.edu (152.1.2.172) port 21 (#1)
< 220 Welcome to NC State Red Hat Linux and Realm Linux mirror.
> USER anonymous
< 331 Please specify the password.
> PASS ftp@example.com
< 230 Login successful.
> PWD
< 257 "/"
* Entry path is '/'
> CWD /
* ftp_perform ends with SECONDARY: 0
< 250 Directory successfully changed.
> CWD pub
< 250 Directory successfully changed.
> CWD CentOS
< 250 Directory successfully changed.
> CWD 7
< 250 Directory successfully changed.
> CWD os
< 250 Directory successfully changed.
> CWD x86_64
< 250 Directory successfully changed.
> CWD images
< 250 Directory successfully changed.
> EPSV
* Connect data stream passively
< 229 Entering Extended Passive Mode (|||46329|).
*   Trying 152.1.2.172...
* Connecting to 152.1.2.172 (152.1.2.172) port 46329
* Connected to ftp.linux.ncsu.edu (152.1.2.172) port 21 (#1)
> TYPE I
< 200 Switching to Binary mode.
> SIZE boot.iso
< 213 394264576
> RETR boot.iso
< 150 Opening BINARY mode data connection for boot.iso (394264576 bytes).
* Maxdownload = -1
* Getting file with size: 394264576
{ [14480 bytes data]

This is not handled gracefully by Nectar:

nectar.downloaders.threaded:ERROR: (16301-86976) No connection adapters were found for 'ftp://ftp.linux.ncsu.edu//pub/CentOS/7/os/x86_64/images/boot.iso'
nectar.downloaders.threaded:ERROR: (16301-86976) Traceback (most recent call last):
nectar.downloaders.threaded:ERROR: (16301-86976)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 229, in _fetch
nectar.downloaders.threaded:ERROR: (16301-86976)     self.config.read_timeout))
nectar.downloaders.threaded:ERROR: (16301-86976)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 480, in get
nectar.downloaders.threaded:ERROR: (16301-86976)     return self.request('GET', url, **kwargs)
nectar.downloaders.threaded:ERROR: (16301-86976)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
nectar.downloaders.threaded:ERROR: (16301-86976)     resp = self.send(prep, **send_kwargs)
nectar.downloaders.threaded:ERROR: (16301-86976)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 597, in send
nectar.downloaders.threaded:ERROR: (16301-86976)     history = [resp for resp in gen] if allow_redirects else []
nectar.downloaders.threaded:ERROR: (16301-86976)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 195, in resolve_redirects
nectar.downloaders.threaded:ERROR: (16301-86976)     **adapter_kwargs
nectar.downloaders.threaded:ERROR: (16301-86976)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 570, in send
nectar.downloaders.threaded:ERROR: (16301-86976)     adapter = self.get_adapter(url=request.url)
nectar.downloaders.threaded:ERROR: (16301-86976)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 644, in get_adapter
nectar.downloaders.threaded:ERROR: (16301-86976)     raise InvalidSchema("No connection adapters were found for '%%s'" %% url)
nectar.downloaders.threaded:ERROR: (16301-86976) InvalidSchema: No connection adapters were found for 'ftp://ftp.linux.ncsu.edu//pub/CentOS/7/os/x86_64/images/boot.iso'
[-] "127.0.0.1" - - [02/Mar/2016:18:16:54 +0000] "GET /var/lib/pulp/content/units/distribution/de/d64602f1a2165af46f6cd4c8c2645a0422607173fb34617458c9f419e2c7bf/images/boot.iso HTTP/1.1" 200 - "-" "python-requests"

Related issues

Related to Pulp - Issue #1170: rpm importer does not support switching protocols from http to ftp mid download using a redirect- fails downloading distribution filesCLOSED - WONTFIX<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Has duplicate Pulp - Issue #2474: CentOs repo that was synced in 2.8 results in "Task Failed Importer indicated a failed response" after upgrade to 2.11CLOSED - DUPLICATE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

History

#1 Updated by mhrivnak over 4 years ago

I don't think this is specific to redirects. Neither nectar nor python-requests supports ftp at all.

#2 Updated by jcline@redhat.com over 4 years ago

Well no, it's not specific to redirects, but it is what occurred when I encountered the problem.

It may well be the case that neither Pulp nor Nectar ever checks URLs to make sure they are only HTTP and HTTPS, but even if they did do that, it wouldn't be enough. That's really what I wanted to capture in my bug report. Perhaps I could have made it more clear.

#3 Updated by mhrivnak over 4 years ago

  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High
  • Triaged changed from No to Yes

#4 Updated by mhrivnak over 4 years ago

  • Related to Issue #1170: rpm importer does not support switching protocols from http to ftp mid download using a redirect- fails downloading distribution files added

#5 Updated by semyers over 4 years ago

We talked about this in a triage meeting today, and came up with some more clarification:
When nectar/requests gets a redirect to FTP, it explodes and breaks the sync. This is bad, thus High priority and severity. This issue is about preventing that error from breaking a sync. A related issue, #1170, exists to handle potentially adding FTP support to nectar/requests, and therefore to rpm sync.

#6 Updated by pthomas@redhat.com almost 4 years ago

  • Related to Issue #2474: CentOs repo that was synced in 2.8 results in "Task Failed Importer indicated a failed response" after upgrade to 2.11 added

#7 Updated by bmbouter almost 4 years ago

  • Related to deleted (Issue #2474: CentOs repo that was synced in 2.8 results in "Task Failed Importer indicated a failed response" after upgrade to 2.11)

#8 Updated by bmbouter almost 4 years ago

  • Has duplicate Issue #2474: CentOs repo that was synced in 2.8 results in "Task Failed Importer indicated a failed response" after upgrade to 2.11 added

#9 Updated by bmbouter over 1 year ago

  • Status changed from NEW to CLOSED - WONTFIX

#10 Updated by bmbouter over 1 year ago

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

#11 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF