ftplib: FTP_TLS seems to have problems with sites that close the encrypted channel themselfes

[jottbe]

BPO	33122
Nosy	@giampaolo
Files	ftplib.py: patched ftplib

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-03-22.14:29:38.491>
labels = ['type-bug', 'library']
title = 'ftplib: FTP_TLS seems to have problems with sites that close the encrypted channel themselfes'
updated_at = <Date 2018-03-26.09:42:57.908>
user = 'https://bugs.python.org/jottbe'

bugs.python.org fields:

activity = <Date 2018-03-26.09:42:57.908>
actor = 'jottbe'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-03-22.14:29:38.491>
creator = 'jottbe'
dependencies = []
files = ['47500']
hgrepos = []
issue_num = 33122
keywords = []
message_count = 3.0
messages = ['314261', '314327', '314444']
nosy_count = 2.0
nosy_names = ['giampaolo.rodola', 'jottbe']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33122'
versions = ['Python 3.6']

[jottbe]

Hi,

I'm not quite sure, if you would actually call this a bug, but it is very molesting at least ;o)

I use ftplib.FTP_TLS to connect to a z/OS ftp server. With a minor change it works very well (happy to have found this library).
The problem I have is, that without any change, an exception is raised after every single command I invoke, even though the server sends back an ok message.

The exception is an OSError which is raised while executing conn.unwrap(). It seems the connection is already closed when this is called and thus an exception is raised. But handling this exception outside the FTP_TLS-class makes no sense, because then every command would raise an exception and the "good" exceptions could not be distinguised from the ones that are really searious so easily anymore (I mean: if i get an exception that a connection could not be closed, because someone else closed it before, that's not very serious, is it?).

Suggestions to solve this:
small solution: allow the programmer to decide what to do, by creating subclasses
This is "factor-out" the unwrap logic in a separate method or function, so at least users of the class can overwrite the behavior, without having to rebuild the whole logic of the affected methods.

In my quick solution I created a new method in class FTP:
    def __handleAutoCloseSSL__(self, conn):
        if self.autoCloseModeSSL == 'NONE' or self.autoCloseModeSSL is None or _SSLSocket is None or not isinstance(conn, _SSLSocket):
            # do nothing
            pass
        elif self.autoCloseModeSSL in ('SAFE', 'HIDE'):
            try:
                conn.unwrap()
            except OSError as ex:
                if self.autoCloseModeSSL != 'HIDE':
                    print('Caught exception %s while calling conn.unwrap()' % str(ex))
        else:
            # Standard mode (usally self.autoCloseModeSSL =='STANDARD' but anything else is accepted as well)
            # the original code was:
            #if _SSLSocket is not None and isinstance(conn, _SSLSocket):
            #    conn.unwrap()
            conn.unwrap()

And the class variable:
autoCloseModeSSL = 'STANDARD'

Then I called it from methods (instead of doing conn.unwrap() there directly):
retbinary
retlines
storbinary
storlines

Ok, maybe not that sexy, but it works :o)
And if you don't like the hack with instance variable autoCloseModeSSL, you could just transfer the original conn.unwrap() in an extra method which could then be overwritten by programmers in subclasses. This would already help me very much, because I know that patching a library is not a good idea. Even more if it is a communication library that might be updated from time to time.

[giampaolo]

Please paste your code and traceback message. Also what's the remote FTP server you're connected to? You should be able to see it in the welcome message (you can set FTP_TLS.debugging to True).

[jottbe]

Hi, thanks for tanking care of this issue.
I am mainly working on a windows client and connect to a z/OS host.
Attached you find the ftplib.py I patched to workaround this.

Here is the output of the list command which ends up in an exception (this time from a unix machine, where I still have found the unpatched version of ftplib.py):

>>> ftplib.FTP_TLS.debugging=True
>>> conn=ftplib.FTP_TLS(host=url, user=user, passwd=passw)
*resp* '220-TCPFT000 IBM FTP CS V2R2 at tcpip06, 11:38:07 on 2018-03-26.\n220 Connection will close if idle for more than 5 minutes.'
*cmd* 'AUTH TLS'
*resp* '234 Security environment established - ready for negotiation'
*cmd* 'USER SBxxxxx'
*resp* '331 Send password please.'
*cmd* 'PASS ********'
*resp* '230 SBxxxxx is logged on.  Working directory is "SBxxxxx.".'
>>> conn.prot_p()
*cmd* 'PBSZ 0'
*resp* '200 Protection buffer size accepted'
*cmd* 'PROT P'
*resp* '200 Data connection protection set to private'
'200 Data connection protection set to private'
>>> conn.retrlines("LIST 'SBxxxxx.SBxxxxx.*'")
*cmd* 'TYPE A'
*resp* '200 Representation type is Ascii NonPrint'
*cmd* 'PASV'
*resp* '227 Entering Passive Mode (53,113,100,193,250,60)'
*cmd* "LIST 'SBxxxxx.SBxxxxx.*'"
*resp* '125 List started OK'
Volume Unit    Referred Ext Used Recfm Lrecl BlkSz Dsorg Dsname
IDV101 3390   2018/03/21 16   68  VB    4092  4096  PS  'SBxxxxx.SBxxxxx.SPUFI.OUT'
Migrated                                                'SBxxxxx.SBxxxxx.SPUFI.SOAOUT'
IDV10T 3390   2018/03/08 13   62  VB    4092  4096  PS  'SBxxxxx.SBxxxxx.SPUFI.WHC'
Migrated                                                'SBxxxxx.SBxxxxx.VI870V'
Migrated                                                'SBxxxxx.SBxxxxx.VI871V'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.4/ftplib.py", line 484, in retrlines
    conn.unwrap()
  File "/usr/lib/python3.4/ssl.py", line 811, in unwrap
    s = self._sslobj.shutdown()
OSError: [Errno 0] Error