Author: Stefan Schwarzer <sschwarzer@sschwarzer.net>
Date: 2020-06-13

# What's new in ftputil 4.0.0?

Version: | 4.0.0
Date: | 2020-06-13
Author: | Stefan Schwarzer <sschwarzer@sschwarzer.net>

Contents

- Supported Python versions
- Path-like objects
- Time shift handling
  - Porting
- ftputil no longer uses the `-a` option by default
  - Porting
- `makedirs` behaves as in Python 3
  - Porting
- Questions and answers
  - Why is Python 2 no longer supported?
  - Why is the minimum version Python 3.6?
  - Why a new time shift definition?

# Supported Python versions

Support for Python 2 is dropped. The minimum required Python 3 version is 3.6.

Find more details in Questions and answers.

# Path-like objects

Methods that take directory and file names as `str` or `bytes` objects now also accept path-like objects.

# Time shift handling

ftputil uses the notion of "time shift" to deal with time zone differences between client and server. This is important for the methods `upload_if_newer` and `download_if_newer`.

The defintion of "time shift" changed from earlier ftputil versions to ftputil 4.0.0.

Previously, the time shift was defined as

> time_used_by_server - local_time_used_by_client

The new definition is

> time_used_by_server - UTC

Both definitions have their pros and cons (detailed in the Questions and answers).

## Porting

If you don't use any methods that deal with time stamps from the FTP server, you can ignore the time shift redefinition. The affected methods are `upload_if_newer`, `download_if_newer` and using the timestamp values from the `stat` and `lstat` methods.

Ideally, you have write access on the server in the current directory. In this case you can call `synchronize_times`:

    with ftputil.FTPHost(host, user, password) as ftp_host:
        ftp_host.synchronize_times()
        ...

If using `synchronize_times` isn't an option, you have to set the time shift explicitly with `set_time_shift`:

    with ftputil.FTPHost(host, user, password) as ftp_host:
        ftp_host.set_time_shift(new_time_shift)
        ...

If you're sure the server in its directory listings uses the same timezone as the client, you can use

    with ftputil.FTPHost(host, user, password) as ftp_host:
        ftp_host.set_time_shift(
          round((datetime.datetime.now() - datetime.datetime.utcnow()).seconds, -2)
        )
        ...

This is roughly equivalent to the old ftputil behavior. The only difference is that the new behavior requires that you adapt the time shift value if there's a switch to or from daylight saving time.

If you want to cover such daylight saving time switches as well, you could override `time_shift()` to return the above value. This will automatically adapt to daylight saving time switches because `datetime.datetime.now()` will return the time for the changed UTC offset. That said, I think this approach is a hack because it means that calls to `set_time_shift` will effectively be ignored. So don't take this approach as a recommendation.

You can use this script to scan your code for uses of `time_shift()` or `set_time_shift`. However, if you implicitly relied on the old default behavior (time shift is 0.0 if client and server use the same time zone), you'll need additional calls for `set_time_shift`.

# ftputil no longer uses the `-a` option by default

Earlier ftputil versions by default sent an `-a` option with the FTP `DIR` command to include "hidden" directories and files (names starting with a dot) in the listing.

That led to problems when the server didn't understand the option and treated it as a directory or file name.

Therefore, ftputil no longer uses the `-a` option by default.

## Porting

You can enable the old behavior by setting `use_list_a_option` on the `FTPHost` instance to `True`:

    with ftputil.FTPHost(host, user, password) as ftp_host:
        ftp_host.use_list_a_option = True
        ...

However, do this only if you're sure the server interprets the option correctly!

# `makedirs` behaves as in Python 3

In Python 2, `os.makedirs` didn't complain if any directory in the path to create already existed. Since ftputil was originally based on Python 2 behavior, this was also the behavior in `FTPHost.makedirs`.

Python 3 added an optional argument `exist_ok` with the default `False`. With this default, `os.makedirs` raises an exception if any directory but the last in the `path` argument exists.

Since Python 2 is used less and less, ftputil 4.0.0 follows the Python 3 semantics.

## Porting

If you want the old behavior of `FTPHost.makedirs`, pass `exist_ok=True`. Note that there's also an unused `mode` argument for consistency with the `os.makedirs` API, so make sure you pass `exist_ok` as a keyword argument.

You can use this script to scan your code for uses of `makedirs()`.

# Questions and answers

## Why is Python 2 no longer supported?

Since the start of the year Python 2 officially is no longer maintained and supporting it in combination with Python 3 led to lots of extra work. Therefore, I decided to drop support for Python 2.

## Why is the minimum version Python 3.6?

Python 3.4 and older versions are no longer supported by the CPython team.

My plan was to support Python 3.5 since it's not yet end-of-life'd and it may still be used in some LTS Linux/Unix distributions. However, dropping Python 3.5 support made it much easier to implement ticket #119, support for path-like objects. Python 3.6 introduced some infrastructure so that code that used to use only `str` and `bytes` paths can now use path-like objects as well. I considered it more important to support path-like objects than Python 3.5. I guess it might have been possible to add support for path-like objects on top of Python 3.5, but it would have been a hassle and Python 3.5 support officially ends in just a few months.

## Why a new time shift definition?

Both the old and the new approach have their pros and cons:

Regarding the old approach:

- Pro: If the server uses the time zone of the client in directory listings, the default time shift of 0.0 will do the right thing.

- Con: Since the time shift depends on two time zones, it's more difficult to reason about the (ftputil) code and write correct code. In the past, there have been multiple bugs in the time zone / time shift handling in ftputil, some of them occuring only under "interesting" conditions (around daylight saving time changes or year changes).

  Actually, I decided to implement the new approach when I ran into a bug around the last new year change.

Regarding the new approach:

- Pro: The new behavior works better when the server is set to UTC. This is a sane setting because it avoids that an hour interval is used twice when there's a switch from daylight saving time to "normal" time. There are even more subtle problems with daylight saving time switches.
- Pro: Since the time shift calculation depends on only one time zone (that of the server), the code in ftputil is easier to reason about and now hopefully more robust.
- Con: The new approach is backward-incompatible, so users may have to adapt their code.
- Con: If the server uses the time zone of the client in directory listings, the time shift must be adjusted whenever there's a switch to or from daylight saving time.

Neither of the two approaches is foolproof. For example, timestamps that are older than the last daylight saving time switch may be calculated wrongly because they may use a different time zone than the one currently set by `set_time_shift`.