Python: concurrent.futures

I had been wanting to try some new features of Python 3 for a while now. I recently found some free time and tried concurrent.futures. Concurrent.futures allows us to write code that runs in parallel in Python without resorting to creating threads or forking processes.

We will use the following problem to write some code — write a script that downloads images over the internet to a local directory.

Our test setting will use the built-in http server that comes bundled with Python. This will make sure that the numbers we see are not influenced by network latency and we also don’t bombard some random server with 1000's of requests.

Image for post
Image for post

We will run the script on my Intel 4 Core laptop running Ubuntu 14.04 with Python 3.4.

Image for post
Image for post

The first step is to establish some base benchmark to see how much time a sequential program would require. This is what the script looks like -

The times for this simple sequential download of images are -

Image for post
Image for post

Note that we have added a sanity check to verify that there is a linear increase in the time taken. So if 10k requests takes 80 seconds, 20k requests will require 160 seconds and so on.

Now let us look at a version that uses concurrent.futures. It is surprisingly easy to change the sequential code to run parallelly -

Image for post
Image for post

Only the download_ntimes method has changed to download images asynchronously using ProcessPoolExecutor. From the documentation -

The ProcessPoolExecutor class uses a pool of processes to execute calls asynchronously. ProcessPoolExecutor uses the multiprocessing module, which allows it to side-step the Global Interpreter Lock.

Note also that we don’t specify how many workers to use. That is because, if max_workers is None or not given, it will default to the number of processors on the machine. The download times now are -

Image for post
Image for post

We see that the download times are down by a factor of 3 (approximately). Theoretically, we should have seen a 4x speed up because of the 4 cores my machine has. The difference is probably due to book keeping that concurrent.futures has to do.

So the question is — how many of us have the luxury of using Python 3 in real life ;). But on the other hand, if you are writing standalone scripts today and not using Python 3, you are missing out on the fun!

Backend engineer interested in distributed systems and programming languages — https://caulagi.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store