How to Download Files From URLs With Python

Using urllib From the Standard Library

Python ships with a package called urllib, which provides a convenient way to interact with web resources. It has a straightforward and user-friendly interface, making it suitable for quick prototyping and smaller projects. With urllib, you can perform different tasks dealing with network communication, such as parsing URLs, sending HTTP requests, downloading files, and handling errors related to network operations.

As a standard library package, urllib has no external dependencies and doesn’t require installing additional packages, making it a convenient choice. For the same reason, it’s readily accessible for development and deployment. It’s also cross-platform compatible, meaning you can write and run code seamlessly using the urllib package across different operating systems without additional dependencies or configuration.

The urllib package is also very versatile. It integrates well with other modules in the Python standard library, such as re for building and manipulating regular expressions, as well as json for working with JSON data. The latter is particularly handy when you need to consume JSON APIs.

In addition, you can extend the urllib package and use it with other third-party libraries, like requests, BeautifulSoup, and Scrapy. This offers the possibility for more advanced operations in web scraping and interacting with web APIs.

To download a file from a URL using the urllib package, you can call urlretrieve() from the urllib.request module. This function fetches a web resource from the specified URL and then saves the response to a local file. To start, import urlretrieve() from urlllib.request:

>>> from urllib.request import urlretrieve

Next, define the URL that you want to retrieve data from. If you don’t specify a path to a local file where you want to save the data, then the function will create a temporary file for you. Since you know that you’ll be downloading a ZIP file from that URL, go ahead and provide an optional path to the target file:

>>> url = (
...     "https://api.worldbank.org/v2/en/indicator/"
...     "NY.GDP.MKTP.CD?downloadformat=csv"
... )
>>> filename = "gdp_by_country.zip"

Because your URL is quite long, you rely on Python’s implicit concatenation by splitting the string literal over multiple lines inside parentheses. The Python interpreter will automatically join the separate strings on different lines into a single string. You also define the location where you wish to save the file. When you only provide a filename without a path, Python will save the resulting file in your current working directory.

Then, you can download and save the file by calling urlretrieve() and passing in the URL and optionally your filename:

>>> urlretrieve(url, filename)
('gdp_by_country.zip', <http.client.HTTPMessage object at 0x7f06ee7527d0>)

The function returns a tuple of two objects: the path to your output file and an HTTP message object. When you don’t specify a custom filename, then you’ll see a path to a temporary file that might look like this: /tmp/tmps7qjl1tj. The HTTPMessage object represents the HTTP headers returned by the server for the request, which can contain information like content type, content length, and other metadata.

You can unpack the tuple into the individual variables using an assignment statement and iterate over the headers as though they were a Python dictionary:

>>> path, headers = urlretrieve(url, filename)
>>> for name, value in headers.items():
...     print(name, value)
...
Date Wed, 28 Jun 2023 11:26:18 GMT
Content-Type application/zip
Content-Length 128310
Connection close
Set-Cookie api_https.cookieCORS=76a6c6567ab12cea5dac4942d8df71cc; Path=/; SameSite=None; Secure
Set-Cookie api_https.cookie=76a6c6567ab12cea5dac4942d8df71cc; Path=/
Cache-Control public, must-revalidate, max-age=1
Expires Wed, 28 Jun 2023 11:26:19 GMT
Last-Modified Wed, 28 Jun 2023 11:26:18 GMT
Content-Disposition attachment; filename=API_NY.GDP.MKTP.CD_DS2_en_csv_v2_5551501.zip
Request-Context appId=cid-v1:da002513-bd8b-4441-9f30-737944134422

This information might be helpful when you’re unsure about which file format you’ve just downloaded and how you’re supposed to interpret its content. In this case, it’s a ZIP file that’s about 128 kilobytes in size. You can also deduce the original filename, which was API_NY.GDP.MKTP.CD_DS2_en_csv_v2_5551501.zip.

Now that you’ve seen how to download a file from a URL using Python’s urllib package, it’s time to tackle the same task using a third-party library. You’ll find out which way is more convenient for you.

Using the Third-Party requests Library

While urllib is a good built-in option, there may be scenarios where you need to use third-party libraries to make more advanced HTTP requests, such as those requiring some form of authentication. The requests library is a popular, user-friendly, and Pythonic API for making HTTP requests in Python. It can handle the complexities of low-level network communication behind the curtain.

The requests library is also known for its flexibility and offers tighter control over the download process, allowing you to customize it according to your project requirements. Some examples include the ability to specify request headers, handle cookies, access data behind login-gated web pages, stream data in chunks, and more.

In addition, the library is designed to be efficient and performant by supporting various features that enhance the overall download performance. Its ability to automatically handle connection pooling and reuse optimizes network utilization and reduces overhead.

Now, you’ll look into using the requests library to download that same ZIP file with GDP by country data from the World Bank Open Data platform. To begin, install the requests library into your active virtual environment using pip:

(venv) $ python -m pip install requests

This command installs the latest release of the requests library into your virtual environment. Afterward, you can start a new Python REPL session and import the requests library:

>>> import requests

Before moving further, it’s worth recalling the available HTTP methods because the requests library exposes them to you through Python functions. When you make HTTP requests to web servers, you have two commonly used methods to choose from:

1. HTTP GET
2. HTTP POST

You’ll use the GET method to retrieve data by fetching a representation of the remote resource without modifying the server’s state. Therefore, you’ll commonly use it to retrieve files like images, HTML web pages, or raw data. You’ll use the GET request in later steps.

The POST method allows you to send data for the server to process or use in creating or updating a resource. In POST requests, the data is typically sent in the request body in various formats like JSON or XML, and it’s not visible in the URL. You can use POST requests for operations that modify server data, such as creating, updating, or submitting existing or new resources.

In this tutorial, you’ll only use GET requests for downloading files.

Next, define the URL of the file that you want to download. To include additional query parameters in the URL, you’ll pass in a dictionary of strings as key-value pairs:

>>> url = "https://api.worldbank.org/v2/en/indicator/NY.GDP.MKTP.CD"
>>> query_parameters = {"downloadformat": "csv"}

In the example above, you define the same URL as before but specify the downloadformat=csv parameter separately using a Python dictionary. The library will append those parameters to the URL after you pass them to requests.get() using an optional params argument:

>>> response = requests.get(url, params=query_parameters)

This makes a GET request to retrieve data from the constructed URL with optional query parameters. The function returns an HTTP response object with the server’s response to the request. If you’d like to see the constructed URL with the optional parameters included, then use the response object’s .url attribute:

>>> response.url
'https://api.worldbank.org/v2/en/indicator/NY.GDP.MKTP.CD?downloadformat=csv'

The response object provides several other convenient attributes that you can check out. For example, these two will let you determine if the request was successful and what HTTP status code the server returned:

>>> response.ok
True

>>> response.status_code
200

A status code of 200 indicates that your request has been completed successfully. Okay, but how do you access the data payload, usually in JSON format, that you’ve retrieved with the requests library? Read on to answer this question in the next section.

Using a Pool of Threads With the requests Library

When you want to make many HTTP requests to the same or different servers, you can take advantage of multithreading to decrease the overall execution time of your code.

In a multithreaded program, you typically spawn multiple threads of execution, each with its own instruction sequence that runs independently and in parallel with the other. These threads can perform different tasks or execute other parts of the program concurrently, improving performance and maximizing the use of the available CPU cores. For example, each thread can make its own HTTP request.

You’ll now try concurrently downloading three ZIP files from the World Bank Open Data platform:

1. Total population by country
2. GDP by country
3. Population density by country

In this example, you’ll use a pool of threads with the requests library to understand how to perform parallel file downloads. First, import the ThreadPoolExecutor class from the concurrent.futures module and the requests library again:

>>> from concurrent.futures import ThreadPoolExecutor
>>> import requests

Next, write a function that you’ll execute within each thread to download a single file from a given URL:

>>> def download_file(url):
...     response = requests.get(url)
...     if "content-disposition" in response.headers:
...         content_disposition = response.headers["content-disposition"]
...         filename = content_disposition.split("filename=")[1]
...     else:
...         filename = url.split("/")[-1]
...     with open(filename, mode="wb") as file:
...         file.write(response.content)
...     print(f"Downloaded file {filename}")
...

This function takes a URL as an argument, makes a GET request using the requests library, and saves the retrieved data in a local file. In this specific example, it first attempts to extract the filename from the Content-Disposition response header, which contains information on which items on the page are displayed inline versus as attachments. If that’s not available, then it gets the filename from a part of the URL.

You’ll be downloading three separate files from the same API, so create a URL template and populate a Python list with it:

>>> template_url = (
...     "https://api.worldbank.org/v2/en/indicator/"
...     "{resource}?downloadformat=csv"
... )

>>> urls = [
...     # Total population by country
...     template_url.format(resource="SP.POP.TOTL"),
...
...     # GDP by country
...     template_url.format(resource="NY.GDP.MKTP.CD"),
...
...     # Population density by country
...     template_url.format(resource="EN.POP.DNST"),
... ]

Here, you call the .format() method on the template string with different resource names that correspond to ZIP files with CSV data on a remote server.

To download these files concurrently using multiple threads of execution, create a new thread pool and map your download_file() function onto each item from the list:

>>> with ThreadPoolExecutor() as executor:
...     executor.map(download_file, urls)
...
<generator object Executor.map.<locals>.result_iterator at 0x7fc9c90f0640>
Downloaded file API_SP.POP.TOTL_DS2_en_csv_v2_5551506.zip
Downloaded file API_NY.GDP.MKTP.CD_DS2_en_csv_v2_5551501.zip
Downloaded file API_EN.POP.DNST_DS2_en_csv_v2_5552158.zip

As before, you use the with statement to help with resource management for your threads. Within the executor’s context, you call the .map() method with the function that you want to execute. You also pass it an iterable, which is the list of URLs in this case. The executor object allocates a pool of threads up front and assigns a separate thread to each download task.

Because downloading data from a network is an I/O-bound task, which is limited by the speed at which the data can be read rather than the CPU speed, you achieve true parallelism. Despite Python’s global interpreter lock (GIL), which would otherwise prevent that, the threads in the executor perform multiple tasks in parallel, resulting in faster overall completion time.

The sample ZIP files are relatively small and similar in size, so they finish downloading at roughly the same time. But what if you used a for loop to download the files instead? Go ahead and call download_file() with each URL on the main thread:

>>> for url in urls:
...     download_file(url)
...

You may have noticed that this operation took much longer. That’s because the files were downloaded sequentially instead of concurrently. So when the first file starts downloading, the second one won’t start until it finishes, and so on. This means that the total time required to download all the files is the sum of the download times for each individual file.

Unfortunately, using threads isn’t always desirable due to their complexity. The example in this section was fairly straightforward because you didn’t have to worry about thread synchronization, coordination, or resource management. But when working with multithreaded code, you should be careful about thread safety to ensure that you safely access and modify shared resources to avoid data corruption.

There are some conditions where multithreading may not improve your performance at all. If the underlying problem is inherently sequential, then there’s no way to parallelize it. Moreover, if your tasks involve CPU-bound operations, then Python won’t be able to take advantage of multiple CPU cores because of the GIL from earlier. The additional cost of context switching may actually reduce performance!

If working with threads makes your head spin, then you might be interested in exploring alternative solutions, like Python’s asynchronous programming capabilities.

Using the Asynchronous aiohttp Library

In addition to multithreading, another method to download multiple files concurrently is by using the async/await pattern in Python. It involves running multiple non-blocking tasks asynchronously in an event loop by allowing them to suspend and resume execution voluntarily as a form of cooperative multitasking. This is different from threads, which require a preemptive scheduler to manage context switching between them.

Asynchronous tasks also differ from multithreading in that they execute concurrently within a single thread instead of multiple threads. Therefore, they must periodically give up their execution time to other tasks without hogging the CPU. By doing so, I/O-bound tasks such as asynchronous downloads allow for concurrency, as the program can switch between tasks and make progress on each in parallel.

A popular Python package to perform asynchronous downloads when retrieving multiple files is the aiohttp library. It’s built on top of the standard library’s asyncio module, which provides a framework for asynchronous programming in Python.

The aiohttp library takes advantage of the concurrency features in the asyncio package, allowing you to write asynchronous code that can handle multiple requests concurrently. The library can perform non-blocking network operations, meaning it’ll let other code run while another task waits for data to arrive from the network.

By using the aiohttp API along with the async def and await keywords, you can write asynchronous code that makes concurrent HTTP requests. In addition, the aiohttp library supports connection pooling, a feature that allows multiple requests to use the same underlying connection. This helps to optimize and improve performance.

To begin, install the aiohttp library using pip in the command line:

(venv) $ python -m pip install aiohttp

This installs the aiohttp library into your active virtual environment.

In addition to this third-party library, you’ll also need the asyncio package from the Python standard library to perform asynchronous downloads. So, import both packages now:

>>> import asyncio
>>> import aiohttp

The next step is defining an asynchronous function to download a file from a URL. You can do so by creating an aiohttp.ClientSession instance, which holds a connector reused for multiple connections. It automatically keeps them alive for a certain time period to reuse them in subsequent requests to the same server whenever possible. This improves performance and reduces the overhead of establishing new connections for each request.

The following function performs an asynchronous download using the ClientSession class from the aiohttp package:

>>> async def download_file(url):
...     async with aiohttp.ClientSession() as session:
...         async with session.get(url) as response:
...             if "content-disposition" in response.headers:
...                 header = response.headers["content-disposition"]
...                 filename = header.split("filename=")[1]
...             else:
...                 filename = url.split("/")[-1]
...             with open(filename, mode="wb") as file:
...                 while True:
...                     chunk = await response.content.read()
...                     if not chunk:
...                         break
...                     file.write(chunk)
...                 print(f"Downloaded file {filename}")
...

The function defined above takes a URL as an argument. It then creates a client session using the async with statement, ensuring that the session is properly closed and resources are released after the program exits this code block. With the session context, it makes an HTTP GET request to the specified URL and obtains the response object using the async with statement.

Inside the infinite loop, you read data in chunks, breaking out of the loop when there are no more chunks. The await keyword indicates that this operation is asynchronous and other tasks can execute in parallel until the data is available. After deriving the filename from the response object, you save the downloaded chunk of data in a local file.

Afterward, you can perform concurrent downloads using the asynchronous capabilities of the aiohttp and asyncio libraries. You may reuse the code from an earlier example based on multithreading to prepare a list of URLs:

>>> template_url = (
...     "https://api.worldbank.org/v2/en/indicator/"
...     "{resource}?downloadformat=csv"
... )

>>> urls = [
...     # Total population by country
...     template_url.format(resource="SP.POP.TOTL"),
...
...     # GDP by country
...     template_url.format(resource="NY.GDP.MKTP.CD"),
...
...     # Population density by country
...     template_url.format(resource="EN.POP.DNST"),
... ]

Finally, define and run an asynchronous main() function that will download files concurrently from those URLs:

>>> async def main():
...     tasks = [download_file(url) for url in urls]
...     await asyncio.gather(*tasks)
...
>>> asyncio.run(main())
Downloaded file API_SP.POP.TOTL_DS2_en_csv_v2_5551506.zip
Downloaded file API_EN.POP.DNST_DS2_en_csv_v2_5552158.zip
Downloaded file API_NY.GDP.MKTP.CD_DS2_en_csv_v2_5551501.zip

In the snippet above, the main() function defines a list comprehension to create a list of tasks, whereby each task calls the download_file() function for each URL in the global variable urls. Note that you define the function with the async keyword to make it asynchronous. Then, you use asyncio.gather() to wait until all tasks in the list are completed.

You run the main() function using asyncio.run(), which initiates and executes the asynchronous tasks in an event loop. This code downloads files concurrently from the specified URLs without blocking the execution of other tasks.

File Sizes to Download

If you’re looking to download large files, then the requests library is a good option that will handle them efficiently. It can stream data, letting you iterate over the message body in chunks for a more efficient process and better memory use.

When downloading multiple files, you can either use the requests library together with multithreading or the aiohttp library with asynchronous downloads to perform concurrent downloads. When used properly, either can improve performance and optimize the download process.

User-Friendliness

If you’re looking for something quick and straightforward, then the urllib package is already included in Python’s standard library, requiring no additional installation. This is a good option if you want to download small files that can fit entirely into memory without any issues.

While you could build many functionalities using urllib, it may be harder to use and require more manual setup for more involved use cases, such as streaming data and downloading multiple files in parallel. In fact, there are third-party libraries with functions that readily support these features, such as the requests library, which can support data streaming with an argument in the requests.get() method.

Additional Features and Flexibility

The requests library has a rich set of features that can help you in numerous other download scenarios. Although you didn’t extensively cover these tasks, the requests library has features that can handle authentication, redirects, session management, and more. These features can give you more control and flexibility for more advanced tasks.

If you’d like to learn more about a project that might require extra features supported by the requests library, then check out this project on building a content aggregator. Creating a content aggregator involves steps to download data from multiple websites, some of which may require authentication and session management, so requests would really come in handy.