Python mmap: Improved File I/O With Memory Mapping

Physical Memory

Physical memory is the least complicated type of memory to understand because it’s often part of the marketing associated with your computer. (You might remember that when you bought your computer, it advertised something like 8 gigabytes of RAM.) Physical memory typically comes on cards that are connected to your computer’s motherboard.

Physical memory is the amount of volatile memory that’s available for your programs to use while running. Physical memory should not be confused with storage, such as your hard drive or solid-state disk.

Virtual Memory

Virtual memory is a way of handling memory management. The operating system uses virtual memory to make it appear that you have more memory than you do, allowing you to worry less about how much memory is available for your programs at any given time. Behind the scenes, your operating system uses parts of your nonvolatile storage, such as your solid-state disk, to simulate additional RAM.

In order to do this, your operating system must maintain a mapping between physical memory and virtual memory. Each operating system uses its own sophisticated algorithm to map virtual memory addresses to physical ones using a data structure called a page table.

Luckily, most of this complication is hidden from your programs. You don’t need to understand page tables or logical-to-physical mapping to write performant I/O code in Python. However, knowing a little bit about memory gives you a better understanding of what the computer and libraries are taking care of for you.

mmap uses virtual memory to make it appear that you’ve loaded a very large file into memory, even if the contents of the file are too big to fit in your physical memory.

Shared Memory

Shared memory is another technique provided by your operating system that allows multiple programs to access the same data simultaneously. Shared memory can be a very efficient way of handling data in a program that uses concurrency.

Python’s mmap uses shared memory to efficiently share large amounts of data between multiple Python processes, threads, and tasks that are happening concurrently.

System Calls

In reality, the call to read() signals the operating system to do a lot of sophisticated work. Luckily, operating systems provide a way to abstract the specific details of each hardware device away from your programs with system calls. Each operating system will implement this functionality differently, but at the very least, read() has to perform several system calls to retrieve data from the file.

All access with the physical hardware must happen in a protected environment called kernel space. System calls are the API that the operating system provides to allow your program to go from user space to kernel space, where the low-level details of the physical hardware are managed.

In the case of read(), several system calls are needed for the operating system to interact with the physical storage device and return the data.

Again, you don’t need a firm grasp on the details of system calls and computer architecture to understand memory mapping. The most important thing to remember is that system calls are relatively expensive computationally speaking, so the fewer system calls you do, the faster your code will likely execute.

In addition to the system calls, the call to read() also involves a lot of potentially unnecessary copying of data between multiple data buffers before the data gets all the way back to your program.

Typically, this all happens so fast that it’s not noticeable. But all these layers add latency and can slow down your program. This is where memory mapping comes into play.

Memory Mapping Optimizations

One way to avoid this overhead is to use a memory-mapped file. You can picture memory mapping as a process in which read and write operations skip many of the layers mentioned above and map the requested data directly into physical memory.

A memory-mapped file I/O approach sacrifices memory usage for speed, which is classically called the space–time tradeoff. However, memory mapping doesn’t have to use more memory than the conventional approach. The operating system is very clever. It will lazily load the data as it’s requested, similar to how Python generators work.

In addition, thanks to virtual memory, you can load a file that’s larger than your physical memory. However, you won’t see the huge performance improvements from memory mapping when there isn’t enough physical memory for your file, because the operating system will use a slower physical storage medium like a solid-state disk to mimic the physical memory it lacks.

Performance Implications

The memory-mapping approach is slightly more complicated than the typical file I/O because it requires creating another object. However, that small change can lead to big performance benefits when reading a file of just a few megabytes. Here’s a comparison of reading the raw text of the famous novel The History of Don Quixote, which is roughly 2.4 megabytes:

>>> import timeit
>>> timeit.repeat(
...     "regular_io(filename)",
...     repeat=3,
...     number=1,
...     setup="from __main__ import regular_io, filename")
[0.02022400000000002, 0.01988580000000001, 0.020257300000000006]
>>> timeit.repeat(
...     "mmap_io(filename)",
...     repeat=3,
...     number=1,
...     setup="from __main__ import mmap_io, filename")
[0.006156499999999981, 0.004843099999999989, 0.004868600000000001]

This measures the amount of time to read an entire 2.4-megabyte file using regular file I/O and memory-mapped file I/O. As you can see, the memory mapped approach takes around .005 seconds versus almost .02 seconds for the regular approach. This performance improvement can be even bigger when reading a larger file.

The API provided by Python’s mmap file object is very similar to the traditional file object except for one additional superpower: Python’s mmap file object can be sliced just like string objects!

mmap Object Creation

There are few subtleties in the creation of the mmap object that are worth looking at more closely:

mmap.mmap(file_obj.fileno(), length=0, access=mmap.ACCESS_READ)

mmap requires a file descriptor, which comes from the fileno() method of a regular file object. A file descriptor is an internal identifier, typically an integer, that the operating system uses to keep track of open files.

The second argument to mmap is length=0. This is the length in bytes of the memory map. 0 is a special value indicating that the system should create a memory map large enough to hold the entire file.

The access argument tells the operating system how you’re going to interact with the mapped memory. The options are ACCESS_READ, ACCESS_WRITE, ACCESS_COPY, and ACCESS_DEFAULT. These are somewhat similar to the mode arguments to the built-in open():

• ACCESS_READ creates a read-only memory map.
• ACCESS_DEFAULT defaults to the mode specified in the optional prot argument, which is used for memory protection.
• ACCESS_WRITE and ACCESS_COPY are the two write modes, which you’ll learn about below.

The file descriptor, length, and access arguments represent the bare minimum you need to create a memory-mapped file that will work across operating systems like Windows, Linux, and macOS. The code above is cross-platform, meaning it will read the file through the memory-mapping interface on all operating systems without needing to know which operating system the code runs on.

Another useful argument is offset, which can be a memory-saving technique. This instructs the mmap to create a memory map starting at a specified offset in the file.

mmap Objects as Strings

As previously mentioned, memory mapping transparently loads the file contents into memory as a string. So, once you open the file, you can perform lots of the same operations you use with strings, such as slicing:

import mmap

def mmap_io(filename):
    with open(filename, mode="r", encoding="utf8") as file_obj:
        with mmap.mmap(file_obj.fileno(), length=0, access=mmap.ACCESS_READ) as mmap_obj:
            print(mmap_obj[10:20])

This code prints ten characters from mmap_obj to the screen and also reads those ten characters into physical memory. Again, the data is read lazily.

The slice does not advance the internal file position. So, if you were to call read() after a slice, then you would still read from the beginning of the file.

Search a Memory-Mapped File

In addition to slicing, the mmap module allows other string-like behavior such as using find() and rfind() to search a file for specific text. For example, here are two approaches to find the first occurrence of " the " in a file:

import mmap

def regular_io_find(filename):
    with open(filename, mode="r", encoding="utf-8") as file_obj:
        text = file_obj.read()
        print(text.find(" the "))

def mmap_io_find(filename):
    with open(filename, mode="r", encoding="utf-8") as file_obj:
        with mmap.mmap(file_obj.fileno(), length=0, access=mmap.ACCESS_READ) as mmap_obj:
            print(mmap_obj.find(b" the "))

These two functions both search a file for the first occurrence of " the " The main difference between them is that the first uses find() on a string object, whereas the second uses find() on a memory-mapped file object.

Here’s the performance difference:

>>> import timeit
>>> timeit.repeat(
...     "regular_io_find(filename)",
...     repeat=3,
...     number=1,
...     setup="from __main__ import regular_io_find, filename")
[0.01919180000000001, 0.01940510000000001, 0.019157700000000027]
>>> timeit.repeat(
...     "mmap_io_find(filename)",
...     repeat=3,
...     number=1,
...     setup="from __main__ import mmap_io_find, filename")
[0.0009397999999999906, 0.0018005999999999855, 0.000826699999999958]

That’s several orders of magnitude difference! Again, your results may differ depending on your operating system.

Memory-mapped files can also be used directly with regular expressions. Consider the following example that finds and prints out all five-letter words:

import re
import mmap

def mmap_io_re(filename):
    five_letter_word = re.compile(rb"\b[a-zA-Z]{5}\b")

    with open(filename, mode="r", encoding="utf-8") as file_obj:
        with mmap.mmap(file_obj.fileno(), length=0, access=mmap.ACCESS_READ) as mmap_obj:
            for word in five_letter_word.findall(mmap_obj):
                print(word)

This code reads the entire file and prints out every word that has exactly five letters in it. Keep in mind that memory-mapped files work with byte strings, so the regular expressions must also use byte strings.

Here’s the equivalent code using regular file I/O:

import re

def regular_io_re(filename):
    five_letter_word = re.compile(r"\b[a-zA-Z]{5}\b")

    with open(filename, mode="r", encoding="utf-8") as file_obj:
        for word in five_letter_word.findall(file_obj.read()):
            print(word)

This code also prints out all five-character words in the file, but it uses the traditional file I/O mechanism instead of memory-mapped files. As before, the performance differs between the two approaches:

>>> import timeit
>>> timeit.repeat(
...     "regular_io_re(filename)",
...     repeat=3,
...     number=1,
...     setup="from __main__ import regular_io_re, filename")
[0.10474110000000003, 0.10358619999999996, 0.10347820000000002]
>>> timeit.repeat(
...     "mmap_io_re(filename)",
...     repeat=3,
...     number=1,
...     setup="from __main__ import mmap_io_re, filename")
[0.0740976000000001, 0.07362639999999998, 0.07380980000000004]

The memory-mapped approach is still an order of magnitude faster.

Memory-Mapped Objects as Files

A memory-mapped file is part string and part file, so mmap also allows you to perform common file operations like seek(), tell(), and readline(). These functions work exactly like their regular file-object counterparts.

For example, here’s how to seek to a particular location in a file and then perform a search for a word:

import mmap

def mmap_io_find_and_seek(filename):
    with open(filename, mode="r", encoding="utf-8") as file_obj:
        with mmap.mmap(file_obj.fileno(), length=0, access=mmap.ACCESS_READ) as mmap_obj:
            mmap_obj.seek(10000)
            mmap_obj.find(b" the ")

This code will seek to location 10000 in the file and then find the location of the first occurrence of " the ".

seek() works exactly the same on memory-mapped files as it does on regular files:

def regular_io_find_and_seek(filename):
    with open(filename, mode="r", encoding="utf-8") as file_obj:
        file_obj.seek(10000)
        text = file_obj.read()
        text.find(" the ")

The code for both approaches is very similar. Let’s see how their performance compares:

>>> import timeit
>>> timeit.repeat(
...     "regular_io_find_and_seek(filename)",
...     repeat=3,
...     number=1,
...     setup="from __main__ import regular_io_find_and_seek, filename")
[0.019396099999999916, 0.01936059999999995, 0.019192100000000045]
>>> timeit.repeat(
...     "mmap_io_find_and_seek(filename)",
...     repeat=3,
...     number=1,
...     setup="from __main__ import mmap_io_find_and_seek, filename")
[0.000925100000000012, 0.000788299999999964, 0.0007854999999999945]

Again, after only a few small tweaks to the code, your memory-mapped approach is much faster.

Write Modes

The semantics of the write operation are controlled by the access parameter. One distinction between writing memory-mapped files and regular files is the options for the access parameter. There are two options to control how data is written to a memory-mapped file:

1. ACCESS_WRITE specifies write-through semantics, meaning the data will be written through memory and persisted on disk.
2. ACCESS_COPY does not write the changes to disk, even if flush() is called.

In other words, ACCESS_WRITE writes both to memory and to the file, whereas ACCESS_COPY writes only to memory and not to the underlying file.

Search and Replace Text

Memory-mapped files expose the data as a string of bytes, but that string of bytes has another important advantage over a regular string. Memory-mapped file data is a string of mutable bytes. This means it’s much more straightforward and efficient to write code that searches and replaces data in a file:

import mmap
import os
import shutil

def regular_io_find_and_replace(filename):
    with open(filename, "r", encoding="utf-8") as orig_file_obj:
        with open("tmp.txt", "w", encoding="utf-8") as new_file_obj:
            orig_text = orig_file_obj.read()
            new_text = orig_text.replace(" the ", " eht ")
            new_file_obj.write(new_text)

    shutil.copyfile("tmp.txt", filename)
    os.remove("tmp.txt")

def mmap_io_find_and_replace(filename):
    with open(filename, mode="r+", encoding="utf-8") as file_obj:
        with mmap.mmap(file_obj.fileno(), length=0, access=mmap.ACCESS_WRITE) as mmap_obj:
            orig_text = mmap_obj.read()
            new_text = orig_text.replace(b" the ", b" eht ")
            mmap_obj[:] = new_text
            mmap_obj.flush()

Both of these functions change the word " the " to " eht " in the given file. As you can see, the memory-mapped approach is roughly the same, but it doesn’t require manually keeping track of an additional temporary file to do the replacement in place.

In this scenario, the memory-mapped approach is actually slightly slower for this file length. So, doing a full search-and-replace on a memory-mapped file may or may not be the most efficient approach. It likely depends on many things such as the file length, your machine’s RAM speed, etc. There might also be some operating system caching that skews the times. As you can see, the regular IO approach sped up on each call.

>>> import timeit
>>> timeit.repeat(
...     "regular_io_find_and_replace(filename)",
...     repeat=3,
...     number=1,
...     setup="from __main__ import regular_io_find_and_replace, filename")
[0.031016973999996367, 0.019185273000005054, 0.019321329999996806]
>>> timeit.repeat(
...     "mmap_io_find_and_replace(filename)",
...     repeat=3,
...     number=1,
...     setup="from __main__ import mmap_io_find_and_replace, filename")
[0.026475408999999672, 0.030173652999998524, 0.029132930999999473]

In this basic search-and-replace scenario, memory mapping results in slightly more concise code, but not always a massive speed improvement. As they say, “your mileage may vary.”