Pointers in Python: What's the Point?

Variables in C

Let’s say you had the following code that defines the variable x:

int x = 2337;

This one line of code has several, distinct steps when executed:

1. Allocate enough memory for an integer
2. Assign the value 2337 to that memory location
3. Indicate that x points to that value

Shown in a simplified view of memory, it might look like this:

Here, you can see that the variable x has a fake memory location of 0x7f1 and the value 2337. If, later in the program, you want to change the value of x, you can do the following:

x = 2338;

The above code assigns a new value (2338) to the variable x, thereby overwriting the previous value. This means that the variable x is mutable. The updated memory layout shows the new value:

Notice that the location of x didn’t change, just the value itself. This is a significant point. It means that x is the memory location, not just a name for it.

Another way to think of this concept is in terms of ownership. In one sense, x owns the memory location. x is, at first, an empty box that can fit exactly one integer in which integer values can be stored.

When you assign a value to x, you’re placing a value in the box that x owns. If you wanted to introduce a new variable (y), you could add this line of code:

int y = x;

This code creates a new box called y and copies the value from x into the box. Now the memory layout will look like this:

Notice the new location 0x7f5 of y. Even though the value of x was copied to y, the variable y owns some new address in memory. Therefore, you could overwrite the value of y without affecting x:

y = 2339;

Now the memory layout will look like this:

Again, you have modified the value at y, but not its location. In addition, you have not affected the original x variable at all. This is in stark contrast with how Python names work.

Names in Python

Python does not have variables. It has names. Yes, this is a pedantic point, and you can certainly use the term variables as much as you like. It is important to know that there is a difference between variables and names.

Let’s take the equivalent code from the above C example and write it in Python:

>>> x = 2337

Much like in C, the above code is broken down into several distinct steps during execution:

1. Create a PyObject
2. Set the typecode to integer for the PyObject
3. Set the value to 2337 for the PyObject
4. Create a name called x
5. Point x to the new PyObject
6. Increase the refcount of the PyObject by 1

In memory, it might looks something like this:

You can see that the memory layout is vastly different than the C layout from before. Instead of x owning the block of memory where the value 2337 resides, the newly created Python object owns the memory where 2337 lives. The Python name x doesn’t directly own any memory address in the way the C variable x owned a static slot in memory.

If you were to try to assign a new value to x, you could try the following:

>>> x = 2338

What’s happening here is different than the C equivalent, but not too different from the original bind in Python.

This code:

• Creates a new PyObject
• Sets the typecode to integer for the PyObject
• Sets the value to 2338 for the PyObject
• Points x to the new PyObject
• Increases the refcount of the new PyObject by 1
• Decreases the refcount of the old PyObject by 1

Now in memory, it would look something like this:

This diagram helps illustrate that x points to a reference to an object and doesn’t own the memory space as before. It also shows that the x = 2338 command is not an assignment, but rather binding the name x to a reference.

In addition, the previous object (which held the 2337 value) is now sitting in memory with a ref count of 0 and will get cleaned up by the garbage collector.

You could introduce a new name, y, to the mix as in the C example:

>>> y = x

In memory, you would have a new name, but not necessarily a new object:

Now you can see that a new Python object has not been created, just a new name that points to the same object. Also, the object’s refcount has increased by one. You could check for object identity equality to confirm that they are the same:

>>> y is x
True

The above code indicates that x and y are the same object. Make no mistake though: y is still immutable.

For example, you could perform addition on y:

>>> y += 1
>>> y is x
False

After the addition call, you are returned with a new Python object. Now, the memory looks like this:

A new object has been created, and y now points to the new object. Interestingly, this is the same end-state if you had bound y to 2339 directly:

>>> y = 2339

The above statement results in the same end-memory state as the addition. To recap, in Python, you don’t assign variables. Instead, you bind names to references.

A Note on Intern Objects in Python

Now that you understand how Python objects get created and names get bound to those objects, its time to throw a wrench in the machinery. That wrench goes by the name of interned objects.

Suppose you have the following Python code:

>>> x = 1000
>>> y = 1000
>>> x is y
True

As above, x and y are both names that point to the same Python object. But the Python object that holds the value 1000 is not always guaranteed to have the same memory address. For example, if you were to add two numbers together to get 1000, you would end up with a different memory address:

>>> x = 1000
>>> y = 499 + 501
>>> x is y
False

This time, the line x is y returns False. If this is confusing, then don’t worry. Here are the steps that occur when this code is executed:

1. Create Python object(1000)
2. Assign the name x to that object
3. Create Python object (499)
4. Create Python object (501)
5. Add these two objects together
6. Create a new Python object (1000)
7. Assign the name y to that object

Isn’t this wasteful? Well, yes it is, but that’s the price you pay for all of the great benefits of Python. You never have to worry about cleaning up these intermediate objects or even need to know that they exist! The joy is that these operations are relatively fast, and you never had to know any of those details until now.

The core Python developers, in their wisdom, also noticed this waste and decided to make a few optimizations. These optimizations result in behavior that can be surprising to newcomers:

>>> x = 20
>>> y = 19 + 1
>>> x is y
True

In this example, you see nearly the same code as before, except this time the result is True. This is the result of interned objects. Python pre-creates a certain subset of objects in memory and keeps them in the global namespace for everyday use.

Which objects depend on the implementation of Python. CPython 3.7 interns the following:

1. Integer numbers between -5 and 256
2. Strings that contain ASCII letters, digits, or underscores only

The reasoning behind this is that these variables are extremely likely to be used in many programs. By interning these objects, Python prevents memory allocation calls for consistently used objects.

Strings that are less than 20 characters and contain ASCII letters, digits, or underscores will be interned. The reasoning behind this is that these are assumed to be some kind of identity:

>>> s1 = "realpython"
>>> id(s1)
140696485006960
>>> s2 = "realpython"
>>> id(s2)
140696485006960
>>> s1 is s2
True

Here you can see that s1 and s2 both point to the same address in memory. If you were to introduce a non-ASCII letter, digit, or underscore, then you would get a different result:

>>> s1 = "Real Python!"
>>> s2 = "Real Python!"
>>> s1 is s2
False

Because this example has an exclamation mark (!) in it, these strings are not interned and are different objects in memory.

Interned objects are often a source of confusion. Just remember, if you’re ever in doubt, that you can always use id() and is to determine object equality.

Using Mutable Types as Pointers

You’ve already learned about mutable types. Because these objects are mutable, you can treat them as if they were pointers to simulate pointer behavior. Suppose you wanted to replicate the following c code:

void add_one(int *x) {
    *x += 1;
}

This code takes a pointer to an integer (*x) and then increments the value by one. Here is a main function to exercise the code:

#include <stdio.h>

int main(void) {
    int y = 2337;
    printf("y = %d\n", y);
    add_one(&y);
    printf("y = %d\n", y);
    return 0;
}

In the above code, you assign 2337 to y, print out the current value, increment the value by one, and then print out the modified value. The output of executing this code would be the following:

y = 2337
y = 2338

One way to replicate this type of behavior in Python is by using a mutable type. Consider using a list and modifying the first element:

>>> def add_one(x):
...     x[0] += 1
...
>>> y = [2337]
>>> add_one(y)
>>> y[0]
2338

Here, add_one(x) accesses the first element and increments its value by one. Using a list means that the end result appears to have modified the value. So pointers in Python do exist? Well, no. This is only possible because list is a mutable type. If you tried to use a tuple, you would get an error:

>>> z = (2337,)
>>> add_one(z)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in add_one
TypeError: 'tuple' object does not support item assignment

The above code demonstrates that tuple is immutable. Therefore, it does not support item assignment. list is not the only mutable type. Another common approach to mimicking pointers in Python is to use a dict.

Let’s say you had an application where you wanted to keep track of every time an interesting event happened. One way to achieve this would be to create a dict and use one of the items as a counter:

>>> counters = {"func_calls": 0}
>>> def bar():
...     counters["func_calls"] += 1
...
>>> def foo():
...     counters["func_calls"] += 1
...     bar()
...
>>> foo()
>>> counters["func_calls"]
2

In this example, the counters dictionary is used to keep track of the number of function calls. After you call foo(), the counter has increased to 2 as expected. All because dict is mutable.

Keep in mind, this is only simulates pointer behavior and does not directly map to true pointers in C or C++. That is to say, these operations are more expensive than they would be in C or C++.

Using Python Objects

The dict option is a great way to emulate pointers in Python, but sometimes it gets tedious to remember the key name you used. This is especially true if you’re using the dictionary in various parts of your application. This is where a custom Python class can really help.

To build on the last example, assume that you want to track metrics in your application. Creating a class is a great way to abstract the pesky details:

class Metrics(object):
    def __init__(self):
        self._metrics = {
            "func_calls": 0,
            "cat_pictures_served": 0,
        }

This code defines a Metrics class. This class still uses a dict for holding the actual data, which is in the _metrics member variable. This will give you the mutability you need. Now you just need to be able to access these values. One nice way to do this is with properties:

class Metrics(object):
    # ...

    @property
    def func_calls(self):
        return self._metrics["func_calls"]

    @property
    def cat_pictures_served(self):
        return self._metrics["cat_pictures_served"]

This code makes use of @property. If you’re not familiar with decorators, you can check out this Primer on Python Decorators. The @property decorator here allows you to access func_calls and cat_pictures_served as if they were attributes:

>>> metrics = Metrics()
>>> metrics.func_calls
0
>>> metrics.cat_pictures_served
0

The fact that you can access these names as attributes means that you abstracted the fact that these values are in a dict. You also make it more explicit what the names of the attributes are. Of course, you need to be able to increment these values:

class Metrics(object):
    # ...

    def inc_func_calls(self):
        self._metrics["func_calls"] += 1

    def inc_cat_pics(self):
        self._metrics["cat_pictures_served"] += 1

You have introduced two new methods:

1. inc_func_calls()
2. inc_cat_pics()

These methods modify the values in the metrics dict. You now have a class that you modify as if you’re modifying a pointer:

>>> metrics = Metrics()
>>> metrics.inc_func_calls()
>>> metrics.inc_func_calls()
>>> metrics.func_calls
2

Here, you can access func_calls and call inc_func_calls() in various places in your applications and simulate pointers in Python. This is useful when you have something like metrics that need to be used and updated frequently in various parts of your applications.

Here’s the full source for the Metrics class:

class Metrics(object):
    def __init__(self):
        self._metrics = {
            "func_calls": 0,
            "cat_pictures_served": 0,
        }

    @property
    def func_calls(self):
        return self._metrics["func_calls"]

    @property
    def cat_pictures_served(self):
        return self._metrics["cat_pictures_served"]

    def inc_func_calls(self):
        self._metrics["func_calls"] += 1

    def inc_cat_pics(self):
        self._metrics["cat_pictures_served"] += 1