Digging Deeper Into Django Migrations

Migration Operations

Let’s look at the operations list first. This table contains the operations that are to be performed as part of the migration. Operations are subclasses of the class django.db.migrations.operations.base.Operation. Here are the common operations that are built into Django:

Note how the operations are named after changes made to model definitions, not the actions that are performed on the database. When you apply a migration, each operation is responsible for generating the necessary SQL statements for your specific database. For example, CreateModel would generate a CREATE TABLE SQL statement.

Out of the box, migrations have support for all the standard databases that Django supports. So if you stick to the operations listed here, then you can do more or less any changes to your models that you want, without having to worry about the underlying SQL. That’s all done for you.

Django provides three more operation classes for advanced use cases:

1. RunSQL allows you to run custom SQL in the database.
2. RunPython allows you to run any Python code.
3. SeparateDatabaseAndState is a specialized operation for advanced uses.

With these operations, you can basically do any changes you want to your database. However, you won’t find these operations in a migration that has been created automatically with the makemigrations management command.

Since Django 2.0, there are also a couple of PostgreSQL-specific operations available in django.contrib.postgres.operations that you can use to install various PostgreSQL extensions:

• BtreeGinExtension
• BtreeGistExtension
• CITextExtension
• CryptoExtension
• HStoreExtension
• TrigramExtension
• UnaccentExtension

Note that a migration containing one of these operations requires a database user with superuser privileges.

Last but not least, you can also create your own operation classes. If you want to look into that, then take a look at the Django documentation on creating custom migration operations.

Migration Dependencies

The dependencies list in a migration class contains any migrations that must be applied before this migration can be applied.

In the 0001_initial.py migration you saw above, nothing has to be applied prior so there are no dependencies. Let’s have a look at the second migration in the historical_prices app. In the file 0002_switch_to_decimals.py, the dependencies attribute of Migration has an entry:

from django.db import migrations, models

class Migration(migrations.Migration):
    dependencies = [
        ('historical_data', '0001_initial'),
    ]
    operations = [
        migrations.AlterField(
            model_name='pricehistory',
            name='volume',
            field=models.DecimalField(decimal_places=3, max_digits=7),
        ),
    ]

The dependency above says that migration 0001_initial of the app historical_data must be run first. That makes sense, because the migration 0001_initial creates the table containing the field that the migration 0002_switch_to_decimals wants to change.

A migration can also have a dependency on a migration from another app, like this:

class Migration(migrations.Migration):
    ...

    dependencies = [
        ('auth', '0009_alter_user_last_name_max_length'),
    ]

This is usually necessary if a model has a Foreign Key pointing to a model in another app.

Alternatively, you can also enforce that a migration is run before another migration using the attribute run_before:

class Migration(migrations.Migration):
    ...

    run_before = [
        ('third_party_app', '0001_initial'),
    ]

Dependencies can also be combined so you can have multiple dependencies. This functionality provides a lot of flexibility, as you can accommodate foreign keys that depend upon models from different apps.

The option to explicitly define dependencies between migrations also means that the numbering of the migrations (usually 0001, 0002, 0003, …) doesn’t strictly represent the order in which migrations are applied. You can add any dependency you want and thus control the order without having to re-number all the migrations.

Viewing the Migration

You generally don’t have to worry about the SQL that migrations generate. But if you want to double-check that the generated SQL makes sense or are just curious what it looks like, then Django’s got you covered with the sqlmigrate management command:

$ python manage.py sqlmigrate historical_data 0001
BEGIN;
--
-- Create model PriceHistory
--
CREATE TABLE "historical_data_pricehistory" (
    "id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
    "date" datetime NOT NULL,
    "price" decimal NOT NULL,
    "volume" integer unsigned NOT NULL
);
COMMIT;

Doing that will list out the underlying SQL queries that will be generated by the specified migration, based upon the database in your settings.py file. When you pass the parameter --backwards, Django generates the SQL to unapply the migration:

$ python manage.py sqlmigrate --backwards historical_data 0001
BEGIN;
--
-- Create model PriceHistory
--
DROP TABLE "historical_data_pricehistory";
COMMIT;

Once you see the output of sqlmigrate for a slightly more complex migration, you may appreciate that you don’t have to craft all this SQL by hand!

Playing Chess With Django

You can think of your models like a chess board, and Django is a chess grandmaster watching you play against yourself. But the grandmaster doesn’t watch your every move. The grandmaster only looks at the board when you shout makemigrations.

Because there’s only a limited set of possible moves (and the grandmaster is a grandmaster), she can come up with the moves that have happened since she last looked at the board. She takes some notes and lets you play until you shout makemigrations again.

When looking at the board the next time, the grandmaster doesn’t remember what the chessboard looked like the last time, but she can go through her notes of the previous moves and build a mental model of what the chessboard looked like.

Now, when you shout migrate, the grandmaster will replay all the recorded moves on another chessboard and note in a spreadsheet which of her records have already been applied. This second chess board is your database, and the spreadsheet is the django_migrations table.

This analogy is quite fitting, because it nicely illustrates some behaviors of Django migrations:

• Django migrations try to be efficient: Just like the grandmaster assumes that you made the least number of moves, Django will try to create the most efficient migrations. If you add a field named A to a model, then rename it to B, and then run makemigrations, then Django will create a new migration to add a field named B.

• Django migrations have their limits: If you make a lot of moves before you let the grandmaster look at the chessboard, then she might not be able to retrace the exact movements of each piece. Similarly, Django might not come up with the correct migration if you make too many changes at once.

• Django migration expect you to play by the rules: When you do anything unexpected, like taking a random piece off the board or messing with the notes, the grandmaster might not notice at first, but sooner or later, she’ll throw up her hands and refuse to continue. The same happens when you mess with the django_migrations table or change your database schema outside of migrations, for example by deleting the database table for a model.

Understanding SeparateDatabaseAndState

Now that you know about the project state that Django builds, it’s time to take a closer look at the operation SeparateDatabaseAndState. This operation can do exactly what the name implies: it can separate the project state (the mental model Django builds) from your database.

SeparateDatabaseAndState is instantiated with two lists of operations:

1. state_operations contains operations that are only applied to the project state.
2. database_operations contains operations that are only applied to the database.

This operation lets you do any kind of change to your database, but it’s your responsibility to make sure that the project state fits the database afterwards. Example use cases for SeparateDatabaseAndState are moving a model from one app to another or creating an index on a huge database without downtime.

SeparateDatabaseAndState is an advanced operation and you won’t need on your first day working with migrations and maybe never at all. SeparateDatabaseAndState is similar to heart surgery. It carries quite a bit of risk and is not something you do just for fun, but sometimes it’s a necessary procedure to keep the patient alive.