27. Database/API Versioning & Migration

Written by Tim Condon

In the first three sections of the book, whenever you made a change to your model, you had to delete your database and start over. That’s no problem when you don’t have any data. Once you have data, or move your project to the production stage, you can no longer delete your database. What you want to do instead is modify your database, which in Vapor, is done using migrations.

In this chapter, you’ll make two modifications to the TILApp using migrations. First, you’ll add a new field to User to contain a Twitter handle. Second, you’ll ensure that categories are unique. Finally, you’re going to modify the app so it creates the admin user only when your app runs in development or testing mode.

Note: The starter project for this chapter is based on the TIL application from the end of chapter 21. The starter project contains extra code, so you should use the starter project from this chapter. This project relies on a PostgreSQL database running locally.

How migrations works

When Fluent runs for the first time, it creates a special table in the database. Fluent uses this table to track all migrations it has run and it runs migrations in the order you add them. When your application starts, Fluent checks the list of migrations to run. If it has run a migration, it will move on to the next one. If it hasn’t run the migration before, Fluent executes it.

Fluent will never run migrations more than once. Doing so would cause conflicts with the existing data in the database. For example, imagine you have a migration that creates a table for your users. The first time Fluent runs the migration, it creates the table. It it tries to run it again a table with the name would already exist, causing an error.

It’s important to remember this. If you change an existing migration, Fluent will not execute it. You need to reset your database as you did in the earlier chapters.

Modifying tables

Modifying an existing database is always a risky business. You already have data you don’t want to lose, so deleting the whole database is not a viable solution. At the same time, you can’t simply add or remove a property in an existing table since all the data is entangled in one big web of connections and relations.

Instead, you introduce your modifications using Vapor’s Migration protocol. This allows you to cautiously introduce your modifications while still having a revert option should they not work as expected.

Modifying your production database is always a delicate procedure. You must make sure to test any modifications properly before rolling them out in production. If you have a lot of important data, it’s a good idea to take a backup before modifying your database.

To keep your code clean and make it easy to view the changes in chronological order, each migration should have its own file. For file names, use a consistent and helpful naming scheme, for example: YY-MM-DD-FriendlyName.swift. This allows you to see the versions of your database at a glance.

Writing migrations

A Migration is generally written as a struct when it’s used to update an existing model. This struct must, of course, conform to Migration. Migration requires you to provide two things:

func prepare(on database: Database) -> EventLoopFuture<Void>

func revert(on database: Database) -> EventLoopFuture<Void>

Prepare method

Migrations require a database connection to work correctly as they must be able to query the MigrationLog model. If the MigrationLog is not accessible, the migration will fail and, in the worst case, break your application. prepare(on:) contains the migration’s changes to the database. It’s usually one of two options:

• Creating a new table
• Modifying an existing table by adding a new property.

Here’s an example that adds a new model to the database:

func prepare(on database: Database) -> EventLoopFuture<Void> {
  // 1
  database.schema("testUsers")
    // 2
    .id()
    .field("name", .string, .required)
    // 3
    .create()
}

1. You specify the schema — or table name — to run the migration on.
2. You specify the modifications to perform on the table. You can specify actions for constraints, fields and foreign keys. This includes marking fields as unique. For fields, you specify the field name, type and any constraints.
3. You specify the action to perform and the model to use. If you’re adding a new table to the database, such as creating a new Model, you use create(). If you’re adding a field to an existing Model type, you use update(). This example uses create() to create a new model with the fields id and name.

Revert method

revert(on:) is the opposite of prepare(on:). Its job is to undo whatever prepare(on:) did. If you use create() in prepare(on:), you use delete() in revert(on:). If you use update() to add a field, you also use it in revert(on:) to remove the field with deleteField(_:).

Here’s an example that pairs with the prepare(on:) you saw earlier:

func revert(on database: Database) -> EventLoopFuture<Void> {
  database.schema("testUsers").delete()
}

Again, you specify the schema to revert and the action to perform. Since you used create() to add the model, you use delete() here.

This method executes when you boot your app with the --revert option.

Note: Fluent will delete only the previous batch of migrations to avoid causing conflicts with old data. When changing a database, including removing fields that you previously added, you should try and “fix forward”. This means creating a new migration to remove the field you added in a previous migration.

FieldKeys

In Vapor 3, Fluent inferred most of the table information for you. This included the column types and the names of the columns. This worked well for small apps such as the TIL app. However, as projects grow, they make more and more changes. Removing fields and changing names of columns was difficult because the columns no longer matched the model. Fluent 4 makes migrations a lot more flexible by requiring you to provide the names of fields and schemas.

However, this means you end up duplicating strings throughout your app, a technique which is prone to mistakes. You can define your own FieldKeys to work around this. In Xcode, open CreateAcronym.swift and add the following at the bottom of the file:

extension Acronym {
  // 1
  enum v20210114 {
    // 2
    static let schemaName = "acronyms"
    // 3
    static let id = FieldKey(stringLiteral: "id")
    static let short = FieldKey(stringLiteral: "short")
    static let long = FieldKey(stringLiteral: "long")
    static let userID = FieldKey(stringLiteral: "userID")
  }
}

Here’s what this new code does:

1. Define an enum in an extension for Acronym. You name the enum with the date you created the extension. This makes it easy to see when you defined columns and when things changed.
2. Define a static property for the name of the schema. This is useful in case you change the table name in the future.
3. Define a FieldKey for each of the columns in the table. You use these in your Migration and Model.

Next, replace the body of CreateAcronym with the following:

func prepare(on database: Database) -> EventLoopFuture<Void> {
  database.schema(Acronym.v20210114.schemaName)
    .id()
    .field(Acronym.v20210114.short, .string, .required)
    .field(Acronym.v20210114.long, .string, .required)
    .field(
      Acronym.v20210114.userID, 
      .uuid, 
      .required, 
      .references(User.v20210113.schemaName, User.v20210113.id))
    .create()
}
  
func revert(on database: Database) -> EventLoopFuture<Void> {
  database.schema(Acronym.v20210114.schemaName).delete()
}

This replaces all the strings in your migration with the keys defined earlier. The reference to User also uses keys from the User migration already defined in the starter project.

Next, open Acronym.swift and replace.

static let schema = "acronyms"

with the following:

static let schema = Acronym.v20210114.schemaName

Next, replace the properties and property wrappers for short, long and user with the following:

@Field(key: Acronym.v20210114.short)
var short: String

@Field(key: Acronym.v20210114.long)
var long: String

@Parent(key: Acronym.v20210114.userID)
var user: User

This replaces the keys for the property wrappers with the FieldKeys you defined in CreateAcronym.swift.

Finally, open CreateAcronymCategoryPivot.swift. Replace:

.field(
  AcronymCategoryPivot.v20210113.acronymID,
  .uuid,
  .required,
  .references("acronyms", "id", onDelete: .cascade))

with the following:

.field(
  AcronymCategoryPivot.v20210113.acronymID, 
  .uuid, 
  .required, 
  .references(
    Acronym.v20210114.schemaName, 
    Acronym.v20210114.id, 
    onDelete: .cascade))

This replaces the strings with the FieldKey and schemaName you defined earlier. Now you have no more strings in your migration or model! This provides type safety to your migrations and makes it simple to change and update fields.

Adding users’ Twitter handles

To demonstrate the migration process for an existing database, you’re going to add support for collecting and storing users’ Twitter handles. In Xcode, create a new file called 21-01-14-AddTwitterToUser.swift in Sources/App/Migrations. This new file will hold the AddTwitterToUser migration.

Next, open CreateUser.swift. In the extension for User, add the following below v20210113:

enum v20210114 {
  static let twitterURL = FieldKey(stringLiteral: "twitterURL")
}

This adds a new FieldKey for the new property. Next, open User.swift and add the following property to User below var acronyms: [Acronym]:

@OptionalField(key: User.v20210114.twitterURL)
var twitterURL: String?

This adds the property of type String? to the model. You declare it as an optional string since your existing users don’t have the property and future users don’t necessarily have a Twitter account. You annotate the property with @OptionalField to tell Fluent the property is an optional field in the database.

Finally, replace the initializer with the following:

init(
  id: UUID? = nil,
  name: String,
  username: String,
  password: String,
  twitterURL: String? = nil
) {
  self.name = name
  self.username = username
  self.password = password
  self.twitterURL = twitterURL
}

This adds the twitterURL parameter to the initializer and provides a default nil value if it’s not provided.

Creating the migration

Open 21-01-14-AddTwitterToUser.swift and add the following to create a migration that adds the new twitterURL field to the model:

import Fluent

// 1
struct AddTwitterURLToUser: Migration {
  // 2
  func prepare(on database: Database) -> EventLoopFuture<Void> {
    // 3
    database.schema(User.v20210113.schemaName)
      // 4
      .field(User.v20210114.twitterURL, .string)
      // 5
      .update()
  }

  // 6
  func revert(on database: Database) -> EventLoopFuture<Void> {
    // 7
    database.schema(User.v20210113.schemaName)
      // 8
      .deleteField(User.v20210114.twitterURL)
      // 9
      .update()
  }
}

Here’s what this does:

1. Define a new type, AddTwitterURLToUser, that conforms to Migration.
2. Define the required prepare(on:).
3. Select the User table using the schema defined.
4. Add the new field with field(_:_) using the FieldKey defined earlier. Set the type to string.
5. Call update() to execute the migration and update the table.
6. Define the required revert(on:).
7. Select the User table using the schema defined.
8. Delete the field defined by the FieldKey earlier.
9. Call update() to execute the migration and remove the field.

Now open configure.swift and register AddTwitterURLToUser as one of the migrations.

Since Fluent executes migrations in order, it must be after the existing migrations in the list. However, since CreateAdminUser creates a new user you must add the migration before. Otherwise, when using a fresh database, CreateAdminUser fails. Add the following before app.migrations.add(CreateAdminUser()):

app.migrations.add(AddTwitterURLToUser())

The next time you launch the app, Fluent adds the new property to User. Build and run your application; you’ll see the new property in your table.

On your development machine, you can see the table’s properties by entering the following in Terminal:

docker exec -it postgres psql -U vapor_username vapor_database
\d "users"
\q

Versioning the API

You’ve changed the model to include the user’s Twitter handle, but you haven’t altered the existing API. While you could simply update the API to include the Twitter handle, this might break existing consumers of your API. Instead, you can create a new API version to return users with their Twitter handles.

To do this, first open User.swift and add following definition after Public:

final class PublicV2: Content {
  var id: UUID?
  var name: String
  var username: String
  var twitterURL: String?

  init(id: UUID?, 
       name: String, 
       username: String, 
       twitterURL: String? = nil) {
    self.id = id
    self.name = name
    self.username = username
    self.twitterURL = twitterURL
  }
}

This creates a new PublicV2 class that includes the twitterURL. Next, create the four convert methods for the version 2 API. Add the following to the extension for User after convertToPublic():

func convertToPublicV2() -> User.PublicV2 {
  return User.PublicV2(
  	id: id, 
  	name: name, 
  	username: username, 
  	twitterURL: twitterURL)
}

Now, add the following to the extension for EventLoopFuture where Value: User after convertToPublic():

func convertToPublicV2() -> EventLoopFuture<User.PublicV2> {
  return self.map { user in
    return user.convertToPublicV2()
  }
}

Then, add the following to the extension for Collection after convertToPublic():

func convertToPublicV2() -> [User.PublicV2] {
  return self.map { $0.convertToPublicV2() }
}

Finally, add the following to the extension for EventLoopFuture where Value == Array<User> after convertToPublic():

func convertToPublicV2() -> EventLoopFuture<[User.PublicV2]> {
  return self.map { $0.convertToPublicV2() }
}

This allows you to convert your Fluent model to PublicV2 in all the instances you may want to. Open UsersController.swift and add the following after getHandler(_:):

// 1
func getV2Handler(_ req: Request) 
    -> EventLoopFuture<User.PublicV2> {
  // 2
  User.find(req.parameters.get("userID"), on: req.db)
    .unwrap(or: Abort(.notFound))
    .convertToPublicV2()
}

This method is just like getHandler(_:) with two changes:

1. Return a User.PublicV2.
2. Call convertToPublicV2() to produce the correct return item.

Finally, add the following at the end of boot(routes:):

// API Version 2 Routes
// 1
let usersV2Route = routes.grouped("api", "v2", "users")
// 2
usersV2Route.get(":userID", use: getV2Handler)

Here’s what this does:

1. Add a new API group that will resolve on /api/v2/users.
2. Connect GET requests for /api/v2/users/<USER_ID> to getV2Handler().

Now you have a new endpoint to get a user, with a v2 in the API, that returns the twitterURL.

Note: For a more complicated API revision, you should create new controllers to handle the new API version. This will simplify how you reason about the code and make it easier to maintain.

Updating the web site

Your app now has all it needs to store a user’s Twitter handle and the API is complete. You need to update the web site to allow a new user to provide a Twitter address during the registration process.

Open register.leaf and add the following after the form group for name:

<div class="form-group">
  <label for="twitterURL">Twitter handle</label>
  <input type="text" name="twitterURL" class="form-control"
   id="twitterURL"/>
</div>

This adds a field for the Twitter handle on the registration form. Next, open user.leaf and replace <h2>#(user.username)</h2> with the following:

<h2>#(user.username)
  #if(user.twitterURL):
  - @#(user.twitterURL)
  #endif
</h2>

This shows the Twitter handle, if it exists, on the user information page. Finally, open WebsiteController.swift and add the following to the end of RegisterData:

let twitterURL: String?

This allows your form handler to access the Twitter information sent from the browser. In registerPostHandler(_:data:), replace

let user = User(
  name: data.name,
  username: data.username,
  password: password)

With:

var twitterURL: String?
if let twitter = data.twitterURL,
   !twitter.isEmpty {
    twitterURL = twitter
}
let user = User(
  name: data.name,
  username: data.username,
  password: password,
  twitterURL: twitterURL)

If the user doesn’t provide a Twitter handle, you want to store nil rather than an empty string in the database.

Build and run. Visit http://localhost:8080/ in your browser and register a new user, providing a Twitter handle. Visit the user’s information page to see the results of your handiwork!

Making categories unique

Just as you’ve required usernames to be unique, you really want category names to be unique as well. Everything you’ve done so far to implement categories has made it impossible to create duplicates, but you’d like that enforced in the database as well. It’s time to create a Migration that guarantees duplicate category names can’t be inserted in the database.

First, create a new file inside the Migrations directory called 21-01-14-MakeCategoriesUnique.swift. Open the new file and enter the following:

import Fluent

// 1
struct MakeCategoriesUnique: Migration {
  // 2
  func prepare(on database: Database) -> EventLoopFuture<Void> {
    // 3
    database.schema(Category.v20210113.schemaName)
      // 4
      .unique(on: Category.v20210113.name)
      // 5
      .update()
  }

  // 6
  func revert(on database: Database) -> EventLoopFuture<Void> {
    // 7
    database.schema(Category.v20210113.schemaName)
      // 8
      .deleteUnique(on: Category.v20210113.name)
      // 9
      .update()
  }
}

1. Define a new type, MakeCategoriesUnique, that conforms to Migration.
2. Define the required prepare(on:).
3. Select the Category schema to tell Fluent to change the table for categories.
4. Use unique(on:) to add a new unique index corresponding to the key for name.
5. Since Category already exists in your database, use update() to modify the database.
6. Define the required revert(on:).
7. Select the Category schema to tell Fluent to change the table for categories.
8. Use deleteUnique(on:) to remove the index corresponding to the key for name.
9. Since Category already exists in your database, use update() to modify the database.

Finally, open configure.swift and register MakeCategoriesUnique as one of the migrations. Add the following after app.migrations.add(CreateAdminUser()):

app.migrations.add(MakeCategoriesUnique())

Build and run; observe the new migration in the console:

Seeding based on environment

In Chapter 18, “API Authentication, Part 1,” you seeded an admin user in your database. As mentioned there, you should never use “password” as your admin password. But, it’s easier when you’re still developing and just need a dummy account for testing locally. One way to ensure you don’t add this user in production is to detect your environment before adding the migration. In configure.swift replace:

app.migrations.add(CreateAdminUser())

With the following:

switch app.environment {
case .development, .testing:
  app.migrations.add(CreateAdminUser())
default:
  break
}

Now the AdminUser is only added to the migrations if the application is in either the development (the default) or testing environment. If the environment is production, the migration won’t happen. Of course, you still want to have an admin in your production environment that has a random password. In that case, you can switch on the environment inside AdminUser or you can create two versions, one for development and one for production.

Where to go from here?

In this chapter, you learned how to modify your database, after your app enters production, using migrations. You saw how to add an extra property — twitterUrl — to User, how to revert this update and how to enforce uniqueness of category names. Finally, you saw how to switch on your environment in configure.swift, allowing you to exclude migrations from the production environment.

You can learn more about migrations in the Vapor documentation at https://docs.vapor.codes/4.0/fluent/migration/.