Vapor and Job Queues: Getting Started

In this Vapor tutorial, you’ll learn how to dispatch a job from your Vapor app to a queue hosted on Redis. You’ll also learn how to schedule jobs that you want to run in the future.

If you’ve ever gone to the supermarket or driven on the highway, chances are you’ve been in a real-life queue. With software, it’s not all that different. The program schedules a series of data to process, one after another.

Usually, this processing pattern follows a first-in-first-out (FIFO) principle.

In this tutorial, you’ll learn how to:

• Configure and run a queue
• Dispatch a job to your queue
• Create a scheduled job

Job queues in particular serve to coordinate asynchronous service-to-service communication. Generally, one process sends while another receives. Each interacts with the queue independent of the other. They likely aren’t even aware of each other’s existence!

Queues are especially used in serverless and microservice architectures. But even monolithic applications can see benefits from job queues, such as offloading CPU or disk-writing intensive tasks.

Here are some examples of tasks that queues could help with:

• Sending emails and text messages, as part of two-factor or email authentication, for example
• Performing complex or long-running database operations, such as importing and processing thousands of items into your database
• Speeding up response time by delaying non-critical processing
• Buffering or batching work when you operate on large data sets
• Smoothing spiky workloads during peak hours by offloading the whole request
• Ensuring increased job integrity and resilience by persisting until the jobs are finished

Getting Started

Download the starter project by clicking the Download Materials button at the top or bottom of this tutorial. Then, navigate to the starter folder.

• Xcode 11 and Swift 5.2 (or newer versions)
• Docker: If you don’t have Docker yet, visit Docker install for Mac.
• A REST client to run requests against your back end. This tutorial uses Paw, but Postman, RESTed, and even cURL work fine.

Looking at the Vapor Project

Open the Vapor app in Xcode by double-clicking the Package.swift.

While you wait for the Swift Package Manager (SwiftPM) to resolve dependencies, check out the existing project in /Sources/App:

File explorer for the starter project with highlight of key folders and files.

File explorer for the starter project with highlight of key folders and files.

You should pay attention to a few things:

1. Controllers: You’ll see one controller — NewsletterRecipientAPIController.swift. It contains the method to sign up for your newsletter.
2. Migrations: You’ll find the migrations for Newsletter and NewsletterRecipient here. You’ll also find a third migration — SeedNewsletter. It creates a new newsletter in your database.
3. Models: This contains the two database models, Newsletter and NewsletterRecipient.
4. configure.swift: Here’s where you’ll find everything required to get started with the tutorial. This includes Databases, Migrations and Routes. This also is where you’ll register your Queue and Jobs later.
5. routes.swift: You’ll need only one route for the project, which is to sign up for a newsletter. It’s registered under /api/newsletter/sign-up.

Now, you’re ready to begin the project!

Running the Vapor Project

After SwiftPM finishes resolving the project’s dependencies, set the Working Directory to your project folder.

In Xcode, go to the Newslettering run scheme.

Under Options, turn on Working Directory and select your project folder.

Set your build folder in Xcode before running the app.

Set your build folder in Xcode before running the app.

Close the Scheme Editor window and verify that your selected platform is My Mac. Then, build and run.

Vapor Queue Build and Run

Vapor Queue Build and Run

Now, switch to Paw (or your preferred REST Client) and prepare the following request:

POST http://localhost:8080/api/newsletter/sign-up
Content-Type: application/json

{
   "email": "test1@newslettering.com",
   "name": "Test User 1"
}

Next, send the request:

Newsletter sign-up request and response.

Newsletter sign-up request and response.

You should get a 201 Created response from the server. If you check the response body, you’ll see the email and name you sent in the request.

Getting Started with Vapor Queues

You’ll use the Vapor Queues package in the rest of the tutorial. It’s a job queue system that allows you to run jobs on request as well as schedule them to run in the future.

To take advantage of the Queues package, you’ll need a job storage driver. For this project, you’ll use Vapor’s QueuesRedisDriver.

Configuring Your QueuesRedisDriver

Open configure.swift and, as part of the list of packages to import, add the following:

import Queues
import QueuesRedisDriver

Here, you expose the Queues and QueuesRedisDriver APIs so you can configure them.

Next, configure your queue to run on Redis.

Inside configure(_:), after try Application.autoMigrate().wait(), add the following:

try app.queues.use(.redis(url: "redis://127.0.0.1:6379"))

This is where you register that the queue will be running on Redis with the predefined URL.

Now, let’s get your Redis server up and running.

Start Your Redis Server

Open your terminal and navigate to the project folder.

Start the Redis container by running:

$ docker-compose run -p 6379:6379 cache

This starts a Docker Redis container that is listening on port :6379 and exposes the port to your local machine.

Running Your Queue and Scheduled Jobs

There are two “modes” you can run your scheduled jobs as:

• Separate process
• In-app process

To run the queue as a separate process, run the following command in your terminal:

$ swift run Run queues

This command starts a separate worker in the background that listens for any dispatched jobs.

To run scheduled jobs in a separate process, pass the --scheduled flag to the same command above.

The second option is to run the queue as an in-app process, which is what you’ll do for the rest of the tutorial.

Inside configure.swift, right under where you configured the queue driver, add the following:

// 1
try app.queues.startInProcessJobs()
// 2
try app.queues.startScheduledJobs()

With this code, you:

1. Start the queue as an in-app process. You’ll run your jobs on the default queue, so you won’t need a queue name as an argument.
2. Start the scheduled jobs as an in-app process. Your app will then check for any scheduled jobs. If none exist, it’ll exit the worker again.

Now, build and run your app. You’ll see a warning in the debug editor before the server starts:

[ WARNING ] No scheduled jobs exist, exiting scheduled jobs worker.

Console output after you have registered the scheduled job

Console output after you have registered the scheduled job

You’ve learned how to run jobs. Now it’s time to dispatch them.