12. Training a Model for Sequence Classification

Written by Chris LaPollo

In the previous chapter, you learned about collecting and analyzing sequences of data, both crucial parts of successfully using machine learning. This chapter introduces a new type of neural network specifically designed for sequential data, and you’ll build one to classify the data you collected as device motions.

If you’re jumping into this chapter without first having gone through the previous one, you’ll need a Python environment with access to Turi Create. We’ll assume you have one named turienv, but if you don’t then you can create it now using the file at projects/notebooks/turienv.yaml. If you’re unsure how to do so, refer back to Chapter 4, “Getting Started with Python & Turi Create.”

Creating a model

You’ve got access to a clean dataset — either the one you made in the previous chapter or one we’ll provide for you — and now you’re ready to train a model. Or maybe several models until you find one that works well. This section shows how to use Turi Create’s task-focused API to train a model for activity detection.

Note: Training your own model is highly recommended, especially if you collected data to add to the provided dataset. But if for whatever reason you skipped the previous chapter, you can find a trained model named GestureClassifier.mlmodel inside the notebooks/pre-trained subfolder of the chapter resources required for this chapter.

In this section you’ll continue working with Jupyter in your turienv Anaconda environment. Create a new notebook in the notebooks folder of the chapter resources. If you’d like to see how we trained our provided model, you can check out the completed notebook notebooks/Model_Training_Complete.ipynb.

Import the same packages as you used in the previous chapter’s notebook:

import turicreate as tc
import activity_detector_utils as utils

Then run the following code to load your training, validation and testing datasets:

train_sf = tc.SFrame("data/cleaned_train_sframe")
valid_sf = tc.SFrame("data/cleaned_valid_sframe")
test_sf = tc.SFrame("data/cleaned_test_sframe")

As mentioned in the previous chapter, Turi Create stores structured data in SFrame objects. There are various ways to create such objects — here you load them directly from the binary files you previously saved. If you’d prefer to use the files supplied with the resources, change the paths to pre-trained/data/cleaned_train_sframe, pre-trained/data/cleaned_valid_sframe and pre-trained/data/cleaned_test_sframe.

Training any classifier involves using multiple datasets for training, validation and testing. But dealing with sequences includes a few wrinkles that require some explanation.

Splitting sequential data

If you’ve ever trained an image classifier, you may have divided the images into training, validation and test sets randomly. Or maybe those sets were provided for you, in which case someone else divided them randomly.

This works because each image is its own sample — no one image relates any more or less to any other image. (See the upcoming Note for an important caveat to this statement.) But the very nature of sequences is that samples do relate to each other. Order and grouping both matter — that’s what makes them sequences! For example, if you’re counting by twos — two, four, six, eight — and then randomly shuffle that data — eight, two, six, four — you’ve lost the sequence and now the data is meaningless. Or worse, you may have accidentally reordered them into a sequence with a different meaning — eight, six, four, two — now the sequence counts down by twos!

So the first rule for training with sequences: keep samples related to individual sequences grouped together and in order. Any shuffling or sampling you do should take into account these groupings.

Note: There can be situations where relationships exist between images in datasets meant to train classifiers, but those usually indicate mistakes that you should try to avoid. For example, if some images are identical or nearly so, as is common when dealing with large numbers of images, then having some in the training set and some in the validation or test sets may mislead you into thinking your model generalizes better than it does. It’s a tricky situation, because sometimes your in-production model will encounter examples that are nearly identical to those it saw while training. For example, consider a model meant to identify product images from the internet — it’s unlikely that you’ll manage to create a good training set without also including some of the very images its meant to recognize. But in general, do your best to keep training and test sets as separate as possible, while realizing there are going to be times when some similarity sneaks in.

There’s a second potential concern, relating to the sources of the sequences. Consider the case you’ve been working on throughout this chapter — gesture recognition. There’s certainly some variation each time you perform a gesture — after all, the app collects several floating point values from multiple sensors, many times per second, so its basically impossible to get two identical recordings. However, identical isn’t the same as really similar.

Different recordings of one person making a gesture are going to be similar to each other. That’s not entirely bad — it’s that similarity you want the model to recognize. You may even find it’s fairly easy to train a model that recognizes gestures from a specific person — it may not even require many training examples. But it might not work as well when you use the model with someone else.

That’s because recordings from one person are more similar to each other than they are to recordings from someone else. For example, the following plots show some data from two people performing the same actions — step up exercises:

These two plots show similar values for several features, but some features are quite different between users. A model trained on data from one of these users might have trouble recognizing the activity when presented with data from the other — and the more data you show your model from one user, the more different the other user’s data will seem. It’s certainly good to collect lots of data from each source, but it’s more important to collect data from lots of sources.

So if you have the choice of getting 1000 recordings from one person, versus 100 recordings from 10, the second will probably produce a better model. And 10 recordings’ from 100 people would probably be even better. By all means, get more data from each person if you can, but definitely try to collect data from as many people as possible.

And that’s the second rule for training with sequences, or really any data where the data’s source affects its features: use data from as many sources as possible. The more sources you have in your training set, the better your model should generalize to unseen examples.

But even if you have a great dataset chock full of examples from many different people, there’s another issue — how best to split it up into train, validation and test sets?

You might be tempted to split the data randomly (keeping in mind the earlier rule about maintaining samples as sequences, of course). However, you should avoid this. Remember how different recordings from the same person are similar to each other? Well, if you train with data from one person, then test with different data from the same person, your model may appear to perform better than it really does. That’s because it essentially trained on some of the test data.

So the third rule for training with sequences: don’t split your datasets by sequence, split them by source. Make sure you know the source of each of your data samples, and try to put all the data from any particular source into the same set: train, validation or test.

Note: Those last two problems occur with more than just sequences. Many data types are affected by their sources. For example, sensors from different phones will report slightly different values in the same situations, camera lenses have slightly different distortions, and so on. All physical devices are produced with some variance, so data collected from different devices can be slightly different even when measuring the same thing. In these cases, the same rules apply: try to train with as many different sources as possible, and try to test on data collected from multiple sources. Unless of course the model is meant to work with a specific source — such as correcting lense distortion for images from a specific camera. Then by all means test on data collected from the same source to ensure your model works correctly in its intended production environment.

But sometimes…

And now, in a shocking plot twist, you’re about to be told to sometimes do what you were just told not to do — train and validate on data from the same people! What?!

Real talk: There are going to be times — maybe most of the time — when you won’t have as much data as you want. In those cases, you can stretch your dataset out a bit by starting with just two datasets — training and test — and then grabbing a chunk of your training set to use for validation.

Depending on how many different sources are present in your training set, you might not be able to follow the recommended procedure of separating based on source. For example, the one provided with the chapter contains data from just two people. You’d lose too much training data if you separated these users, so you’d need to accept training and validating on data from both of them.

It’s not ideal — your validation accuracy will be artificially closer to your training accuracy because the two datasets are more similar, making it harder to tell if your model overfits. But if there’s enough variety in your training set to start, then this still works fairly well.

To help split your training data, Turi Create provides a nice utility function that divides an SFrame randomly into two smaller SFrames, while still maintaining proper sequence groupings. The following code demonstrates how to use it:

train, valid = tc.activity_classifier.util.random_split_by_session(
  train_sf, session_id='sessionId', fraction=0.9)

This uses Turi Create’s activity_classifier.util.random_split_by_session function with a training set, telling it which column name identifies the sessions, and what percentage of the data should be used in the first split. It returns two SFrames, the first will contain the given percentage of the original SFrame’s sessions, and the second will contain the remaining sessions.

After running this code, train would contain about 90% of the sessions and valid would contain the other 10%. You would then use these two SFrames for your training and validation sets.

The most important thing about this function is that it splits data based on session IDs, which means it keeps sequences organized together. Any samples with the same session ID are kept together and in order, but any particular session ID could end up in the training or the validation set.

The results of this call are not necessarily going to give you a perfectly balanced split. For example, here are the results of calling utils.count_activities with train and valid from one sample run:

That’s probably fine, but if you see a particularly bad split — especially when you know the original data was well balanced — then you should try splitting it again.

If you want to experiment later, try combining the training and validation data and then use this function to randomly split it. You’ll end up with more variety in your training data in exchange for a less trustworthy validation set. For now, you’ll just use the separate datasets you’ve already built.

Training the model

Now it’s time to build and train your model. Almost.

Whenever you train with a new model or dataset, it’s good to first take a small portion of your training data and see if you can get the model to overfit it. Overfitting is usually a bad thing — it means your model is memorizing the training data instead of learning a more general solution — but it also shows that the model is actually capable of learning something from your data. If your model is going to work on a real dataset, then it should definitely be able to overfit on a tiny version of it. And if it can’t, then you’ve got one of several problems you’ll need to address:

• A bug in the model. This is especially common when implementing neural nets from scratch using frameworks such as Keras.
• A model too simple to solve the problem. You might need more layers, or more nodes per layer.
• A model architecture incapable of solving the problem. Different architectures work better for different problems, so pick something appropriate.
• Poorly tuned hyperparameters. Sometimes all it takes is a change to the learning rate, other times you might need different activation functions, optimization algorithms or loss functions.
• Maybe the problem is the problem itself. Machine learning isn’t the right solution to every problem, so don’t try to force it.

The point of this exercise is to prove to yourself that your dataset is applicable to the problem, your model is built correctly and it’s tuned well enough to learn. You’ll still usually have to do more tuning later with your full dataset, but those training sessions take longer. This step is critical to keep yourself from wasting time trying to tune a model that isn’t ever going to work.

To save space we don’t show the results of the overfitting step here, but you can find them in the notebook Model_Training_Complete.ipynb in the notebooks folder.

OK, now it’s time to build and train your model. Turi Create’s activity classification API makes this process easy — it just takes one function call! Add the following code to a notebook cell, but don’t run it yet:

model = tc.activity_classifier.create(
  dataset=train_sf, session_id='sessionId', target='activity',
  features=[
    "rotX", "rotY", "rotZ", "accelX", "accelY", "accelZ"],
  prediction_window=20, validation_set=valid_sf,
  max_iterations=20)

This one line of code is doing a lot, so it warrants quite a bit of explanation. Here goes:

• dataset: Your training dataset, stored as an SFrame.
• session_id: The name of the column in dataset that stores the session ID associated with each row. create keeps data with the same session ID grouped together and in order, and then trains over it in chunks the size of prediction_window rows.
• target: The name of the column that contains the labels you want the model to predict. In this case, it’s activity.
• features: This is an optional list of columns to use for training. If you don’t supply it, then create uses all the columns as features except for the ones you specified for session_id and target. More on this in a bit.
• prediction_window: How many samples (i.e. rows of data) the model looks at to make a prediction. More on this later.
• validation_set: Your validation dataset, stored as an SFrame. This is optional — if you don’t supply it, and dataset contains more than 100 sessions, then create will automatically make a validation set by randomly selecting sessions from dataset. But if it contains fewer sessions than that, create trains the model without a validation set. It’s best not to rely on this logic, and supply your own data instead.
• max_iterations: The maximum number of epochs create will train over. That is, the number of times it will go through the training set. Note: the parameter name and documentation claim this is a “maximum,” as if create could stop training sooner. However, there appears to be no evidence that training ever stops before this value is reached, so think of it as the actual number instead of a maximum.

Notice how the features parameter is a list including just six of the 12 motion features available in your dataset — the rotation and acceleration due to the user.

These were chosen a bit arbitrarily, mostly to show that you don’t need to use all columns in your dataset. In the previous section you saw how each activity appeared with a distinct pattern. But take a look at the following plots, which show just user rotation values for samples of each activity:

As you can see, there are still clearly visible patterns, even when using just these three features. You are encouraged to train models with different feature combinations to see if/how it effects the results. There is no one correct answer here — many combinations will produce usable models for this project.

Note: For any specific problem, there is likely some minimum set of features necessary to train a good model. It just needs enough information to perceive differences between the classes, and different features may be more or less useful for each class. The final set of features you settle on will always be project dependant, but when in doubt — use more. That gives your model the most leeway to decide for itself.

The prediction window is an important aspect of Turi Create’s activity classification model. It specifies how many samples the model needs to look at each time it makes a prediction. That means this value — combined with Core Motion’s update interval — determines the amount of time each prediction represents.

For example, if the prediction window is 50 and Core Motion sends the app 10 updates per second, it will take five seconds to collect enough data to make one prediction. But if you’re getting updates 100 times per second, it would take 0.5 seconds. As was mentioned earlier, be sure you train your model with a prediction window that makes sense for the update rate you are using. You collected data at 25 samples per second, so this window size of 20 means the model needs 0.8 seconds worth of data per prediction.

The prediction window suggested here works well with the provided dataset and satisfies our goals for the book. However, you should train multiple models using different window sizes to see what you think works best. You won’t really know if you’re satisfied until you use the model in its target environment — in this case, the game you’ll make in the next chapter. There’s no one “correct” size — it’s based on the specific use case, the dataset, and a bit of personal preference. Traditional software developers often struggle with this aspect of machine learning more than any other — you can’t usually sit down and just write the “solution” to a problem; it’s more about running lots of experiments until you discover what works best for your specific use case.

OK, now create your model by running the cell with your call to create. The first output you’ll see will be something like this:

Here’s a rundown of what this tells you:

• The first line reports how many samples — individual rows — are in the training dataset. The function performs some pre-processing on the data, including chunking it into fixed-length sequences.

• The second line let’s you know Turi Create will be training on sequences of 400 samples. That’s because you’re training with a prediction window of 20 samples, and Turi Create’s underlying implementation always trains in chunks of 20 consecutive windows. If a session doesn’t have enough samples available, the end of the sequence gets padded with zeros.

  This is why you shouldn’t have very short recording sessions — tiny sessions result in too much padding and the model will have trouble learning.

• Finally, it reports the number of sessions in your training dataset. This matters most when you don’t supply a validation set, because create will use some of these sessions for validation if there are more than 100 sessions available.

After that, you’ll see updates appear for each training iteration — or epoch — as your model continues to train. You need to check these updates for signs of overfitting. If the training accuracy continues to improve but the validation accuracy stalls or begins to decline, then the model has begun to overfit.

The output for the provided model isn’t included here, but you can see it in notebooks/Model_Training_Complete.ipynb. It actually overfits slightly, but we decided to stick with that model anyway for a couple reasons.

First, it’s good to show to readers as an example of overfitting. And secondly, the difference between the final model’s validation accuracy and the epoch with the best value was only about 1%. The validation set is very small, with only four recordings of each gesture from each of two users.

A 1% difference in accuracy in such a small dataset really isn’t significant enough to prove anything about the model’s expected real world performance — it might just be certain epochs arrived at weights that happened to work well with that particular validation set. This is why you should strive to get a lot of variety in your datasets by collecting data from many different sources.

The final epoch for the model that ships with the book as notebooks/pre-trained/GestureClassifier.mlmodel had a training accuracy of 98.5% and a validation accuracy of 95.2%. If you include training data collected from other sources, you’re likely to get lower training accuracy while getting higher validation accuracy. Don’t get hung up on the specific numbers, though — the idea is just to get something that looks like it trained well before moving on to testing with your test set.

Note: Turi Create is great, as it builds and trains sophisticated models without you needing to do much more than provide the data. However, that comes at the cost of flexibility. There isn’t much you can do here to tweak your model’s performance.

Besides changing your dataset, you can also try different prediction windows, feature combinations, batch sizes (not discussed here — you just used the default), and number of epochs.

If none of that leads to a model suited to your app, then you’ll need to build something customized in a more flexible framework like Keras. You’ll work with sequences in Keras in later chapters.

When you think the model’s ready for testing, go ahead and run the following code:

metrics = model.evaluate(test_sf)
print(metrics['accuracy'])

You use the model’s evaluate method to classify everything in your test set and gather the results inside a dictionary named metrics. You’ve also displayed the accuracy the model achieved with those classifications, which for the provided dataset should be in the very high 90s — the model included with the chapter resources scores over 97%.

Accuracy isn’t everything, though. You have access to various other results, including precision, recall, a confusion matrix, and more. You can access each of these by name, like you did with accuracy. To see a quick rundown, just print the entire metrics object:

print(metrics)

The confusion matrix is particularly useful here. It lets you know not just whether or not your model was correct, but where it made mistakes. This lets you see if there’s a particular class that’s giving your model extra trouble. If so, you might need to tweak your datasets by gathering more data for the more difficult classes.

But you should also consider trying a different prediction window size, since sometimes models are better at recognizing different classes using different windows — your goal is to find the one that gives you the best overall performance.

Note: If you find different activities are only recognized at different window sizes, then you might need a more complicated setup using multiple models, each trained to spot a subset of your classes. You may know about “ensemble” methods already, where multiple models combine their predictions to produce a final answer. The technique required here is almost an ensemble, but it’s slightly more complex, because it requires extra logic in your app to ensure your different models predict on different schedules. That won’t be covered further in this book.

Here’s the confusion matrix for the model included with the book:

The first thing you might notice is the large numbers of predictions — your dataset didn’t have nearly that many gestures, did it? That’s because it’s providing a prediction for every window, not every activity. So it makes many predictions over any single activity sequence, and this shows the results for all of them.

Next, notice the predictions with the highest counts: They are all correct predictions, with over 5,000 for each of the gestures and over 13,000 for rest_it. On the other hand, each of the incorrect predictions happened only a small number of times, with the fewest being rest_it predicted as drive_it only 17 times and the most being drive_it predicted as rest_it 184 times. Almost all of the errors involved the rest_it activity, which you would expect. After all, you never know what people did while recording their rest data — they may even have been doing the other gestures!

In fact, notice that the only incorrect predictions that did not involve the rest_it activity were the 176 chop_it gestures predicted as shake_it. It makes sense that there might be mistakes between these two gestures, since chopping is actually quite similar to shaking — if a person chops very quickly it might appear similar to a shake, or if the shake is over exaggerated it might look a bit like a chop.

Keep in mind, your model’s performance in your app may be better than its test results, because you’ll ignore low-confidence predictions. But if you’re still unhappy with the model, you should create another one. Some sticklers will tell you not to reuse your training data because you’ll be leaking data into your model. That’s technically true, and you should listen to them… except you probably won’t. Unless you have an endless stream of free data available, you probably don’t have the luxury of testing just once per test set. The good news is — in many cases that’s probably OK. For example, with a project like this one, you want the app to perform well, and your test data is just a tool to help you get there. Once your model works well on that, you’ll run it on actual devices with live data from real people. Those are your real tests, and they are always unique — so you can even tell those sticklers you’re using a new test set each time! Metrics like test accuracy are great, but be sure to beta test your app with many people before releasing it, so you know it really works the way you want it to.

When you think your model is ready for testing on a device, go ahead and save it with the following code:

model.export_coreml("GestureClassifier.mlmodel")
model.save("GestureClassifier")

This exports it to Core ML for use in your app, and saves a copy that you can reload in Python in case you want to work more with it later.

You’ve saved your model, trained to analyze iPhone motion data and recognize when that data indicates specific gestures have occurred. It seems to perform well, at least when tested against recorded motion data. That’s a good start, but you want it to work in real time, evaluating motion data as it’s produced on the device. For that you need an app! Continue reading to learn how to build one.

Getting to know your model

Open the GestureIt starter project in Xcode. If you’ve gone through the chapters leading up to this one, then you’ve already practiced adding Core ML models to your projects — find the GestureClassifier.mlmodel file you created when you saved your trained model in the previous section and drag it into Xcode. Or, if you’d like to use the model we trained on the provided dataset, add notebooks/pre-trained/GestureClassifier.mlmodel instead.

Note: Now that you have the model in Xcode, the rest of this section is all theory. You can safely skip it if you aren’t interested in this discussion right now.

Select GestureClassifier.mlmodel in the Project Navigator and you’ll see the following, which is similar to — but also quite different from — models from Section 1 of this book:

Here you can see GestureClassifier is an activity classifier from Turi Create. It’s under 1MB — that’s pretty good for a neural net that isn’t taking advantage of models pre-installed on iOS, as did some of the ones you made earlier. But then comes the Model Evaluation Parameters section, where things get a bit more complicated.

First, the more recognizable items:

• features: MLMultiArray of Doubles you’ll pass as input. If you haven’t seen MLMultiArray before, don’t worry, it’s nothing too new. It’s basically just a multidimensional array that Core ML uses to work efficiently with data.

This one is sized to store a single prediction window’s worth of values for each of the features you used while training: rotation and acceleration due to the user around the X, Y and Z axes.

• activityProbability: Dictionary the model outputs that includes the probabilities assigned to predictions for each of the classes. In the case of this project, that means probabilities for the gesture types “rest_it,” “shake_it,” etc.
• activity: String the model outputs indicating the activity class predicted with the highest probability.

But what about these other things: hiddenIn, cellIn, hiddenOut and cellOut? And what’s this mysterious new acronym “LSTM” mentioned in all their descriptions?

Recurrent neural networks

So far in this book you’ve mostly dealt with convolutional neural networks — CNNs. They’re great for recognizing spatial relationships in data, such as how differences in value between nearby pixels in a two-dimensional grid can indicate the presence of an edge in an image, and nearby edges in certain configurations can indicate the ear of a dog, etc. Another kind of network, called a recurrent neural network — RNN — is designed to recognize temporal relationships. Remember, a sequence generally implies the passage of time, so this really just means they recognize relationships between items in a sequence.

To do that, they look at sequences one item at a time, and produce an output for each item based on the current item and on the output they produced for the previous item. But what does that really mean?

Consider how you read the following sentence: “The quick brown fox jumps over the lazy dog.” You don’t look at each word individually and ignore the rest, right? Instead, each element of the sentence adds to your understanding. What’s happening? Jumping. Who’s jumping? The fox. What’s it look like? It’s brown. And so forth.

RNNs are designed to do something similar, interpreting each element in a sequence by considering the elements they’ve already seen.

So what’s that look like as a network? You may come across RNN diagrams like this one:

In the above image, the circle represents a single layer of an RNN, not a single node. Remember from what you learned earlier — a layer in a neural network can contain any number of nodes, with more nodes providing that layer with more representation power. Input elements in a sequence are referenced by timesteps, and layers process the element at time T by looking at both that input and the layer’s own output from the previous input at timestep T-1. That loop where the layer’s output feeds back into itself is known as a recurrent connection — i.e. it occurs repeatedly — giving RNNs their names.

While diagrams like that might be useful to describe the theory behind an RNN, it can be easier to visualize if you think of the network as multiple layers. Looked at this way, each successive layer receives the next element in the input sequence along with the output from the previous layer. The following image shows what that would look like when processing the earlier example sentence:

Now it’s clearer how the layers process a sequence one element at a time, combining each element of the input with the output generated for the previous element. Notice that the RNN cannot process a given item until after it has processed the items that came before it in the sequence. It’s this serial nature of RNNs that makes them slower than other neural networks, such as CNNs. This is true of both training and inference.

These recurrent connections allow RNN layers to adjust their output based on what they’ve seen so far in the sequence, much like you interpret the word “fish” in the following two sentences differently depending on the words before it: “I like to fish.” and “I like fish.” In the first sentence, the speaker likes to catch fish, or at least try to; in the second, the speaker probably likes to eat fish, but may also just enjoy fish as an animal in general. Either way, the definition of “fish” depends on its context.

Note: The previous diagram shows two outputs from each layer, one going to the next layer and one going off to… somewhere? That’s to indicate how the output for each timestep can be used within an RNN layer, through the recurrent connection, as well as passed along to the next, possibly also recurrent, layer of the network. The final output of an RNN layer can be either the output for the sequence’s last timestep, or the entire sequence of outputs the layer generated while processing the input sequence.

Early implementations of this basic RNN design showed it was possible to learn relationships across timesteps in a sequence, but they don’t actually do it very well. Due to how the underlying math works, they take too long to train and can’t relate items separated by too many timesteps. For example, imagine an RNN processing our example sentence — it would likely remember the fox is brown, but it might have forgotten there is a fox at all by the time it gets to the dog at the end of the sentence.

In reality a basic RNN could probably handle short sentences like that, but relationships span much greater distances in many sequences. To continue with our reading example, while words within a sentence are surely related to each other, they can also be related to words in sentences earlier in the same paragraph, many pages ago in the same chapter or even several chapters ago in a book. The distance between relationships can be arbitrarily long, and basic RNNs simply aren’t suited to handle that.

But then along came LSTMs.

Long short-term memory

The acronym LSTM stands for the odd-sounding phrase long short-term memory, and it refers to a different kind of recurrent unit capable of dealing with relationships separated by longer distances in the sequence. Conceptually, the following diagram shows the pertinent details of how an LSTM works.

It uses our earlier sample sequence and shows the recurrent steps unrolled as separate layers to help clarify its behavior:

As you can see, an LSTM is a recurrent unit enhanced with an internal memory. LSTMs are used just like regular recurrent layers, but instead of processing only their input and previous output, an LSTM also considers the contents of its memory. And instead of just producing an output, the LSTM can also update its memory to remember (or forget) information it thinks is important about the sequence so far.

But terms like remembering, forgetting and thinking make it sound like LSTMs have more agency than they really do. Just like with other parts of a neural network, the LSTM’s “memory” is really just a bunch of numbers that get manipulated by various math functions. And it doesn’t really choose to remember or forget, it just learns weights that cause it to react differently to different sequences.

Note: LSTM units are more complex than they appear in the above diagram, with each cell made up of four layers combined by various math operations. If you’re interested in their inner workings, check out this excellent blog post: https://bit.ly/2RSmm7B. But the truth is, unless you’re working to invent new types of neural network layers, you probably won’t need to know those low-level details.

The important thing to know about LSTMs is that they train much more easily than the basic RNNs that came before them, and they offer much better performance. Most RNNs in use today use some variation of the LSTM unit, as is the case with the activity classifier you trained in Turi Create.

Turi Create’s activity classifier

So far we’ve been discussing RNNs — and more specifically, LSTMs — as deep learning’s solution to working with sequences. But it turns out that’s not the whole story.

Many state of the art results have been achieved using other network types, especially our old friend the CNN.

Current research trends seem to be moving away from RNNs because they don’t scale with hardware as well as other models do. But for now, recurrent models are still a popular choice in practice.

What approach does Turi Create’s activity classifier take? It’s actually a combination of a CNN and an RNN. It uses convolutional layers to extract features from short sequences — the prediction windows mentioned earlier in the chapter — and it uses an LSTM layer to reason over sequences of predictions.

That lets it recognize sporadic activities, such as the gestures you trained your model to classify, as well as activities spanning longer periods of time, perhaps made up of several smaller ones. For example, imagine the following sequence of activities: putting a teabag in a cup, pouring hot water in a cup, waiting patiently, and removing a teabag from a cup.

Each of those individual activites might be recognizable from small sequences of data — like what you could provide in a single prediction window. But when that series of activities occurs over multiple prediction windows, then the model might be able to recognize the overarching activity — making a cup of tea.

The following diagram shows a high level overview of Turi Create’s activity classifier:

You provide a sequence of sensor data as input — one prediction window’s worth — and the model’s first layer treats each input feature as a separate channel and performs a one-dimensional convolution over them. A 1D convolution is just like the 2D convolutions you’ve already used, except it uses kernels that are vectors instead of matrices.

Each kernel is the length of the prediction window and gets applied to all the input features to produce a new output channel. The current version of the code applies 64 such kernels.

The convolutional layer in this diagram may seem confusing because it looks like the waves are two-dimensional, but these are actually just vectors with numbers in them that we are displaying as a 2D image. To display a vector in two dimensions, we treat each item’s index in the vector as its value along the X axis. That is, each item in the vector represents a feature value at a specific point in time.

Remember from the discussions on transfer learning earlier in the book, how the pre-trained CNN model extracts features from images and then the layers you train use those extracted features as inputs? This is basically what Turi Create’s model does, except the CNN isn’t pre-trained. The CNN layer learns to output a vector encapsulating any interesting temporal features found within the prediction window. For example, maybe it notices certain patterns of peaks and valleys that are helpful when identifying a shaking phone. These extracted features flow into the LSTM layer as if they were a single item in a sequence.

To understand why CNNs might be well suited to this task, it can help to think of this as a vision problem instead: Imagine you plotted the sensor data for a prediction window, similar to what we show in the previous diagram, and then passed that image to a CNN. If CNNs can learn to recognize dogs in images, they should be capable of learning to recognize patterns in sequences just like the ones you saw in the previous chapter when exploring the dataset.

After the LSTM layer receives the extracted features from the CNN layer, it produces an output based on those features combined with its own internal memory and its output from the previous prediction window. The LSTM’s output passes through fully connected layers with batch normalization and dropout, and finally a softmax layer that outputs probabilities for each of the classes the model knows about. You learned about all those layer types earlier in the book so they aren’t discussed here.

This talk about internal memory and previous predictions brings up an important question: What about when a sequence doesn’t relate to those that came before it? Data doesn’t always arrive as one long, unbroken stream, so do you really want your model to always consider its past predictions as part of the current sequence?

Well, that finally brings us back to those new items you saw in Xcode: hiddenIn, cellIn, hiddenOut and cellOut. The names may seem backwards, but hiddenOut is the output from the LSTM itself, while cellOut is the LSTM’s internal memory state after making the prediction. And hiddenIn and cellIn are the inputs you use to pass to the model those outputs from the previous prediction. Each of these is a vector of 200 Doubles stored as an MLMultiArray — you don’t need to worry about that, it’s just how the model’s LSTM layer encodes its state information.

So to indicate the start of a new sequence, you’ll pass nil to the model for both hiddenIn and cellIn. On the other hand, when the current prediction is picking up where the last one ended — as will often be the case with streaming motion data — you’ll take the hiddenOut and cellOut values from the previous prediction and pass those back to the model as hiddenIn and cellIn, respectively. Using the output and memory from the previous step like this allows the LSTM to recognize longer sequences.

Continuing with our text example, it’s as if the first prediction window you pass is for the word “The,” the next window is for “quick,” then “brown” and so on.

This whole chapter has been talking about classifying sequences of sensor data, but it turns out the model you made with Turi Create is looking at its inputs in two different ways — as sequences of sensor data, and as sequences of sequences of sensor data. The prediction window contains enough information to classify the first kind of sequence, but these extra inputs and outputs allow the LSTM portion of the network to reason over longer periods of time to classify the second kind of sequence.

While models combining CNNs and LSTMs have achieved state-of-the-art results for tasks such as activity detection and speech recognition, there are also other techniques that deliver excellent performance when working with sequences. These include: Attention — a sort of memory added to other networks that helps guide their focus; Transformers — networks that use attention exclusively instead of recurrent or convolutional layers; and Temporal Convolutional Networks — CNNs designed for processing sequences. And new research seems to appear on a weekly basis, so there may be even more options by the time you’re reading this. You’ll read a bit more about some of these in later chapters.

A note on sequence classification

In the previous section you learned about the model architecture of Turi Create’s activity classifier. Recall how the final layer had a node for each class the model recognizes, with a softmax activation to produce a probability distribution over them.

We didn’t underscore it there, but it’s important to realize that using neural networks to classify sequences works the same way as it does for other types of data. You build a network with whatever layers make sense for the problem — convolutions, LSTMs, etc. — and then a final layer of nodes — one for each possible output — with a softmax activation function to produce probabilities over them.

In fact, you can even use networks with this architecture to predict the next item in a sequence instead of the class of a sequence. The difference is that during training, instead of providing sequences as inputs and class labels as outputs, you give partial sequences as inputs and the next item in the sequence as the output. You’ll get to do this yourself in a later chapter about translating natural language.

Key points

• Turi Create’s activity classification API can help you easily make models capable of recognizing human activity from motion data. However, it can be used for more than just human activity detection — it’s basically a generic classifier for numeric sequences.
• Try isolating data from a single source into one of the train, validation or test sets.
• Prefer a balanced class representation. In cases where that’s not possible, evaluate your model with techniques other than accuracy, such as precision and recall.
• Sample/shuffle sequential data as full sequences, not as individual rows.
• First train on just a small portion of your training set and make sure you can get the model to overfit. That’s the best way to find problems with your model, because if it can’t overfit to a small dataset, then you likely need to make changes before it will be able to learn at all.
• Train multiple models and run multiple experiments until you find what works best for your app.
• RNNs process data serially, so they’re slower than CNNs, both when training and performing inference.
• One-dimensional convolutions are commonly used to extract temporal features from sequences prior to passing them into RNNs.
• RNNs are a good choice for sequential data, with LSTMs being the most commonly used variant because they train (relatively) easily and perform well. However, they are not the only models that work well for sequences.

Where to go from here?

You’ve collected some data and created a model. Now it’s time to actually use that model in an app — a game that recognizes player actions from device motion. When you’re ready, see you in the next chapter!