6. Taking Control of Training with Keras

Written by Matthijs Hollemans

In the previous chapters, you’ve learned how to train your own models using Create ML and Turi Create. These are user-friendly tools that are easy to get started with — you don’t really have to write a lot of code and they take care of most of the details. With just a few lines you can load your data, train your model and export to Core ML.

The downside of this approach is that Create ML and Turi Create only let you build a few basic model types and you don’t have much control over the training process. This is fine if you’re just getting your feet wet with machine learning. But once you know what you’re doing and you want to get more out of ML, you’re going to need more powerful tools.

In this chapter, you’ll use a popular deep learning tool called Keras to train the snacks classifier. Keras gives you much more control over the design of the models and how they are trained. Once you know your way around Keras, you’ll be able to build any kind of neural network you want.

Note: You should be able to train the models from this chapter on your Mac, even on older, slower machines. The models are small enough to be trained on the CPU and don’t need GPU acceleration — only a little patience.

Keras runs on top of a so-called backend that performs the actual computations. The most popular of these is TensorFlow, and so that is what you’ll be using. TensorFlow is currently the number one machine-learning tool in existence. However, it can be a little tricky to use due to its low-level nature. Keras makes using TensorFlow a lot easier.

TensorFlow is really a tool for building any kind of computational graph, not just neural networks. Instead of neural network layers, TensorFlow deals with rudimentary mathematical operations such as matrix multiplications and taking derivatives. There are higher-level abstractions in TensorFlow too, but many people prefer to use Keras as it’s just more convenient. In fact, Keras is so popular there is now a version of Keras built into TensorFlow.

Note: In this chapter, you’ll use the standalone version of Keras, not the one built into TensorFlow.

Getting started

First, you need to set up a Python environment for running Keras. The quickest way is to perform these commands from a Terminal window:

$ cd /path/to/chapter/resources
$ conda env create --file=starter/kerasenv.yaml
$ conda activate kerasenv
$ jupyter notebook

If you downloaded the snacks dataset for a previous chapter, copy or move it into the starter folder. Otherwise, double-click starter/snacks-download-link.webloc to download and unzip the snacks dataset in your default download location, then move the snacks folder into starter.

Note: In this book we’re using Keras version 2.2.4 and TensorFlow version 1.14. Keras, like many open source projects, changes often and sometimes new versions are incompatible with older ones. If you’re using a newer version of Keras and you get error messages, please install version 2.2.4 into your working environment. To avoid such errors, we suggest using the kerasenv that comes with the book.

Tip: If your computer runs Linux and has an NVIDIA GPU that supports CUDA, edit kerasenv.yaml and replace tensorflow=1.14 with tensorflow-gpu=1.14. Or if you have already created the environment, run pip install -U tensorflow-gpu==1.14. This will install the GPU version of TensorFlow, which runs a lot faster.

Back to basics with logistic regression

One of the key topics in this book is transfer learning: a logistic regression model is trained on top of features extracted from the training images. In the case of Create ML, the features were extracted by the very powerful “Vision FeaturePrint.Scene” neural network that is built into iOS 12. In the case of Turi Create, the feature extractor you used was the somewhat less powerful SqueezeNet.

The big advantage of transfer learning is that it is much quicker than training from scratch, because your model can take advantage of the knowledge that is already contained in the pre-trained feature extractor. Hence, you are transferring knowledge from one problem domain to another. In this case, the feature extractors are trained on the general problem of recognizing objects in photos, and you’ll adapt them to the specific problem of recognizing 20 different types of snacks.

We also claimed that this approach of using a feature extractor works better than training the logistic regression classifier on the image pixels directly. To demonstrate the difference, you’ll use Keras to build a logistic regression model that skips the feature extraction part and works directly on pixels.

This is a good way to get started with Keras, and doing this will prove that it’s very hard for a logistic regression model to learn to classify directly from pixel data. Over the course of this chapter and the next, you’ll make the model more and more capable, until at the end you have a classifier that is pretty darn accurate.

A quick refresher

Logistic regression is a statistical model used in machine learning that tries to find a straight line between your data points that best separates the classes.

Of course, this only works well if these data points can be separated by a straight line, or by what is known as a hyperplane in higher dimensions.

Just to give you an idea of what is going on under the hood when you apply a logistic regression, let’s dive into the math a little. It’s OK if you’re not a fan of math, feel free to just skim this section and skip the bits that make your head spin. Knowing the math is not a prerequisite, but it can be helpful to understand what is going on — and it shows that these models are really not magical at all.

Let’s talk math

In the above illustration, data points are two dimensional: They have two coordinates, x[0] and x[1]. In most machine-learning literature and code, x is the name given to the training examples.

In practice, your data points will often be placed in much higher-dimensional spaces. Recall that for an image of size 227×227, the number of dimensions is over 150,000. But for the purpose of explanation, imagine that each data point is just made up of two values.

Hopefully, you still remember from high school math that the algebraic formula for a straight line is:

y = a*x + b

Here, x is a coordinate in the first dimension, a is the slope of the line — how steep it is, also known as the coefficient — and b is the y-intercept. You’ve probably seen this formula before. This is the formula that is learned by linear regression, which tries to find a line that fits best between the data points.

Logistic regression is a small modification of linear regression, so it makes sense that we look at the linear regression formula first.

The above formula is for one-dimensional data, i.e., for data points that consist of just a single x value. In the illustration above, the data points are two dimensional and therefore have two values, x[0] and x[1]. You can easily extend the line formula to the following:

y = a[0]*x[0] + a[1]*x[1] + b

In general, y is the name we use for the predictions made by the model, as well as for the labels that the model is trained on.

Since there are two values in each data point, there are also two coefficients or slopes, a[0] and a[1]. Here, a[0] is the slope of the line for the data point’s first coordinate, x[0]. In other words, a[0] is how much y increases as x[0] becomes larger.

Likewise, a[1] is the slope for the second coordinate, or how much y increases as x[1] becomes larger.

The b is still the y-intercept — the value of y at the origin of the coordinate system — although in machine learning it is called the bias. This is the value of y when both x[0] and x[1] are 0.

It’s a little tricky to draw the value of y on top of a flat picture, but it looks something like this:

Note that y is no longer on the vertical axis. In the above example, the vertical axis is used for x[1], the second coordinate of the data points. Since the data points use two coordinates, the formula is no longer the equation for a line but for a plane in a three-dimensional coordinate space. a[0] and a[1] are still slopes, but now of a plane instead of a simple line, and b is the height of this plane at the origin.

The data points from class A are in the area where y is negative and the data points from class B are in the area where y is positive. The decision boundary that separates the two classes is exactly where y = 0. The further away you go from the decision boundary, the larger the value of y is (positive or negative).

The coefficients a[0] and a[1] are constants. b is also a constant. In fact, what logistic regression learns during training is the values of these constants. Therefore, we call those the learned parameters of the model. Our model currently has three learned parameters: a[0], a[1] and b.

After training the model on this tiny example dataset, you might find that a[0] = 1.2, a[1] = -1.5 and b = 0.2. The reason a[1] is a negative number is that for large values of x[1], it’s more likely the data point belongs to class A, and therefore y should be negative. You can verify this in the image.

For large values of x[0], the model wants y to be positive and so a[0] is a positive number. For data points close to the decision boundary, it depends just on how the numbers turn out.

By the way, when programmers say parameters, we often refer to the values that we pass into functions. Mathematicians call these arguments. To a mathematician, a parameter is a constant that is used inside the function. So, technically speaking, parameters and arguments are two different things — and if you have to believe the mathematicians then we programmers tend to use the wrong term.

If we put the linear regression formula in code it would look like this:

func formula(x0: Double, x1: Double) -> Double {
  let a0 = 1.2
  let a1 = -1.5
  let b = 0.2
  return a0*x0 + a1*x1 + b
}

Notice how x0 and x1 are the arguments that are passed into the function, while a0, a1 and b are constants that are always the same for this function. Machine learning is the process of learning the proper values for these constants, and then you can use this function with different kinds of inputs x0 and x1.

Into the 150,000th dimension

Two-dimensional data is easy enough to understand, but how does this work when you have data points with 150,000 or more dimensions? You just keep adding coefficients to the formula:

y = a[0]*x[0] + a[1]*x[1] + a[2]*x[2]
	 + ... + a[149999]*x[149999] + b

This is a bit labor-intensive, which is why mathematicians came up with a shorter notation: the dot product. You can treat a and x as arrays — or vectors in math-speak — with 150,000 elements each. And then you can write:

y = dot(a, x) + b

Here, dot() is a function that takes the dot-product between two vectors. It multiplies each element of the first vector with the corresponding element from the second vector, and then sums up these products. The result of a dot product is always a single number. Here is how you could implement dot() in Swift:

func dot(_ v: [Double], _ w: [Double]) -> Double {
  var sum: Double = 0
  for i in 0..<v.count {
    sum += v[i] * w[i]
  }
  return sum
}

Using dot() is a nice shorthand way of writing the full formula, plus it works for any number of dimensions, no matter how big a and x are.

So far, the formula we’ve talked about for the line (actually, hyperplane) is for linear regression, not logistic. The linear regression formula just describes the best line that goes between the data points, which is useful in case you want to predict what x[1] is when you only have a given x[0].

Linear regression, usually just called regression, is a statistical model and machine-learning technique that is used to find the relationship between two or more variables. If x is the square footage of a house and y is the selling price of that house, then linear regression can learn a model that is used to predict house prices based on the size of the house (and possibly any other variables that would be relevant).

But you’re not trying to solve that kind of problem here; you’re trying to build an image classifier. To turn this into a classifier, you have to decide for each data point on which side of the line it is to determine its class, and also how far away it is from the line. Further away gives us greater confidence in the class prediction.

To do this, you could simply look at whether y is a positive or negative number, but there is a neat trick that lets you interpret y as a probability value.

From linear to logistic

To turn the linear regression formula into a classifier, you extend the formula to make it a logistic regression:

probability = sigmoid(dot(a, x) + b)

The sigmoid function, also known as the logistic sigmoid, takes the decision boundary and looks at which side of the line the given point x is. The formula for sigmoid is:

sigmoid(x) = 1 / (1 + exp(-x))

When you plot this sigmoid function, it looks like this:

This should explain the name of the function: It’s S-shaped, and “sigmoid” literally means “like the letter sigma” — sigma being the Greek letter S.

You can see in the figure that the output of the sigmoid function is 0 for large negative input values, is 1 for large positive inputs, and is somewhere in between for input values between -6 and +6.

For our example, an output of 0 means the data point is in class A, because the input to the sigmoid would have been a (large) negative number. An output of 1 means the data point is in class B — because the input to the sigmoid would have been a (large) postive number.

However, the output of the logistic sigmoid function is usually interpreted as being a probability, so 0 really means there is 0% chance that this data point belongs to class B and 1 means 100% of it being class B. The probability that the data point belongs to class A is therefore 1.0 - probability.

For data points that are close to the decision boundary, you saw that y was a small positive or negative number. For such a number, the sigmoid output is somewhere between 0 and 1, for example 0.3. This means the algorithm is 30% confident that the data point is class B, so it’s not entirely sure. Usually we choose 50% as the cut-off point; anything higher is B, anything lower is A. But sometimes it makes sense to choose a higher or a lower cut-off point for making this decision.

So logistic regression is just linear regression with the sigmoid function applied to it. This sigmoid function turns the value of y into a value between 0 and 1 that we can interpret as being a probability percentage.

Not everything is black and white…

What if you have more than two classes? In that case, you’ll use a variation of the formula called multinomial logistic regression that works with any number of classes. Instead of one output, you now compute a separate prediction for each class:

probability_A = sigmoid(dot(a_A, x) + b_A)
probability_B = sigmoid(dot(a_B, x) + b_B)
probability_C = sigmoid(dot(a_C, x) + b_C)
probability_D = sigmoid(dot(a_D, x) + b_D)
...and so on...

If you have K classes, you end up with K different logistic regressions. Each has its own slopes and bias, which is why you now don’t have just one a and b but several different ones. For each class, you do the dot product of the input x with the coefficients for that class, add the bias, and take the sigmoid.

So instead of a single decision boundary, each class now has its own decision boundary that separates its data points from the data points of all other classes. For example, if the probability_A is 0.95, it means that the classifier is 95% sure that this data point lies on the side of the line for class A, with a 5% chance that it’s actually one of the other classes. This is also known as a “one-vs.-all” or “one-vs.-rest” classifier.

In practice all of these individual slopes are combined into a big matrix called the weights matrix. This matrix has size N×K, where N is the number of elements in the input vector x and K is the number of classes. All the bias values are combined into a vector of K values. Then the computation is:

output = matmul(W, x) + b

The matmul() function performs a matrix multiplication between the input x and the weight matrix W and then adds the bias vector b. The output is a vector of K values, one for each class.

If your matrix math is rusty, don’t panic. This just performs the dot products for the different classes in a single mathematical operation. Just like the dot product itself is shorthand for a[0]*x[0] + a[1]*x[1] + ..., so is a matrix multiplication shorthand for doing a bunch of different dot products.

The result of all this arithmetic, output, contains K different values, one for each class. You can then apply the sigmoid function to each of these K values independently, to get the probability that the data point x belongs to each class:

probability_A = sigmoid(output[0])
probability_B = sigmoid(output[1])
probability_C = sigmoid(output[2])
...and so on...

It’s now possible for more than one class to be chosen, since these K probabilities are independent from one another. This is known as a multi-label classifier. You would use this kind of classifier if you wanted to identify more than one kind of object in the same image.

However, for a multi-class classifier, such as the one you’ve been reading about in the past chapters, you don’t want independent probabilities. Instead, you want to choose the best class amongst the K different ones. You can do that by applying a different function instead of the logistic sigmoid, called softmax:

probabilities = softmax(matmul(W, x) + b)

The softmax function takes the exponent of each value and then divides it by the sum of all exponentiated values. You may immediately forget this, just know that the result of this operation is that now all the numbers are between 0 and 1, and together they sum up to 1.0. This allows you to interpret the output from the logistic regression as a probability distribution over all the classes taken together. To find the winning class, you simply pick the class with the highest probability.

In practice, you’ll see both sigmoid (multi-label) and softmax (multi-class) used with multinomial logistic regression, depending on the problem that’s being solved. If you’re just interested in the best class, use the softmax.

All right, that’s the end of the math lesson. Let’s get back to doing actual machine learning!

Building the model

In this section, you’ll turn the above math into code using Keras. Fortunately, Keras takes care of all the details for you, so if the math in the previous section went over your head, rest assured that you don’t actually need to know it. Phew!

Fire up Jupyter and create a new Python 3 notebook. You can also follow along with the LogisticRegression.ipynb notebook from this chapter’s downloaded resources.

The first thing you’ll do is import the required packages:

import numpy as np
import keras
from keras.models import Sequential
from keras.layers import *
from keras import optimizers

Like most machine-learning and scientific computing packages, Keras heavily depends on NumPy so you import that first. You also import a few modules from Keras.

Note: It’s not unusual to see a warning message when you execute some Keras or TensorFlow code. You can safely ignore such warning messages. They are usually harmless notifications about deprecated APIs that will be removed in the future.

Next, define some constants:

image_width = 32
image_height = 32
num_classes = 20

Because the model makes predictions for 20 different types of objects (apples, bananas, etc.), you set num_classes to 20.

You’ll use images of 32×32 pixels as input. The SqueezeNet model from Turi Create used 227×227 images. You could certainly use 227×227 here, or any size really, but it will make the model much larger and slower to train. If you have access to a fast GPU, feel free to experiment with a larger image_width and image_height.

Let’s now define the regression model using Keras:

model = Sequential()
model.add(Flatten(input_shape=(image_height, image_width, 3)))
model.add(Dense(num_classes))
model.add(Activation("softmax"))

The model you’re building is a so-called Sequential model, which is a simple pipeline that consists of a list of layers. Each layer is a stage in the pipeline that transforms the data in some particular way.

Here, you’re adding three layers to the model.

The features that the logistic regression works on, are the pixels from the input images. The first layer is Flatten, which takes the three-dimensional image input and turns it into a one-dimensional vector.

“Wait a minute,” I hear you thinking, “an image surely has just two dimensions, not three!” The third dimension is for the pixel’s RGB values. Each pixel is made up of three numbers describing its color: red, green and blue. We consider this the image’s third dimension, or the “depth” dimension. Images often have an alpha channel, too (RGBA), but we typically ignore the alpha channel in machine learning.

Also note that the image dimensions are given as (height, width, 3), not (width, height, 3). It’s common for programmers to describe the size of an image as width-by-height, but the image is actually stored in memory as rows × columns × RGB. So in machine learning the size of the image is usually given as height-by-width.

Note: This difference in the order of the dimensions, height coming before width, is easy to overlook and can cause subtle bugs in your model, especially if the width and height are the same, and so it’s easy to mix them up. Pay close attention to the order that tools like Keras expect the input data to be in. When you load an image from a file, it’s already loaded as height × width × 3 into memory, so you don’t actually have to do anything special. Just be aware that height goes before width.

Since logistic regression expects a one-dimensional vector as input, the Flatten layer simply unrolls the image’s lines of pixels into one big strip:

The input image is 32×32 pixels times three channels, and so the flattened vector has length 3,072. Flatten doesn’t do any computation, it just changes the shape of the input.

The real meat of the logistic regression happens in the Dense layer. This performs the matrix multiplication between the 3,072 inputs and the 20 outputs. This layer has 20 outputs because that’s the number of classes in the snacks dataset. In a Dense layer, each input is connected to each output.

This is simply the equation you’ve seen before:

y = a[0]*x[0] + a[1]*x[1] + ... + a[3071]*x[3071] + b

This time, it’s expressed in a slightly more efficient form as a matrix, so that Keras can compute this entire thing with a single matrix multiplication.

The weights a represent the strength of the connections between the inputs and the outputs, shown in the illustration as thick and thin lines. The larger the value of the weight a[i], the more the corresponding input x[i] counts in the final result.

The Dense layer also adds a bias value for each output, b in the above equation. Because there are 20 outputs, b is a vector of 20 elements. The bias is just a fixed number that’s added to every output, and gives the distance of how far away the decision boundary is from the coordinate system’s origin. This is necessary because the data points might not be nicely distributed around the origin, and so the bias can compensate for that.

Note: Dense layers are also known as fully connected layers, affine layers, or linear layers. In machine learning a single concept often has multiple names.

When you create the Dense layer, it assigns random numbers to the weights for the matrix multiplication and zeros to the bias values. The reason it uses random numbers for the weights and not zeros, is that multiplying the inputs with zero makes the outputs zero too, and it’s hard to turn that back into something that is not zero. In practice, training just works better from a randomly chosen starting point.

When you train the logistic regression model, it will learn the best values to use for these weights and biases.

Finally, you need to apply the softmax function to turn the output from the Dense layer into a probability distribution. That’s what Activation("softmax") does. An activation function is some non-linear operation that gets applied to the output of a layer from the model. There are many different types of activation functions, but the one at the end of the model is usually the softmax function, at least for classifiers.

Without this softmax function, the model would be a plain linear regression that only tells you how to best fit a line (hyperplane) through all the data points for the training images. By adding the softmax, the model becomes a multinomial logistic regression classifier that tells you which classes the data points belong to, depending on which side of the line they fall.

After you construct a model, it’s useful to verify that all the pieces are in the right place. Keras provides a handy function for this:

model.summary()

This outputs a list of all the layers in the model:

______________________________________________________________
Layer (type)                 Output Shape              Param #   
==============================================================
flatten_1 (Flatten)          (None, 3072)              0         
______________________________________________________________
dense_1 (Dense)              (None, 20)                61460     
______________________________________________________________
activation_1 (Activation)    (None, 20)                0         
==============================================================
Total params: 61,460
Trainable params: 61,460
Non-trainable params: 0
______________________________________________________________

The Output Shape column gives the size of the data after it has been transformed by that layer. As expected, Flatten shows a vector with 3,072 elements and Dense outputs a vector with one element for each of the 20 classes.

Even though Flatten produces a one-dimensional vector, the output shape shown in the summary actually has two dimensions with the first dimension being None.

Keras automatically adds a dimension to the front of the layer’s output, which is the batch dimension. This extra dimension is used during training, so that you can train on multiple images at the same time. The images are combined into a so-called batch or mini-batch. If you were to train on a typical batch size of 64 images at once, the output shape of the Flatten layer is actually a (64, 3072) tensor. Typically, you don’t specify the batch size yet when you construct the model, which is why Keras shows it as None.

What the !%#& is a tensor? It finally happened, we used the T-word, so we’d better explain what a tensor is at this point. Are you ready? Tensor is a fancy word for multi-dimensional array. Yup, that’s all.

In machine learning, you often use multi-dimensional arrays to store your data. You’ve already seen that an image is stored as an array of shape (height, width, 3). This is a three-dimensional array where the first dimension is the height of the image, the second dimension is the width of the image, and the third and final dimension is for three color channels (RGB). But often you’ll use arrays with even more dimensions: four, five or six.

As the data flows through the pipeline it changes shape: the dimensions can become larger or smaller, and you can even add or remove dimensions, like what Flatten does. Since “multi-dimensional array” is a mouthful, we prefer to use the word “tensor” instead. This term originally comes from the mathematical field of topology, where it has a somewhat more specific meaning, but in ML it’s just shorthand for multi-dimensional array. This is where TensorFlow gets its name from: it describes the data flow — what we’ve been calling a pipeline — between tensors.

In math terminology, we call a one-dimensional array a vector, a two-dimensional array a matrix, and anything with more dimensions a tensor. The number of dimensions is the rank of the tensor. A vector is a tensor of rank 1, a matrix is a tensor of rank 2, an image is a tensor of rank 3, a batch of images is a tensor of rank 4 and so on. By the way, scalars or single numbers are tensors of rank 0, or zero-dimensional arrays.

At this point, you may be getting confused by the term dimensions. The tensor that stores an image has three dimensions, but the image itself can be considered a point in 150,000-dimensional space. Or in the case of the 32×32 images you’re using here, a point in 3,072-dimensional space. It’s a little confusing that the same word is used in both cases. For tensors we often also use the word “axis” to describe a dimension, so an image tensor has three axes with the first axis being the height, the second axis the width, and the third axis being the color channels.

The Param # column in the summary shows the number of learnable parameters in each layer. In this simple model, only the Dense layer has learnable parameters: the values of the weights or coefficients a and the values of the bias vector b. There are 3,072×20 weights plus 20 additional bias values, so this model has 61,460 learnable parameters in total.

Turi Create’s model only had 19,090 parameters. Your model is a bit bigger… but is it also better? No spoilers, you’ll have to keep reading!

Compiling the model

Before you can use the model you first need to compile it. This tells Keras how to train the model.

model.compile(loss="categorical_crossentropy",
              optimizer=optimizers.Adam(lr=1e-3),
              metrics=["accuracy"])

The compile() function takes three important arguments:

• The loss function to use: Recall from the introduction that the loss function determines how good — or rather, how bad — the model is at making predictions. During training, the loss is initially high as the model just makes random predictions at the start. But as training progresses the loss should become lower and lower while the model gets better and better.

  It’s important to choose a loss function that makes sense for your model. Because your model uses softmax to produce the final output, the corresponding loss function is the categorical cross-entropy. That sounds nasty, but categorical just means you’re building a classifier with more than two classes, and cross-entropy is the loss that belongs with softmax. For a classifier with two classes, you’d use binary cross-entropy loss instead.

• An optimizer: This is the object that implements the Stochastic Gradient Decent or SGD process that finds the best values for the weights and biases. As the loss function computes how wrong the model is at making predictions, the optimizer uses that loss and tweaks the learnable parameters in the model to make the model slightly better. Mathematically speaking, the optimizer finds the parameters that minimize the loss.

  There are different types of optimizers but they all work in kind of the same way. You’re using the Adam optimizer, which is a good default choice, with learning rate 1e-3 or 0.001. The learning rate or LR determines how big the steps are taken by the optimizer. If the LR is too big, the optimizer will go nuts and the loss never becomes any smaller (or may even blow up into a huge number). If the LR is too small, it will take forever for the model to learn anything.

  The learning rate is one of the most important hyperparameters that you can set, and finding a good value for the LR is key to getting your model to learn. The author tried out a few different values and settled on 1e-3 as a good choice for this particular model.

• Any metrics you want to see: As it is training your model, Keras will always print out the loss value, but you’re also interested in the accuracy of the model as that is an easier metric to interpret. A loss value of 0.35 by itself doesn’t say much about how good the model is, but an accuracy value of 94% correct does.

Cool, now you’re ready to start training this model. But for that you need some data.

Loading the data

You’ve already seen the snacks dataset in the previous chapters. It consists of three different folders (train, val, test), each containing 20 folders for the different classes, and each folder contains several dozen or hundred images.

Add some variables that point to these folders:

images_dir = "snacks"
train_data_dir = images_dir + "/train/"
val_data_dir = images_dir + "/val/"
test_data_dir = images_dir + "/test/"

Important: point images_dir at the folder where you’ve downloaded the dataset.

At this point it’s a good idea to actually look at the training data with your own two eyes, to make sure it is correct. To view an image in the notebook, do the following:

from keras.preprocessing import image
img = image.load_img(
  train_data_dir + "apple/cecd90f5d46f57b0.jpg",
  target_size=(image_width, image_height))

This loads the specified JPEG image into the img variable. This is a PIL image object. PIL is a popular image library for Python 2. We’re in fact using the Python 3-specific fork: Pillow, but the concepts are identical. Potential confusion alert: the image variable here refers to the Keras module for dealing with images, while img is the actual image object.

The load_img() function can automatically resize the image to the size your model accepts, given here by the target_size argument. Note that here the size of the image is specified as (width, height) not (height, width). Told you… you’ve got to keep paying attention to the order of these dimensions.

To show the image in the notebook you can use Matplotlib, a very handy Python library for drawing plots and graphs.

%matplotlib inline
import matplotlib.pyplot as plt
plt.imshow(img)

The %matplotlib inline directive tells Jupyter to show the image inside the notebook. Without this, it will open in a new window.

Keras cannot train directly on PIL images, it always expects data to be in the form of NumPy arrays. So first convert from a PIL image to a NumPy array:

x = image.img_to_array(img)

You called this variable x because it is a convention in machine learning that the input data is called x or sometimes capital X. If you now write x or print(x) in a new cell and press Shift-Enter this prints the pixel values from the image:

array([[[215., 215., 217.],
        [211., 211., 211.],
        [207., 207., 207.],
        ...,
        [152., 150., 137.],
        [148., 146., 133.],
        [149., 147., 132.]], ...

As you might have expected if you’ve worked with images before, the pixels have values between 0 and 255. In principle you can train the model directly on these pixel values but it is customary to normalize the data before you start training on it.

Normalizing or feature scaling means that the data will have an average value or mean of 0 and usually also a standard deviation of 1. This is important when different features are not all in the same numerical range. For example, if your data has one feature with values between 0 and 1000 and another feature with values between 5 and 10, training will generally work better if you first normalize the features so that they both are between -1 and +1.

In your case it’s not such a big deal since all the features — the pixels — are on the same scale from 0 to 255. But normalization is good practice so let’s do it anyway. Write this new function:

def normalize_pixels(image):
    return image / 127.5 - 1

This simply scales the pixel values from 0 to 255 to a new range that goes from -1 to +1. Sometimes people subtract different mean values for the red, green, and blue channels and also divide by a standard deviation, but the above method is good enough for dealing with most kinds of images.

Note: In this function, image is a tensor with 32×32×3 elements. When you write image / 127.5, NumPy will perform the division on each of the tensor’s elements separately. This kind of “vectorized” processing, where you perform an operation on an entire tensor at once, is much simpler — and faster! — than writing a for loop. You’ll see this sort of thing a lot in Python code.

The steps to normalize an image img are then:

x = image.img_to_array(img)
x = normalize_pixels(x)
x = np.expand_dims(x, axis=0)

If you now look at x, the values are much smaller:

array([[[[ 0.6862745 ,  0.6862745 ,  0.7019608 ],
         [ 0.654902  ,  0.654902  ,  0.654902  ],
         [ 0.62352943,  0.62352943,  0.62352943],
         ...,
         [ 0.19215691,  0.17647064,  0.07450986],
         [ 0.16078436,  0.14509809,  0.04313731],
         [ 0.1686275 ,  0.15294123,  0.03529418]], ...

If you’re curious, you can print the mean and standard deviation of this training image with x.mean() and x.std(). The mean of a single training image may not be exactly 0, but across the entire training set it will be close to 0. The standard deviation should be around 0.5.

The np.expand_dims() function added a new dimension to the front, to turn this single image into a batch of images with batch size 1. The tensor containing this image is now of rank 4. You can view this with:

x.shape

This prints (1, 32, 32, 3). It’s always a good idea to double-check the sizes of your images and other data objects, to verify they are correct. Adding this batch dimension is necessary because the Keras training functions always work on a batch of images, and expect this dimension to be there.

Too soon to start making predictions?

Even though the model isn’t trained yet, you can already make a prediction on the input image:

pred = model.predict(x)
print(pred)

Note: If your Jupyter kernel crashes when you run this cell, execute the following command from the Terminal: conda install nomkl. This fixes a package conflict that sometimes causes trouble on the Mac.

You should get an array with 20 values, one probability for each class. Since you haven’t trained the model yet, these predictions aren’t very useful. You’ll see something like the following:

[[0.04173137 0.00418671 0.02269506 0.02889681 0.08140159 0.03577968
  0.03044504 0.04758682 0.07940029 0.07274284 0.04531444 0.0115772
  0.17158438 0.02129039 0.0233359  0.1150756  0.00603842 0.08578367
  0.03525693 0.03987688]]

You’ll probably get different results since your model will be initialized with different random values for the weights and biases. But note that most of these values are pretty close to 1/20 or 0.05. If you add them all up with pred.sum(), it will print out 1.0. Floating point numbers have limited precision, so sometimes you will see 0.99999999 instead of 1.0. Close enough.

An untrained model will make a prediction for each class that is very close to the average, because it hasn’t learned yet how to distinguish the classes. It’s unlikely you’ll see a high percentage such as 90% in the output at this point. Most classes will have a probability score of around 0.05, or 1/num_classes, although it can vary a bit because of the random initialization.

If you were to make predictions for the entire dataset at this point, each class would be predicted the same number of times and the overall accuracy would be 0.05 or 5% — basically a random guess. The goal of machine learning is to train a classifier that can do better than random guessing.

To figure out what the actual predicted class is for this image, you find the maximum value amongst the predicted probabilities:

np.argmax(pred)

For the prediction array shown above, this prints 12, because the element at index 12 is the highest (0.17158438). Note that the np.max() function returns the actual maximum value, while np.argmax() returns the index of the element with the maximum value.

So which class is this? Well, you actually haven’t assigned class labels to each of the 20 outputs yet. That will be done automatically by Keras during training. It will usually do this alphabetically, so the winning class here would be “orange” since that is the 12th class; as usual we start counting at 0.

But, remember, at this point the predictions are still totally bogus. That said, it’s still useful to run model.predict() before training, to make sure that your model actually predicts what you’d expect — in this case, something close to average probability for each class. If the model had returned something else at this point, such as all zeros, then something is broken — and you don’t want to waste any time training a model that is fundamentally buggy.

Using generators

You’ve seen how to load an image into a tensor and how to plot it in the notebook. That’s handy for verifying that the training data is correct. During training, you won’t have to load the training images by hand. Keras has a useful helper class called ImageDataGenerator that can automatically load images from folders.

from keras.preprocessing.image import ImageDataGenerator
datagen = ImageDataGenerator(
  preprocessing_function=normalize_pixels)

The data generator takes the normalize_pixels function as its preprocessing function so that it automatically normalizes the images as it loads them. The data generator can do other stuff as well, as you’ll see in the next chapter when we talk about data augmentation. Using this ImageDataGenerator object you can create three other generators, one for each subset of images:

batch_size = 64

train_generator = datagen.flow_from_directory(
                    train_data_dir,
                    target_size=(image_width, image_height),
                    batch_size=batch_size,
                    class_mode="categorical",
                    shuffle=True)

val_generator = datagen.flow_from_directory(
                    val_data_dir,
                    target_size=(image_width, image_height),
                    batch_size=batch_size,
                    class_mode="categorical",
                    shuffle=False)

test_generator = datagen.flow_from_directory(
                    test_data_dir,
                    target_size=(image_width, image_height),
                    batch_size=batch_size,
                    class_mode="categorical",
                    shuffle=False)

The train_generator is for images from the train folder, the val_generator for images from the val folder, and the test_generator for images from the test folder.

A generator in Python is an object that can produce other objects. In this case you’re making a generator than can produce images by loading them from the given folder. The reason you need to use generators is that you cannot possibly load all the images into memory all at once, since that would require many gigabytes or even terabytes of RAM — more than fits in your computer! The only way to deal with that much data is to load the images on-demand. That’s what the Keras generators allow you to do.

The three generators all do the same thing — load images from their respective folders — but the train generator has shuffle=True while the others have shuffle=False. During training, you want to pick the images at random so that the model doesn’t attempt to learn anything about the order of the images. During testing, however, you want to pick the images in a fixed order as that makes it easier to match them to the correct answers.

The argument class_mode="categorical" tells Keras that there is a subfolder for each image category. Keras will use the name of the subfolder as the class label for the images from that folder. The batch size is 64, and so the generator will try to load 64 images at a time.

When you run the above code, the Jupyter notebook says:

Found 4838 images belonging to 20 classes.
Found 955 images belonging to 20 classes.
Found 952 images belonging to 20 classes.

These are the number of training, validation, and test images respectively.

To see what a generator outputs, you call next() on it:

x, y = next(train_generator)
print(x.shape)
print(y.shape)

You won’t ever need to call next() yourself during training, but it’s useful to test that your generators work. This grabs the next batch of images x and their corresponding labels y from the train folder. The shape of the x tensor is (64, 32, 32, 3) because it contains 64 RGB images of 32×32 pixels.

Since you’ll train on 64 training images at a time, the batch also includes the labels for these 64 images. Recall that these labels, also known as the ground-truths, are used to compute the loss or how “wrong” the model’s predictions are.

Because the model produces 20 output values — one probability value for each class — the ground-truth label for a given image also needs to have 20 values. This is why the shape of the y tensor is (64, 20).

Have a look at the first of these labels, y[0]:

array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 1., 0., 0., 0., 0., 0., 0.], dtype=float32)

You may have expected to see a label like 'apple' or 'cake', but instead you get a vector with 20 numbers. When you try this, you’ll probably get a different label than what’s printed in the book, since the training set is randomly shuffled. But whatever label you get, it should consist of 19 zeros and a single one.

This is called one-hot encoding. The position of the 1 corresponds to the name of the class. In this case the 1 is in the 13th position, which belongs to class pineapple. You can see this by executing the cell:

train_generator.class_indices

This outputs:

{'apple': 0,
 'banana': 1,
 'cake': 2,
 'candy': 3,
 'carrot': 4,
 'cookie': 5,
 'doughnut': 6,
 'grape': 7,
 'hot dog': 8,
 'ice cream': 9,
 'juice': 10,
 'muffin': 11,
 'orange': 12,
 'pineapple': 13,
 'popcorn': 14,
 'pretzel': 15,
 'salad': 16,
 'strawberry': 17,
 'waffle': 18,
 'watermelon': 19}

To print the name of the label for y[0], you can do the following:

index2class = {v:k for k,v
  in train_generator.class_indices.items()}

This is a so-called Python dictionary comprehension. It takes all the key-value pairs in the class_indices dictionary and creates a new dictionary that flips the order of the key and value. Now you can look up the name of the class by the index of the element that is 1 in the one-hot encoded vector for the label.

To find the name of the class, you do np.argmax() to find the index of the 1, and then look up the name in the new dictionary:

index2class[np.argmax(y[0])]

For the y[0] from this book, this will print 'pineapple'.

Why go through all this trouble? Most machine-learning algorithms can only handle numbers, not strings. The text label 'pineapple' doesn’t mean anything to the logistic regression. So, instead, you first convert this string into something numeric, a one-hot encoded vector. Now the machine-learning algorithm can tell the classes apart because each class has its own unique one-hot encoded vector:

'apple'      [1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
'banana'     [0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
'cake'       [0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
'candy'      [0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
              . . .
'waffle'     [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0]
'watermelon' [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1]

The model has 20 outputs, and so the one-hot encoded label also needs to have 20 elements. This also means that the first output from the model is the probability the class is apple, the second output is the probability that the class is banana, and so on. These one-hot encoded vectors establish the relationship between the model’s outputs and the class labels.

Think of this one-hot encoded vector as the “ideal” probability distribution for the corresponding training image. If the label of a training image is 'apple' then this ideal probability distribution should have class apple at 100% (the 1 in the one-hot encoded vector) and the other classes at 0% (the 0s in the vector).

The generator automatically makes these one-hot encoded vectors for you. It looks at the name of the folder to determine the correct class label for the image, and then one-hot encodes it, to turn to it into a numeric label that can be given to the machine learning algorithm.

The first evaluation

At this point, it’s a good idea to run the untrained model on the entire test set, to verify that the model and the generators actually work.

model.evaluate_generator(test_generator,
                         steps=len(test_generator))

It’s as easy as that. Keras now uses the test_generator to load all the images from the test set, gives them to the model to make predictions, and compares the model’s output to the ground-truth label for each test image.

For example, if the model’s thirteenth output has the highest probability value, the model has predicted this image contains a pineapple. If the label for that image really is 'pineapple', then this counts as a correct prediction. But if the label was something else, then it counts as a wrong prediction. The accuracy of the model is the number of correct predictions divided by the number of total predictions.

The steps argument tells Keras how many batches to evaluate. To get the number of batches a generator will produce, you can call len(generator). With a batch size of 64, the test generator creates 15 batches, because there are 952 test images in total.

Tip: If you get an out-of-memory error at this point, reduce the batch size. It’s common to use powers of two for this, so if a batch size of 64 is too large, try 32. If that’s still too large, try 16, and so on. If you keep getting memory errors even with a batch size of 1, you’ll need to restart the notebook and run all the cells again. Sometimes Keras or TensorFlow cannot recover from these out-of-memory errors, and it’s best to start afresh.

After about 10 seconds or so of number crunching, evaluate_generator() prints out values similar to the following:

[3.311799808710563, 0.059873949579831935]

The first one is the loss, the second, accuracy. Your values should be similar, but will be slightly different because of the different random initialization of the model weights.

At this point, the accuracy across the entire test set should be about 0.05 or 5% correct, which is the same as randomly picking an answer from the 20 categories. Of course, that’s exactly what happens because the model currently consists of all random numbers.

The initial loss for a classifier that uses the cross-entropy loss function should be approximately np.log(num_classes), where log is the natural logarithm. Here, np.log(20) = 2.9957 so the loss is slightly higher. But it’s close enough. Again, this discrepancy is the result of the random initialization. If you were to get a loss that is much larger or much smaller than about 3.0, something is not right with the model. It also tells you that if the loss becomes smaller than 3.0 during training, the model is actually learning something.

Note: Try for yourself what the initial loss and accuracy are on the training and validation sets. Evaluating the training set may take a few minutes instead of seconds because it has more images.

Training the logistic regression model

All the pieces are in place to finally train the model. First, do the following:

import warnings
warnings.filterwarnings("ignore")

As a responsible programmer, you know it’s not a good idea to ignore warnings but unfortunately the PIL library that is used to load the training images will complain about the EXIF data on some of the JPEG files. That just causes a lot of sloppy debug output in the Jupyter notebook, and so it’s cleaner to disable those warnings.

Training is really just a matter of calling fit_generator() on the model. To start with, you’ll train for five epochs — an epoch is one pass through all the training images.

To get good results, you’ll need to show each training image more than once — dozens or hundreds of times, in fact — which is why you need to train for multiple epochs:

model.fit_generator(train_generator,
                    steps_per_epoch=len(train_generator),
                    validation_data=val_generator,
                    validation_steps=len(val_generator),
                    epochs=5,
                    workers=4)

Depending on the speed of your computer, this may take a few minutes to complete.

The generator you used here is train_generator because that loads the training images. You also pass in the val_generator to use as the validation data.

During training, Keras calculates the accuracy on the training images, but this can be misleading since it doesn’t tell you anything about how well the model does on images it hasn’t seen before. Training accuracy going up — and training loss going down — only means that the model is learning something, but you can’t be sure it is really learning the thing you are aiming to teach it.

That’s why, after every epoch of training, Keras uses the validation set to compute the validation accuracy and loss, to give you an idea of whether the model really is working or not. If training accuracy is high but validation accuracy is low, you’ve got a problem.

Note: The workers=4 argument tells Keras it can use multiple threads to load and prepare the images. If you have more than four CPU cores in your computer, feel free to increase this number for some extra speed.

What happens during training?

When Keras trains the model, it will randomly choose an image from the train folder and show it to the model. Say it picks an image from the banana folder. The model will then make a prediction, for example pretzel. Of course, this is totally wrong.

Initially, when you create the model, the learnable parameters are just randomly chosen numbers and the predictions will be way off. Over the course of training, these random numbers will slowly change into something more reasonable that can actually make good predictions.

Since it knows what folder the image came from, banana, Keras can compute a loss between the prediction (pretzel) and the ground-truth label for the image (banana). Of course, banana and pretzel are meaningless concepts to Keras, but, after turning them into numbers — using one-hot encoding — Keras can compute some kind of difference between them.

The ground-truth for banana is this one-hot encoded vector:

[ 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.,
  0., 0., 0., 0., 0., 0., 0., 0., 0., 0. ]

The dot behind the numbers means that these are floating-point values. Think of these as probabilities: The probability for class banana is 1.0 or 100%, the probabilities for all other classes are 0%. That is because we are 100% sure this image contains a banana, since that is how we labeled it when we created the dataset. No doubt there.

The prediction from the model for this banana image may be something like this:

[ 0.01360181, 0.21590623, 0.00830788, 0.01217055, 0.05090828,
  0.01749134, 0.01430813, 0.07134261, 0.02015499, 0.00142231,
  0.01328659, 0.01184934, 0.01497147, 0.04739711, 0.00372085,
  0.38552788, 0.03598726, 0.0047219 , 0.01521332, 0.04171015 ]

This is the output of the softmax layer, which makes sure that the most confident prediction is large, less confident predictions are smaller, and all the numbers add up to 1.0. That probability distribution looks a lot messier than the ground-truth:

The highest number in this vector is for pretzel (38.55%) but note that the model isn’t entirely certain and even thinks it might be a banana after all (21.59%). Especially early on in the training process, the model will not be very certain about its predictions yet.

Now that you have two vectors of 20 elements each, it means you can compare them. The formula for this is known as the cross-entropy loss. This chapter has already had enough math in it, so let’s just say that this compares each element between the two vectors in some fashion, and adds up the results. This gives the loss for this particular image, which is just a single number.

If the prediction was also (mostly) banana, then the softmax output looks a lot like the ground-truth and the loss is very small; if the prediction was 100% banana, then the loss is 0 because it’s exactly right.

If the prediction for this image is not banana, then the loss is a larger number. The worse the prediction is, the less the predicted probabilities match the ground-truth probabilities, and the higher the loss will be.

For this particular example, the loss is 1.5329. That number by itself doesn’t tell you very much, it’s just a number. What’s important is that this number goes down over time while the model is being trained. Say that, after a few more epochs of training, the prediction for this image now has 0.9 for banana and the remaining 0.1 is spread out amongst the other classes. The new loss is then 0.1054. This prediction is much better, and so the loss is also lower.

Once it has computed a loss value, Keras uses the Adam optimizer you provided when you compiled the model, to figure out which parts of the model contributed to this loss.

The optimizer finds the parts of the model that were responsible for making this (bad) prediction and “punishes” them. It does this by slightly tweaking the learnable parameters by moving them in the opposite direction — a positive number becomes a little more negative, a negative number becomes more positive — so that next time this image is shown to the model it will make a slightly better prediction.

In practice, Keras won’t compute the loss for a single image but for a mini-batch of multiple images at a time. You are using a batch of 64 images. The loss for this batch is the average of the 64 individual losses. There are two reasons for using batches:

1. It uses the CPU or GPU more efficiently, if you’re lucky enough to have a GPU for training. The key to efficient GPU performance is to keep it busy, and with a batch you use more of the GPU’s memory bandwidth. The size of the batch is limited by the amount of RAM on the GPU. For a large model with many layers, a batch size of 64 may be too big to fit on the GPU and you’ll have to smaller batches.

2. Mathematically speaking, the “true” loss function really ought to be computed over the entire training set at once. So when you’re using batches, which only contain a small portion of the training set, you’re not actually computing the true loss of the model. That would seem to be a bad thing, but the opposite is true: using only 64 or fewer images at a time introduces a certain amount of randomness into the training process. And it turns out that this randomness makes it easier for the model to learn. Strange, but true. That’s why the S in SGD stands for stochastic, which means “random” but sounds more impressive.

Hey, it’s progress!

While the training process is happening, Keras outputs a progress bar:

Epoch 1/5
76/76 [==============================] - 3s 38ms/step -
	loss: 3.2150 - acc: 0.1050 -
	val_loss: 3.2654 - val_acc: 0.1162
Epoch 2/5
76/76 [==============================] - 2s 26ms/step -
	loss: 2.7257 - acc: 0.2079 -
	val_loss: 3.2375 - val_acc: 0.1152
Epoch 3/5
76/76 [==============================] - 2s 27ms/step -
	loss: 2.4124 - acc: 0.2990 -
	val_loss: 3.2756 - val_acc: 0.1120
Epoch 4/5
76/76 [==============================] - 2s 27ms/step -
	loss: 2.1712 - acc: 0.3722 -
	val_loss: 3.2727 - val_acc: 0.1246
Epoch 5/5
76/76 [==============================] - 2s 26ms/step -
	loss: 1.9735 - acc: 0.4462 -
	val_loss: 3.3359 - val_acc: 0.1141

During training, Keras reports the training loss loss and training accuracy acc. After each epoch, Keras also computes the validation loss val_loss and accuracy val_acc over the entire validation set.

Notice how loss and acc are improving over time. The training loss goes down while the accuracy goes up. That’s the good news. However, the bad news is that the validation loss doesn’t seem to be getting much better and the validation accuracy never gets higher than about 12%.

Even if you keep training for more epochs, the training accuracy keeps improving but the validation accuracy does not. Try it out, run model.fit_generator() again and see what happens. In fact, if you repeat this enough times, the validation accuracy may get worse over time. After 50 or so epochs of training, the training accuracy was 90% but the validation accuracy had dropped to 8%.

To make sure this is not a fluke, you can also try this trained model on the test set:

model.evaluate_generator(test_generator,
                         steps=len(test_generator))

This should print something like:

[3.142886356145394, 0.12079831951556086]

Again, that is only about 12% accuracy on the images from the test set. Also note that the loss reported here, 3.1428, is only marginally better than the test set loss you saw on the untrained model, which was 3.31179.

It could be better…

What does this mean? Well, the model did learn something. After all, you started with a validation accuracy of 0.05 and it went up to about 0.12. So the model did gain a little bit of knowledge about the dataset. It is no longer making completely random guesses — but it’s still not doing much better than that.

How come the training accuracy is so high then? It goes up to about 90% after 50 epochs… This is an extreme case of overfitting. Yup, there it is again. The model isn’t actually learning to classify images, it’s just learning to tell apart the images that are in the training set. It’s likely that the model is learning which combinations of pixels belong to which training image — and that’s not what you want. You want the model to understand what those pixels represent in a more abstract sense.

The model has 61,460 learnable parameters and there are only 4,838 images in the training set, so the model easily has enough capacity to remember which class goes with what image in the training set. In fact, with a training accuracy of 90%, and a very low accuracy on the validation set and test set, it means that the model managed to memorize the class for nine out of 10 images. In the previous chapter, you saw that the Turi Create model also suffered from overfitting and it had fewer parameters than this model, only 19,019. In general, the more parameters a model has, the worse a problem overfitting becomes.

You don’t want to train a model that remembers specific training images; you want a model that can learn to classify images it hasn’t seen yet. And this model fails spectacularly at that. There are several techniques you can use to dissuade the model from overfitting, but it’s clear already that trying to learn directly from pixels what these 20 different types of categories are, is a task that logistic regression is not up to.

Now don’t let this section make you believe that logistic regression is a bad machine learning model. It isn’t. In fact, for many ML problems it is the go-to solution. But for logistic regression to work well it is important that the number of features is much less than the number of training examples.

In our case, we had 3,072 features — the pixel values — but only about 4,800 training images. The logistic regression model might work better if we had 10 times or 100 times as many training images.

Note: For fun, try making the input images smaller or larger, thereby changing the number of features, and see what kind of effect that has on the training and validation accuracy. If you do, you may also need to make the learning rate larger or smaller, so experiment with that too.

For better results on our kinds of images, we’ll need to create a better model. Learning directly from the pixel values is just too hard, as the logistic regression (the Dense layer) cannot extract enough meaning from them.

The hyperplanes it can draw through this 3,072-dimensional space do not separate the data points cleanly by their classes. This is why Turi Create first converts the pixels into a smaller number of features using SqueezeNet, and why Create ML does the same with Vision FeaturePrint.Scene. For machine learning to work well on image data, it needs to go through more transformations than just this one Dense layer!

In classical computer vision, before the advent of deep learning, people carefully hand-crafted feature extractors (with names such as SIFT, SURF, HOG, ORB, etc.) in order to turn the pixel data into something more meaningful that they then could apply logistic regression to. However, deep learning can automatically learn to extract features from the pixels, and generally does a better job than man-made feature extractors.

It’s clear that logistic regression directly on the image pixels isn’t going to work. Let’s make the model more powerful by turning it into an artificial neural network.

Your first neural network

Logistic regression is considered to be one of the classical machine-learning algorithms. Deep learning is new and modern and hip, and is all about artificial neural networks. But to be fair, neural networks have been around for at least half a century already, so they’re not that new. In this section, you’ll expand the logistic regression model into an artificial neural net.

A classical neural network looks like this:

The idea is that this kind of network mimics connections between neurons in the human brain, in which the circles in the picture represent the neurons. Notice how similar this is to the picture of the Dense layer from earlier? That’s because you can think of this kind of neural network as being two or more logistic regressions in a row.

You can do this in Keras by adding a second Dense layer to the previous model:

model = Sequential()
model.add(Flatten(input_shape=(image_height, image_width, 3)))
model.add(Dense(500, activation="relu"))  # this line is new
model.add(Dense(num_classes))
model.add(Activation("softmax"))

Now the model.summary() looks like this:

______________________________________________________________
Layer (type)                 Output Shape              Param #   
==============================================================
flatten_1 (Flatten)          (None, 3072)              0         
______________________________________________________________
dense_1 (Dense)              (None, 500)               1536500   
______________________________________________________________
dense_2 (Dense)              (None, 20)                10020     
______________________________________________________________
activation_1 (Activation)    (None, 20)                0         
==============================================================
Total params: 1,546,520
Trainable params: 1,546,520
Non-trainable params: 0
______________________________________________________________

The first Dense layer connects all flattened 3,072 input pixel values to 500 intermediate hidden neurons, and the second Dense layer connects these 500 neurons to the 20 outputs. This kind of neural network is called a two-layer feed-forward network.

The first half of this neural network, from the input to the output of the first Dense layer is the first logistic regression. The second half of the network, from the second Dense layer to the end is the second logistic regression. So all you’ve done is stick two separate logistic regression models together.

The activation function at the end of the model is still the softmax that converts the outputs into probabilities. The new Dense layer also has an activation function. This is not a softmax but a relu, also called ReLU or rectified linear unit.

In most neural networks every layer is followed by an activation function. This is usually a very simple mathematical operation that transforms the output of the layer in some non-linear way.

Such non-linearities are necessary because otherwise the model can only learn linear relationships between their inputs (the pixels) and their outputs (the classes) and that gives very limited results.

Remember that the goal is to transform the input data in such a way that the model can draw an imaginary straight line or hyperplane between the classes. Without these non-linear activation functions, you’d only be able to do that if you could already draw that straight line between the original input data points — in which case you wouldn’t need to train a model at all. It is the non-linearities that allow the model to learn all kinds of interesting data transformations.

ReLU is an extremely simple mathematical function that looks like this:

In code, it is:

y = max(0, x)

In other words, if the number x is less than 0, the output of the ReLU is 0, otherwise the number passes through to the next layer unchanged.

There are other activation functions, too, such as the logistic sigmoid that you’ve seen in the math section (if you didn’t skip it), but usually you’d use ReLU. The linear unit part of ReLU’s name means that it’s just a straight line, and rectified means the line gets flattened for negative values, making this function non-linear.

It turns out that the actual shape of the activation function that you’re using doesn’t really matter so much, as long as it introduces non-linear behavior into the model. Most machine-learning models use ReLU because it’s really simple and fast to compute. For a real logistic regression, you would actually use the sigmoid activation function, so technically speaking the first half of this neural network isn’t truly a logistic regression because it uses ReLU instead, but that’s a small detail you will conveniently ignore.

The math for this network is something like this:

output_dense_1 = relu(matmul(W_1, x) + b_1)
output_dense_2 = softmax(matmul(W_2, output_dense_1) + b_2)

For each layer, the pattern is the same: a matrix multiplication of the layer’s input with the weights, plus the bias, and an activation function applied to it. Repeated twice because you have two Dense layers.

Note: This model has 1.5 million parameters. That’s a lot for a model with just two layers. This is a downside of using Dense or fully-connected layers. Since each of the 3,072 inputs is connected to each of the 500 intermediate neurons, this requires 3,072×500 = 1.5 million connections, plus 500 bias values. Because this model has so many learned parameters, you can expect it to overfit again on the relatively small dataset.

As usual, don’t forget to compile the model first or you cannot train it:

model.compile(loss="categorical_crossentropy",
              optimizer=optimizers.Adam(lr=1e-3),
              metrics=["accuracy"])

Train this model by calling fit_generator(), and you’ll see that the results will be a little better than before:

model.fit_generator(train_generator,
                    steps_per_epoch=len(train_generator),
                    validation_data=val_generator,
                    validation_steps=len(val_generator),
                    epochs=3,
                    workers=4)

Training for three epochs gives the following results:

Epoch 1/3
76/76 [==============================] - 2s 28ms/step -
	loss: 3.2228 - acc: 0.1315 -
	val_loss: 3.1306 - val_acc: 0.1351
Epoch 2/3
76/76 [==============================] - 2s 24ms/step -
	loss: 2.4553 - acc: 0.2849 -
	val_loss: 3.0794 - val_acc: 0.1466
Epoch 3/3
76/76 [==============================] - 2s 27ms/step -
	loss: 2.0033 - acc: 0.4284 -
	val_loss: 3.1929 - val_acc: 0.1613

The validation score is a little better now than with the logistic regression model, so this new model has learned how to classify images a bit better — but it’s still nothing to write home about. The reason you’re only doing three epochs is that the validation score becomes worse if you train for longer because of overfitting.

See what it does on the test set:

model.evaluate_generator(test_generator,
                         steps=len(test_generator))

For the author, the output is about 0.15 or 15% correct. It’s better than a random guess, and better than the model with just a single Dense layer, but not by much. Adding more Dense layers might boost the validation and test scores by a little, but you’re still very far off from the accuracy scores you got from Create ML and Turi Create.

It should be clear by now that these classical methods, logistic regression and fully connected neural networks, just don’t work very well for image data. One reason is that the model you’ve created actually destroys the spatial nature of the training data.

Images have a width and height, but the first thing the model does is Flatten the image so it can be connected to a Dense layer. As you’ve seen, Flatten unrolls the original three-dimensional image data — height, width and color channels — into a one-dimensional vector. This destroys the relationships between neighboring pixels that was present in the original image. By doing this, you’ve unintentionally been making it hard on the model to understand our data.

It would be better if you could use a model that kept the spatial relationships intact, and that understood the true nature of images. That’s exactly what convolutional layers do. And that’s the topic of the next chapter!

Challenge

Challenge 1: Add layers to the neural network

Try adding more layers to the neural network, and varying the number of neurons inside these layers. Can you get a better test score this way? You’ll find that the more layers you add, the harder it actually becomes to train the model.

Key points

• Linear regression is one of the most basic machine-learning models, dating back to the 1800s when Gauss and others discovered the method of Ordinary Least Squares. It models the relationship between different variables. You can turn linear regression into logistic regression with the sigmoid function, making it a classifier model.

• To build a logistic regression classifier in Keras, you just need one Dense layer followed by softmax activation. To use images with the Dense layer, you need to Flatten the image data into a one-dimensional vector first.

• To train a model in Keras, you need to choose a loss function — cross-entropy for a classifier — as well as an optimizer. Setting the optimizer’s learning rate is important or the model won’t be able to learn anything.

• Load your data with ImageDataGenerator. Use a normalization function to give your data a mean of 0 and a standard deviation of 1. Choose a batch size that fits on your GPU — 32 or 64 is a good default choice.

• Be sure to check the loss and accuracy of your test set on the untrained model, to see if you get reasonable values. The accuracy should be approximately 1/num_classes, the loss should be close to np.log(num_classes).

• Keep your eye on the validation accuracy during training. If it stops improving while the training accuracy continues going up, your model is overfitting.

• A classical neural network is just two or more logistic regressions in a row.

• Logistic regression and classical feed-forward neural networks are not the best choice for building image classifiers.