Instruction 1

In the previous lesson, you set up a Python environment for the conversion of a PyTorch model into CoreML. You might wonder why you need to go through that much effort. Machine learning projects often come with specific version requirements. Having isolated environments keeps projects from interfering with each other but does little to solve the need to identify and satisfy these interdependencies.

First, you’ll look at converting a PyTorch segmentation model to CoreML. An image segmentation model divides an image into multiple parts or regions, each of which is considered the same class. A model can define a class by anything from color to attempting to detect the objects within the image. Most current research focuses on identifying the objects in the image and their locations within the image.

Open a terminal and enter the CoreML Tools environment that you created in the last lesson by entering conda activate coremltools. Now, create a new folder for this lesson and go into that folder by entering:

mkdir lesson-two
cd lesson-two

Now open this lesson’s starter project and find the sample-image folder. Copy the sample-image.jpg file from that folder to the lesson-two folder you just created. You may also want to copy the wrappedresnet.py file, as you’ll use it later in this section. Check the image, you’ll see a photo of two cats lying on a bed. You’ll use this image to explore the library both alone and within the app that you’ll develop in this module.

Python can be run either by executing a script or interactively by entering commands one at a time. You’ll use that approach to explore the library and convert it to CoreML. To begin, enter the following at the terminal:

python

This will start the Python interpreter. Since you specified version 3.11 when creating the environment, conda installed the latest 3.11 version, 3.11.9. The interpreter will print the version information, show a few commands, and then give you the Python prompt of three greater-than symbols (>>>).

Now, import torch and coremltools by executing:

import torch
import coremltools as ct

The import command tells Python to load the indicated modules. These were installed when you installed the coremltools python package in the previous lesson.

Early versions of CoreML Tools required an intermediate step in converting the model to the open ONNX format. Now, you can do the conversion directly by converting the model to a Torchscript format and then using the converter to change the Torchscript model into a CoreML model.

Enter the following:

pt_model = \
  torch.hub.load('pytorch/vision:v0.6.0', 'deeplabv3_resnet101', \
  pretrained=True)

The \ characters in the command split the line for readability by telling the Python interpreter that the command continues on the next line. You ask PyTorch for the deeplabv3_resnet101 model with pre-trained weights so it’s ready for use. The fir`st time you run this, Python will download the 170 MB model to your computer and store it in a cache for future use.

Enter the following at the prompt:

model = pt_model.eval()

This sets the downloaded model to evaluation mode, which turns off operations that are useful during training. It also produces more consistent and reliable model behavior. Think of it as setting the model ready for use as opposed to working on the model itself. This setting results in a more efficient model when you perform the conversion.

One more step.

To convert the model to CoreML, you need to convert it to TorchScript format, which removes the dependence on Python. You do this by running the model through PyTorch’s torch.jit.trace JIT Tracer. Tracing runs an example input through the model and captures the operations as that input makes its way through the model’s layers and produces TorchScript. This means you need both sample input and a model that matches the expected output for Python’s JIT model.

You can load a real image, but an image of random or zero data for conversion will also suffice. Enter the following at the Python prompt:

im = torch.zeros(1, 3, 640, 640)

If you traced the model right now using torch.jit.trace, you’ll get an error that the output of the model doesn’t match the supported outputs for Python’s JIT module. The model returns a dictionary, while the module only supports returning tensors. You can fix this by wrapping the model inside a class to extract only the tensor data in the output. Enter the following one line at a time. Note that spacing in Python is important as blocks are defined by spacing, similar to how curly braces are in Swift.

class WrappedResnet(torch.nn.Module):
  def __init__(self):
    super(WrappedResnet, self).__init__()
    self.model = torch.hub.load('pytorch/vision:v0.6.0', 'deeplabv3_resnet101', pretrained=True).eval()
  def forward(self, x):
    res = self.model(x)
    x = res["out"]
    return x

The initializer simply loads the original model and sets it to eval mode. The forward(self, x) method extracts the “out” element from the dictionary, which contains the tensor and returns only that, satisfying the expectations of the JIT module.

Note: If you have problems getting the class defined in interactive mode, you can load the class from the starter project by entering exec(open("/path to starter project/wrappedresnet.py").read()) at the Python prompt, replacing path to starter project with the location of the project on your computer.

Now you can trace the model. Enter each of the following one line at a time:

traceable_model = WrappedResnet().eval()
trace = torch.jit.trace(traceable_model, im)

This will load the wrapped model and start tracing it. This should only take a few seconds. You’re finally ready to convert using the convert() method or CoreML Tools. Enter the following at the Python prompt:

mlmodel = ct.convert(
  trace,
  inputs=[ct.TensorType(name="input", shape=im.shape)]
)

The first parameter contains the traced model, which is now in TorchScript format. The second defines the input to the model. Now, to save the model, enter:

mlmodel.save("resnet101.mlpackage")

The model is now ready for use in your Xcode projects. Before getting there, you’ll look at some other options to convert models.

Exit Python by entering:

exit()