Core Image Tutorial for iOS: Custom Filters

Displaying a Built-In Filter’s Output

Add the following to the end of applyBuiltInEffect(input:):

if let outputImage = renderAsUIImage(compositeImage) {
  output = outputImage
}

This converts compositeImage, which is a CIImage, to a UIImage using renderAsUIImage(_:). You then save the result to output.

Add the following new method to ImageProcessor:

// 1
func process(painting: Painting, effect: ProcessEffect) {
  // 2
  guard
    let paintImage = UIImage(named: painting.image),
    let input = CIImage(image: paintImage)
  else {
    print("Invalid input image")
    return
  }
  switch effect {
  // 3
  case .builtIn:
    applyBuiltInEffect(input: input)
  default:
    print("Unsupported effect")
  }
}

Here, you:

1. Create a method that acts as an entry point to ImageProcessor. It takes an instance of Painting and an effect to apply.
2. Check for a valid image.
3. If the effect is of type .builtIn, you call applyBuiltInEffect(input:) to apply the filter.

Open PaintingWall.swift. Below selectedPainting = paintings[index] in the action closure of Button, add:

var effect = ProcessEffect.builtIn
if let painting = selectedPainting {
  switch index {
  case 0:
    effect = .builtIn
  default:
    effect = .builtIn
  }
  ImageProcessor.shared.process(painting: painting, effect: effect)
}

Here, you set the effect to .builtIn for the first painting. You also set it as the default effect. Then you apply the filter by calling process(painting:, effect:) on ImageProcessor.

Build and run. Tap the “Mona Lisa”. You’ll see a built-in filter applied in the output!

Great job making the sun shine on Mona Lisa. No wonder she’s smiling! Now it’s time to create your filter using CIKernel.

Meet CIKernel

With CIKernel, you can put in place custom code, called a kernel, to manipulate an image pixel by pixel. The GPU processes these pixels. You write kernels in the Metal Shading Language, which offers the following advantages over the older Core Image Kernel Language, deprecated since iOS 12:

• Supports all the great features of Core Image kernels like concatenation and tiling.
• Comes pre-compiled at build-time with error diagnostics. This way, you don’t need to wait for runtime for errors to appear.
• Offers syntax highlighting and syntax checking.

There are different types of kernels:

• CIColorKernel: Changes the color of a pixel but doesn’t know the pixel’s position.
• CIWarpKernel: Changes the position of a pixel but doesn’t know the pixel’s color.
• CIBlendKernel: Blends two images in an optimized way.

To create and apply a kernel, you:

1. First, add custom build rules to the project.
2. Then, add the Metal source file.
3. Load the kernel.
4. Finally, initialize and apply the kernel.

You’ll implement each of these steps next. Get ready for a fun ride!

Creating Build Rules

You need to compile the Core Image Metal code and link it with special flags.

Select the RayVinci target in the Project navigator. Then, select the Build Rules tab. Add a new build rule by clicking +.

Then, set up the first new build rule:

This calls the Metal compiler with the required -fcikernel flag.

This produces an output binary that ends in .ci.air.

1. Set Process to Source files with name matching:. Then set *.ci.metal as the value.
2. Uncheck Run once per architecture.
3. Add the following script:

   xcrun metal -c -fcikernel "${INPUT_FILE_PATH}" \
     -o "${SCRIPT_OUTPUT_FILE_0}"
   

   This calls the Metal compiler with the required -fcikernel flag.

4. Add the following in Output Files:

   $(DERIVED_FILE_DIR)/${INPUT_FILE_BASE}.air

   This produces an output binary that ends in .ci.air.

xcrun metal -c -fcikernel "${INPUT_FILE_PATH}" \
  -o "${SCRIPT_OUTPUT_FILE_0}"

$(DERIVED_FILE_DIR)/${INPUT_FILE_BASE}.air

Next, add another new build rule by clicking + again.

Follow these steps for the second new build rule:

This calls the Metal linker with the required -cikernel flag.

This produces a file ending with .ci.metallib in the app bundle.

1. Set Process to Source files with name matching:. Then set *.ci.air as the value.
2. Uncheck Run once per architecture.
3. Add the following script:

   xcrun metallib -cikernel "${INPUT_FILE_PATH}" -o "${SCRIPT_OUTPUT_FILE_0}"
   

   This calls the Metal linker with the required -cikernel flag.

4. Add the following in Output Files:

   $(METAL_LIBRARY_OUTPUT_DIR)/$(INPUT_FILE_BASE).metallib

   This produces a file ending with .ci.metallib in the app bundle.

xcrun metallib -cikernel "${INPUT_FILE_PATH}" -o "${SCRIPT_OUTPUT_FILE_0}"

$(METAL_LIBRARY_OUTPUT_DIR)/$(INPUT_FILE_BASE).metallib

Next, it’s time to add the Metal source.

Adding the Metal Source

First, you’ll create a source file for a color kernel. In the Project navigator, highlight RayVinci right under the RayVinci project.

Right-click and choose New Group. Name this new group Filters. Then, highlight the group and add a new Metal file named ColorFilterKernel.ci.metal.

Open the file and add:

// 1
#include <CoreImage/CoreImage.h>
 
// 2
extern "C" {
  namespace coreimage {
    // 3
    float4 colorFilterKernel(sample_t s) {
      // 4
      float4 swappedColor;
      swappedColor.r = s.g;
      swappedColor.g = s.b;
      swappedColor.b = s.r;
      swappedColor.a = s.a;
      return swappedColor;
    }
  }
}

Here’s a code breakdown:

1. Including the Core Image header lets you access the classes the framework provides. This automatically includes the Core Image Metal Kernel Library, CIKernelMetalLib.h.
2. The kernel needs to be inside an extern "C" enclosure to make it’s accessible by name at runtime. Next, you specify the namespace of coreimage. You declare all the extensions in the coreimage namespace to avoid conflicts with Metal.
3. Here, you declare colorFilterKernel, which takes an input of type sample_t. sample_t represents a single color sample from an input image. colorFilterKernel returns a float4 that represents the RGBA value of the pixel.
4. Then, you declare a new float4, swappedColor, and swap the RGBA values from the input sample. You then return the sample with the swapped values.

Next, you’ll write the code to load and apply the kernel.

Loading the Kernel Code

To load and apply a kernel, start by creating a subclass of CIFilter.

Create a new Swift file in the Filters group. Name it ColorFilter.swift and add:

// 1
import CoreImage

class ColorFilter: CIFilter {
  // 2
  var inputImage: CIImage?

  // 3
  static var kernel: CIKernel = { () -> CIColorKernel in
    guard let url = Bundle.main.url(
      forResource: "ColorFilterKernel.ci",
      withExtension: "metallib"),
      let data = try? Data(contentsOf: url) else {
      fatalError("Unable to load metallib")
    }

    guard let kernel = try? CIColorKernel(
      functionName: "colorFilterKernel",
      fromMetalLibraryData: data) else {
      fatalError("Unable to create color kernel")
    }

    return kernel
  }()

  // 4
  override var outputImage: CIImage? {
    guard let inputImage = inputImage else { return nil }
    return ColorFilter.kernel.apply(
      extent: inputImage.extent,
      roiCallback: { _, rect in
        return rect
      },
      arguments: [inputImage])
  }
}

Here, you:

You pass the entire image, so the filter will apply to the entire image. roiCallback determines the rect of the input image needed to render the rect in outputImage. Here, the rect of inputImage and outputImage doesn’t change, so you return the same value and pass the inputImage in the arguments array to the kernel.

1. Start by importing the Core Image framework.
2. Subclassing CIFilter involves two main steps:
   • Specifying the input parameters. Here, you use inputImage.
   • Overriding outputImage.
3. Then, you declare a static property, kernel, that loads the contents of the ColorFilterKernel.ci.metallib. This way, the library loads only once. You then create an instance of CIColorKernel with the contents of the ColorFilterKernel.ci.metallib.
4. Next, you override outputImage. Here, you apply the kernel by using apply(extent:roiCallback:arguments:). The extent determines how much of the input image gets passed to the kernel.

   You pass the entire image, so the filter will apply to the entire image. roiCallback determines the rect of the input image needed to render the rect in outputImage. Here, the rect of inputImage and outputImage doesn’t change, so you return the same value and pass the inputImage in the arguments array to the kernel.

Now that you’ve created the color kernel filter, you’ll apply it to an image.