Core Image: From CIImage to Metal and Beyond
Core Image is an oft-overlooked first-party framework on Apple platforms for the efficient processing of images and videos. It includes a wide range of built-in filters, as well as the ability to build your own custom filters using Metal. Discover how to integrate Core Image into your own apps, and how the optimized image processing can save your users time and battery life. Learn how to perform real-time video processing, and how you can extend Core Image by implementing your own custom filters directly in Metal. By Sam Davies.
Who is this for?
iOS, macOS and tvOS developers that have heard of and may have used Core Image in the past, and are interested in learning more about this powerful framework.
Covered concepts
- The Core Image Pipeline: image input, processing & output. Why is it efficient?
- Common issues with importing, displaying and exporting images using Core Image.
- Using built-in filters, and the new iOS 13 API.
- Creating custom composite filters.
- Using image mattes to perform segmented image processing.
- Creating custom filters using Metal.
- Color, warp and general kernels.
- Video processing with Core Image.
- How to perform real-time processing of a camera stream.
Part 1: The Core Image Pipeline
Create a CIImage from a file, discover how to handle rotation and see how to import a depth map in a portrait photo.
Instantiate a CIFilter and apply it to an image. Compare the string-based filter instantiation to the new iOS 13 static method approach.
Integrate multiple filters in a chain to create complex image processing effects. See how Core Image efficiently concatenates kernels.
Display the result of the Core Image pipeline in a UIImageView. Discover how to cope with issues with aspect ratio, by creating a CGImage.
Save the output from a Core Image pipeline into a file, using a Core Image context.
Part 2: Built-in Filters
List the categorized built-in filters, their documentation and available parameters.
Photos taken in portrait mode have powerful segmentation algorithms applied, splitting the foreground from the background. Apply background blurring with a single filter.
Use an image matte to selectively apply a complex composite chain of filters to the background of an image.
Apply a masked filter to the foreground of the image, and composite the foreground and background back together.
Use a Core Image generator filter to create a fake sun and then a blend filter to insert it into your photographic masterpiece.
Part 3: Creating Custom Filters with Metal
Filter kernels come in three forms, and represent highly optimised image processing routines that are performed on the GPU. Create your first kernel—for a passthrough filter.
To use a metal kernel, you must access it from inside a CIFilter. See how to create a filter that utilizes your freshly forged kernel.
Color kernels manipulate pixel values independent of spatial location. You can use them to change a colorspace.
Custom kernels give you the full power to create any filter you can imagine. See how to build a bilateral filter in Core Image.
Combining custom filters is very similar to combining built-in filters, and uses the same kernel optimization techniques. Combine the color space transform and bilateral filter kernel for some hardcore image processing!
Part 4: Video Processing with Core Image
Set up an AVFoundation session to access the camera and stream frames into Core Image, by creating a Core Image camera object.
See how to create a Metal-backed view that is designed to display video frames once they've been processed.
Create a simple Core Image pipeline and use it to process video frames from the camera in real time.