Building a Camera App With SwiftUI and Combine

How AVCaptureSession Works

Whenever you want to capture some sort of media — whether it’s audio, video or depth data — AVCaptureSession is what you want.

The main pieces to setting up a capture session are:

• AVCaptureDevice: a representation of the hardware device to use.
• AVCaptureDeviceInput: provides a bridge from the device to the AVCaptureSession.
• AVCaptureSession: manages the flow of data between capture inputs and outputs. It can connect one or more inputs to one or more outputs.
• AVCaptureOutput: an abstract class representing objects that output the captured media. You’ll use AVCaptureVideoDataOutput, which is a concrete implementation of this class.

Apple’s awesome flowchart explains this nicely:

This shows a high-level view of how each of these pieces fit together. In your app, you’ll be using these parts to configure the capture session.

Configuring the Capture Session

For anyone who has never used a capture session, it can be a little daunting to set one up the first time. Each time you do it, though, it gets a little easier and makes more sense. Fortunately, you usually need to configure a capture session just once in your app — and this app is no exception.

Add the following code to CameraManager:

private func configureCaptureSession() {
  guard status == .unconfigured else {
    return
  }
  session.beginConfiguration()
  defer {
    session.commitConfiguration()
  }
}

So far, this is pretty straightforward. But it’s worth noting that any time you want to change something about an AVCaptureSession configuration, you need to enclose that code between a beginConfiguration and a commitConfiguration.

At the end of configureCaptureSession(), add the following code:

let device = AVCaptureDevice.default(
  .builtInWideAngleCamera,
  for: .video,
  position: .front)
guard let camera = device else {
  set(error: .cameraUnavailable)
  status = .failed
  return
}

This code gets your capture device. In this app, you’re getting the front camera. If you want the back camera, you can change position. Since AVCaptureDevice.default(_:_:_:) returns an optional, which will be nil if the requested device doesn’t exist, you need to unwrap it. If for some reason it is nil, set the error and return early.

After the code above, add the following code to add the device input to AVCaptureSession:

do {
  // 1
  let cameraInput = try AVCaptureDeviceInput(device: camera)
  // 2
  if session.canAddInput(cameraInput) {
    session.addInput(cameraInput)
  } else {
    // 3
    set(error: .cannotAddInput)
    status = .failed
    return
  }
} catch {
  // 4
  set(error: .createCaptureInput(error))
  status = .failed
  return
}

Here, you:

1. Try to create an AVCaptureDeviceInput based on the camera. Since this call can throw, you wrap the code in a do-catch block.
2. Add the camera input to AVCaptureSession, if possible. It’s always a good idea to check if it can be added before adding it. :]
3. Otherwise, set the error and the status and return early.
4. If an error was thrown, set the error based on this thrown error to help with debugging and return.

Have you noticed that camera management involves a lot of error management? When there are so many potential points of failure, having good error management will help you debug any problems much more quickly! Plus, it’s a significantly better user experience.

Next up, you need to connect the capture output to the AVCaptureSession!

Add the following code right after the code you just added:

// 1
if session.canAddOutput(videoOutput) {
  session.addOutput(videoOutput)
  // 2
  videoOutput.videoSettings =
    [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA]
  // 3
  let videoConnection = videoOutput.connection(with: .video)
  videoConnection?.videoOrientation = .portrait
} else {
  // 4
  set(error: .cannotAddOutput)
  status = .failed
  return
}

With this code:

1. You check to see if you can add AVCaptureVideoDataOutput to the session before adding it. This pattern is similar to when you added the input.
2. Then, you set the format type for the video output.
3. And force the orientation to be in portrait.
4. If something fails, you set the error and status and return.

Finally, there’s one last thing you need to add to this method before it’s finished — right before the closing brace, add:

status = .configured

And with that, your capture session can be configured by calling configureCaptureSession()!

You’ll do that now.

Camera Manager Final Touches

There are a couple of small things you need to take care of to hook all the camera logic together.

Remember that configure() you initially added with the class definition? It’s time to fill that in. In CameraManager, add the following code to configure():

checkPermissions()
sessionQueue.async {
  self.configureCaptureSession()
  self.session.startRunning()
}

Now, check for permissions, configure the capture session and start it. All of this happens when CameraManager is initialized. Perfect!

The only question is: How do you get captured frames from this thing?

You’ll create a FrameManager, which will receive delegate calls from AVCaptureVideoDataOutput. However, before you can do that, you need to add one last method to CameraManager:

func set(
  _ delegate: AVCaptureVideoDataOutputSampleBufferDelegate,
  queue: DispatchQueue
) {
  sessionQueue.async {
    self.videoOutput.setSampleBufferDelegate(delegate, queue: queue)
  }
}

Using this method, your upcoming frame manager will be able to set itself as the delegate that receives that camera data.

Pat yourself on the back and take a quick break! You just completed the longest and most complicated class in this project. It’s all smooth sailing from now on!

Next, you’ll write a class that can receive this camera data.

Designing the Frame Manager

FrameManager will be responsible for receiving data from CameraManager and publishing a frame for use elsewhere in the app. This logic could technically be integrated into CameraManager, but splitting these up also divides the responsibility a bit more clearly. This will be especially helpful if you choose to add more functionality — such as preprocessing or synchronization — to FrameManager in the future.

Add a new Swift file named FrameManager.swift in the Camera group. Replace the contents of the file with the following:

import AVFoundation
// 1
class FrameManager: NSObject, ObservableObject {
  // 2
  static let shared = FrameManager()
  // 3
  @Published var current: CVPixelBuffer?
  // 4
  let videoOutputQueue = DispatchQueue(
    label: "com.raywenderlich.VideoOutputQ",
    qos: .userInitiated,
    attributes: [],
    autoreleaseFrequency: .workItem)
  // 5
  private override init() {
    super.init()
    CameraManager.shared.set(self, queue: videoOutputQueue)
  }
}

With this initial implementation, you:

1. Define the class and have it inherit from NSObject and conform to ObservableObject. FrameManager needs to inherit from NSObject because FrameManager will adopt AVCaptureSession‘s video output. This is a requirement, so you’re just getting a head start on it.
2. Make the frame manager a singleton.
3. Add a published property for the current frame received from the camera. This is what other classes will subscribe to to get the camera data.
4. Create a queue on which to receive the capture data.
5. Set FrameManager as the delegate to AVCaptureVideoDataOutput.

Right about now, Xcode is probably complaining that FrameManager doesn’t conform to AVCaptureVideoDataOutputSampleBufferDelegate. That’s kind of the point!

To fix this, add the following extension below the closing brace of FrameManager:

extension FrameManager: AVCaptureVideoDataOutputSampleBufferDelegate {
  func captureOutput(
    _ output: AVCaptureOutput,
    didOutput sampleBuffer: CMSampleBuffer,
    from connection: AVCaptureConnection
  ) {
    if let buffer = sampleBuffer.imageBuffer {
      DispatchQueue.main.async {
        self.current = buffer
      }
    }
  }
}

In this app, you’re checking if the received CMSampleBuffer contains an image buffer and then sets the current frame. Once again, since current is a published property, it needs to be set on the main thread. That’s that. Short and simple.

You’re close to being able to see the fruits of your oh-so-hard labor. You just need to hook this FrameManager up to your FrameView somehow. But to do that, you’ll need to create the most basic form of view model first.