How to Play, Record and Merge Videos in iOS and Swift

Orienting Video

AVAsset has a preferredTransform that contains the media orientation information. It applies this to a media file whenever you view it using the Photos app or QuickTime.

In the code above, you haven’t applied a transform to your AVAssets, hence the orientation issue. Fortunately, this is an easy fix.

Before you can do it, however, you need to add the following helper method to VideoHelper in VideoHelper.swift:

static func orientationFromTransform(
  _ transform: CGAffineTransform
) -> (orientation: UIImage.Orientation, isPortrait: Bool) {
  var assetOrientation = UIImage.Orientation.up
  var isPortrait = false
  let tfA = transform.a
  let tfB = transform.b
  let tfC = transform.c
  let tfD = transform.d

  if tfA == 0 && tfB == 1.0 && tfC == -1.0 && tfD == 0 {
    assetOrientation = .right
    isPortrait = true
  } else if tfA == 0 && tfB == -1.0 && tfC == 1.0 && tfD == 0 {
    assetOrientation = .left
    isPortrait = true
  } else if tfA == 1.0 && tfB == 0 && tfC == 0 && tfD == 1.0 {
    assetOrientation = .up
  } else if tfA == -1.0 && tfB == 0 && tfC == 0 && tfD == -1.0 {
    assetOrientation = .down
  }
  return (assetOrientation, isPortrait)
}

This code analyzes an affine transform to determine the input video’s orientation.

Next, add the following import:

import AVFoundation

and one more helper method to the class:

static func videoCompositionInstruction(
  _ track: AVCompositionTrack,
  asset: AVAsset
) -> AVMutableVideoCompositionLayerInstruction {
  // 1
  let instruction = AVMutableVideoCompositionLayerInstruction(assetTrack: track)

  // 2
  let assetTrack = asset.tracks(withMediaType: AVMediaType.video)[0]

  // 3
  let transform = assetTrack.preferredTransform
  let assetInfo = orientationFromTransform(transform)

  var scaleToFitRatio = UIScreen.main.bounds.width / assetTrack.naturalSize.width
  if assetInfo.isPortrait {
    // 4
    scaleToFitRatio = UIScreen.main.bounds.width / assetTrack.naturalSize.height
    let scaleFactor = CGAffineTransform(
      scaleX: scaleToFitRatio,
      y: scaleToFitRatio)
    instruction.setTransform(
      assetTrack.preferredTransform.concatenating(scaleFactor),
      at: .zero)
  } else {
    // 5
    let scaleFactor = CGAffineTransform(
      scaleX: scaleToFitRatio,
      y: scaleToFitRatio)
    var concat = assetTrack.preferredTransform.concatenating(scaleFactor)
      .concatenating(CGAffineTransform(
        translationX: 0,
        y: UIScreen.main.bounds.width / 2))
    if assetInfo.orientation == .down {
      let fixUpsideDown = CGAffineTransform(rotationAngle: CGFloat(Double.pi))
      let windowBounds = UIScreen.main.bounds
      let yFix = assetTrack.naturalSize.height + windowBounds.height
      let centerFix = CGAffineTransform(
        translationX: assetTrack.naturalSize.width,
        y: yFix)
      concat = fixUpsideDown.concatenating(centerFix).concatenating(scaleFactor)
    }
    instruction.setTransform(concat, at: .zero)
  }

  return instruction
}

This method takes a track and an asset and returns a AVMutableVideoCompositionLayerInstruction which wraps the affine transform needed to get the video right-side up. Here’s what’s going on, step-by-step:

Because there are two landscapes, the aspect ratio will match but the video might be rotated 180 degrees. The extra check for a video orientation of .down handles this case.

1. You create AVMutableVideoCompositionLayerInstruction and associate it with your track.
2. Next, you create AVAssetTrack from your AVAsset. An AVAssetTrack provides the track-level inspection interface for all assets. You need this object to access the dimensions of the asset and preferredTransform.
3. Then, you save the preferred transform and the amount of scale required to fit the video to the current screen. You’ll use these values in the following steps.
4. If the video is in portrait, you need to recalculate the scale factor — the default calculation is for videos in landscape. All you need to do then is apply the orientation rotation and scale transforms.
5. If the video is in landscape, there’s a similar set of steps to apply the scale and transform. However, there’s one extra check, because the user could have produced the video in either landscape left or landscape right.

   Because there are two landscapes, the aspect ratio will match but the video might be rotated 180 degrees. The extra check for a video orientation of .down handles this case.

With the helper methods set up, find merge(_:) in MergeVideoViewController.swift. Locate where firstInstruction and secondInstruction are created and replace them with the following:

let firstInstruction = VideoHelper.videoCompositionInstruction(
  firstTrack,
  asset: firstAsset)

let secondInstruction = VideoHelper.videoCompositionInstruction(
   secondTrack,
   asset: secondAsset)

The changes above will use the new helper functions and implement the rotation fixes you need.

Whew — that’s it!

Build and run. Create a new video by combining two videos — and, optionally an audio file — and you’ll see that the orientation issues disappear when you play it back.

Where to Go From Here?

Download the final project using the Download Materials link at the top or bottom of this tutorial.

You should now have a good understanding of how to play video, record video and merge multiple videos and audio in your apps.

AV Foundation gives you a lot of flexibility when playing with videos. You can also apply any kind of CGAffineTransform to merge, scale or position videos.

If you haven’t already done so, take a look at the WWDC videos on AV Foundation, such as WWDC 2016 session 503, Advances in AVFoundation Playback.

Also, be sure to check out the Apple AVFoundation Framework documentation.

I hope this tutorial has been useful to get you started with video manipulation in iOS. If you have any questions, comments or suggestions for improvement, please join the forum discussion below!