9. RealityKit

Written by Chris Language

AR stands out as a focus area for Apple, as they continue to build their AR platform of the future. Thanks to AR Quick Look, AR has become extremely accessible and is now deeply integrated into iOS, macOS and tvOS.

Creating immersive AR experiences has historically been difficult, requiring a vast amount of skill and knowledge. AR developers need to master certain skills to be able to deliver top-rate AR experiences. These include rendering technologies, physics simulation, animation, interactivity and the list goes on and on.

Thankfully, that all changed with the introduction of RealityKit.

With RealityKit in your toolbox, creating AR experiences has never been easier.

In this section, you’ll learn all about RealityKit and face tracking. You’ll create a SnapChat-like face filter app with SwiftUI called AR Funny Face, where you get to mock up your face with funny props. You’ll also create an animated mask that you can control with your eyes, brows and mouth.

What is RealityKit?

RealityKit is a new Swift framework that Apple introduced at WWDC 2019. Apple designed it from the ground up with AR development in mind. Its main purpose is to help you build AR apps and experiences more easily. Thanks to the awesome power of Swift, RealityKit delivers a high-quality framework with a super simple API.

RealityKit is a high-quality rendering technology capable of delivering hyper-realistic, physically-based graphics with precise physics simulation and collisions against the real-world environment. It does all of the heavy lifting for you, right out of the box. It makes your content look as good as possible while fitting seamlessly into the real world. It’s impressive feature list includes skeletal animations, realistic shadows, lights, reflections and post-processing effects.

Are you ready to give it a try? Open RealityKit and take a look at what’s inside.

At its core, you’ll find many of Apple’s other frameworks, but the ones doing most of the work are ARKit and Metal.

Here’s a breakdown of RealityKit’s coolest features:

• Rendering: RealityKit offers a powerful new physically-based renderer built on top of Metal, which is fully optimized for all Apple devices.

• Animation: It has built-in support for Skeletal animation and Transform-based animation. So, if you want, you can animate a zombie or you can move, scale and rotate objects with various easing functions.

• Physics: With a powerful physics engine, RealityKit lets you throw anything at it — pun intended! You can adjust real-world physics properties like mass, drag and restitution, allowing you to fine-tune collisions.

• Audio: Spacial audio understanding and automatic listener configuration let you attach sound effects to 3D objects. You can then track those sounds, making them sound realistic based on their position in the real world.

• ECS: From a coding perspective, RealityKit enforces the Entity Component System design pattern to build objects within the world.

• Synchronization: The framework has built-in support for networking, designed for collaborative experiences. It even offers automatic synchronization of entities between multiple clients.

Enough talk, it’s time to dive into some code!

Creating a RealityKit project

Now that you have some understand about RealityKit’s features, you’ll create your first RealityKit project. Launch Xcode and get ready to create a new Augmented Reality App project from scratch.

Note: If you’d rather skip creating the project from scratch and use the starter project instead — which also includes the app icons — you can load it from starter/ARFunnyFace.xcodeproj. Feel free to skip to the next section.

Choose the following template iOS ▸ Augmented Reality App:

Click Next to continue. Set the Product Name to ARFunnyFace and fill in the rest of the information.

Xcode now has a newly-added RealityKit option under Content Technologies. Make sure to choose SwiftUI for your User Interface:

Click Next and save the project somewhere safe. Xcode will continue to generate the project for you, with the end result looking like this:

Before doing anything else, build and run the project to give it a quick try before taking a look at what Xcode generated.

Wow, out of the box your app is already doing quite a bit. It’s requesting access to the camera, it’s detecting horizontal surfaces and it has proper environmental lighting and reflections. And where did that smooth-looking cube come from? You didn’t have to write one single line of code to achieve all that. Nice!

Well, you’ll be even more impressed when you look at what’s inside the project.

Reviewing the project

At first glance within the project, you’ll notice the usual suspects — but there are a few new things, too:

• AppDelegate.swift: This is the app’s starting point.

• ContentView.swift: Since this is a SwiftUI-based app, the user interface is defined here. From the preview, you can see that the UI is currently a blank slate. Internally, the ContentView constructs an ARView that loads and presents the scene located within the Experience.rcproject Reality Composer project. It’s also important to point out that this file updates ARView.

• Experience.rcproject: This is a Reality Composer project, which is essentially a 3D scene that contains the box and the box anchor you used in the previous step.

• Assets.xcassets: This contains all of your project assets, like images and app icons.

• LaunchScreen.storyboard: Here, you’ll find the UI the user sees while your app is launching.

• Info.plist: Contains the app’s basic configuration settings. Note that there’s already a Camera Usage Description property, you just need to change it to something more appropriate for your app. This allows the app to request access to the camera from the user, which you need to deliver the AR experience through the camera view.

RealityKit API components

Now, take a look at a few main components that form parts of the RealityKit API.

Here’s an example of a typical structure containing all of the important elements:

• ARView: The ARView sits at the core of any RealityKit experience, taking responsibility for all of the heavy lifting. It comes with full gesture support, allowing you to attach gestures to entities. It also handles the post-processing camera effects, which is very similar to the effects you saw in AR Quick Look.

• Scene: Think of this as the container for all of your entities.

• Anchor: RealityKit exposes ARKit’s available anchors — plane, face, body, image and object — as first-class citizens. Anchors form the local root for entity structures. Note that content attached to an anchor will stay hidden until you successfully identify it and connect it to the real world.

• Entity: You can picture each element of the virtual content in a scene as an entity — the basic building block of your experience. You can establish a tree-like hierarchical structure by parenting entities to other entities.

• Components: Entities consist of different types of components. These components give the entities specific features and functionality, like how they look, how they respond to collisions and how they react to physics.

Building the UI with SwiftUI

When you created the app, you selected SwiftUI for the user interface. Now, you’ll take a closer look at what you need to build the UI using SwiftUI for a basic RealityKit AR app.

The UI is very simple, and it requires only three basic buttons: Next, Previous and Shutter.

You’ll use the Next and Previous buttons to switch between various AR scenes, while the Shutter button will take the all-important selfie. But your first order of business is to learn how to track the active prop by implementing the Next and Previous buttons.

Tracking the active prop

Your AR experience is going to contain multiple scenes with various props to make your pictures funnier. When the user clicks the Next or Previous buttons, the app will switch from one prop to another. You’ll implement that functionality now.

Open ContentView.swift and define a variable to keep track of the active prop by adding the following line of code at the top of ContentView:

@State var propId: Int = 0

@State indicates that SwiftUI will manage propId’s storage. When the state value changes, the view invalidates its appearance, which will recompute the body. State variables are the single source of truth for the view.

Add the following line of code to the top of ARViewContainer:

@Binding var propId: Int

@Binding creates a two-way connection between the property that stores the data and the view that changes and displays the data.

You now need to pass in propId as a parameter to ARViewContainer() to clear the error and complete the binding process.

Find the following line of code within ContentView:

return ARViewContainer().edgesIgnoringSafeArea(.all)

Replace it with the following code block:

// 1
ZStack(alignment: .bottom) {
  // 2
  ARViewContainer(propId: $propId).edgesIgnoringSafeArea(.all)
  // 3
  HStack {
  }
}

Take a look at what’s happening here:

1. To overlay the UI buttons on the AR view, you place the elements into a ZStack.
2. You provide the $propId as a parameter for ARViewContainer(), completing the binding process. So when the value of propId changes, it invalidates the ARView.
3. Finally, you stack the buttons horizontally within an HStack.

Great! You’ve now created a variable to keep track of the active prop. You’ll update this variable when the user presses the Next and Previous buttons to swap between the various scenes within the Reality Composer experience.

Note: This will momentarily cause a compiler error. Ignore this for now, you’ll fix the problem in the next section.

Adding buttons

The buttons all use images. So next, you’ll add the required images to the project by dragging and dropping all the image files from starter/resources/images into Assets.xcassets.

Under the Properties panel, be sure to set Image Set ▸ Render As to Original Image. Otherwise, the images will display with a blue highlight.

Now, you can reference these images within ContentView.swift.

Starting with the Previous button, add the following line of code inside the HStack:

Button(action: {
  self.propId = self.propId <= 0 ? 0 : self.propId - 1
}) {
  Image("PreviousButton").clipShape(Circle())
}

This creates a Button view with a defined action (and clears the error). When the user presses the button, the value of propId decreases by 1, but the value will never decrease below 0.

You also place an Image view within the Button view, using the image reference named PreviousButton, which you clip into a circular shape.

As you did with the Previous button, add the following line of code for the Shutter button:

Button(action: {
  //self.TakeSnapshot()
}) {
  Image("ShutterButton").clipShape(Circle())
}

This code works nearly the same in the Previous button. The only difference is that the action makes a call to a function named self.TakeSnapShot(), which you’ll define at a later stage. Leave it commented out for now.

Finally, add the following code for the Next button:

Button(action: {
  self.propId = self.propId >= 2 ? 2 : self.propId + 1
}) {
  Image("NextButton").clipShape(Circle())
}

This also operates like the Previous button. The difference is that the action will increase the value of the propId by 1, but the value can’t exceed 2.

The end result should look like the preview on the right:

To space the buttons more evenly, add a Spacer() before, between and after each button:

Spacer()

There should be a total of four spacers when you’re done. This spreads the UI elements evenly across the HStack.

The final result will look like this:

Excellent, you’ve now defined all of the UI elements. Now it’s time to add some functionality to the Shutter button.

Taking selfies

What good is an AR face mockup app if you can’t use it to take selfies?

For quick access to the ARView, add the following declaration to the top of ContentView.swift:

var arView: ARView!

Find the following line of code within ARViewContainer:

let arView = ARView(frame: .zero)

Then simply remove the let keyword so it looks like this:

arView = ARView(frame: .zero)

This initializes the quick-accessible arView variable instead of a local one.

Next, add the following helper function to ContentView:

func TakeSnapshot() {
  // 1
  arView.snapshot(saveToHDR: false) { (image) in
    // 2
    let compressedImage = UIImage(
      data: (image?.pngData())!)
    // 3
    UIImageWriteToSavedPhotosAlbum(
      compressedImage!, nil, nil, nil)
  }
}

Wow, is that all it takes? Here’s a breakdown:

1. This takes a snapshot of the current ARView, providing image as a result.
2. Here, you create a compressed version of the image to reduce the image size.
3. Finally, you save the compressed image into the photo’s album.

Don’t forget to uncomment the call to TakeSnapshot() in the action for the Shutter button:

self.TakeSnapshot()

Fantastic! When the user chooses the Shutter button now, the app takes a snapshot of ARView and stores the image in the photo’s album.

Requesting access to the camera and photos

You’re not done just yet. You still need to make sure the app asks for access to the camera and the photo’s album.

Open Info.plist and find Privacy — Camera Usage Description. Set the value to Access required for AR experience.

When the user starts the app for the first time, the app now requests access to the camera.

Add a new key by clicking the Plus Sign button next to the current key.

For the new key, select Privacy — Photo Library Additions Usage Description. Set the value to Access required to save selfies.

This will request access to the photo library when the user takes a selfie.

Finally, reap the fruits of your labor and do a quick build and run to test the app.

The app starts and requests access to the camera. The scene loads and presents the cube. The glorious UI displays on top of the ARView. When you select the Shutter button, it requests access to the photo library, then it takes and stores a snapshot.

For now, the Previous and Next buttons don’t do much, but you’ll deal with that in the next chapter.

Key points

You’ve reached the end of this chapter. To recap some of the key takeaways:

• You now know about Apple’s latest and greatest RealityKit framework, designed for AR. It helps reduce the complexities AR developers face.

• Reality Composer is tightly integrated into Xcode, which generates companion code that gives you strongly-typed access to your virtual scenes and content.

• Creating a UI with SwiftUI for RealityKit apps is super easy, now that you know how.

Where to go from here?

There’s much more content about RealityKit waiting for you from WWDC 2019. I highly recommend you check out the following:

• Information about RealityKiy and Reality Composer — https://apple.co/2OYxIsG

• WWDC’s video on RealityKit & Reality Composer — https://apple.co/2GP6iyI

See you in the next chapter, where you’ll continue building your AR Funny Face app and learn more about face anchors.