12. ARKit

Written by Chris Language

Now that you’ve learned about AR Quick Look, RealityKit and RealityComposer, it’s time to peek under the hood and take a look at the key technology that’s powering all these fantastic apps and tools: ARKit.

In this section, you’ll combine the power of ARKit with the power of SpriteKit. You’ll create a fun 2D SpriteKit game where you get to pop little emojis before they explode in your face!

In this chapter, you’ll focus on ARKit first.

Note: This chapter is theory based only. Feel free to jump ahead to the next chapter if you want to get your hands dirty with an ARKit based project.

What is ARKit?

ARKit is Apple’s mobile AR development framework. With it, you can create an immersive, engaging experience that allows you to augment virtual 2D and 3D content with the live camera feed of the world around you.

Augmented reality frameworks aren’t new. Vuforia, for example, has been around for many years. What sets ARKit apart from other AR frameworks is that ARKit performs markerless tracking.

Being markerless means that ARKit doesn’t need tracking cards. It understands the world around it and quickly identifies surfaces that it can place virtual content on.

Interestingly enough, older frameworks quickly incorporated ARKit into their APIs, instantly giving them ARKit’s awesome markerless tracking capabilities.

ARKit transforms any Apple device with an A9 or higher processor into an AR-capable device. Just think about that for a second: At this very moment, millions of Apple users already have a sophisticated AR-capable device right in their pockets.

Note: As Apple continues to develop new AR-capable hardware for its most recent devices, they improve ARKit, too, to provide support for the newer hardware. It’s important to note that some of the more modern ARKit features will only work on devices with specific hardware.

Animojis & Memojis

ARKit is the key technology responsible for making iMessage one of the coolest instant messaging apps out there. ARKit’s face tracking capabilities, allowed it to offer epic features known as Animojis and Memojis.

Using Animojis, you can become a mystical unicorn or a fearless lion. You can record yourself while you puppeteer one of these highly-detailed creatures using your own facial expressions.

Apple’s engineers didn’t stop there; they simply had to take it even further with Memojis.

On the surface, Memojis look like the human counterpart of Animojis, but under the hood lies a very sophisticated character creator engine.

With the power of Memojis, you can re-create and transform yourself into a fun animated character. You can then turn on the selfie camera and allow the magic of ARKit to bring yourself to life.

Even apps like FaceTime have started incorporating Memojis. You now have even more freedom of choice to express yourself while meeting with friends, family and colleagues.

I personally love to use it, especially on those bad hair days… which is every day, nowadays.

Handsome as always, but wow, do I need a haircut! :]

Technologies behind ARKit

You might be surprised to find that some of the technologies behind ARKit are quite familiar and have been around for a while.

• AVFoundation: ARKit uses this framework to gain full control over audio and video input and output.

• CoreMotion: ARKit uses Visual Inertial Odometry (VIO) to track the motion of the device and the world around it. VIO fuses the AVFoundation-based input from the camera sensor with the device motion data captured via CoreMotion.

• CoreML: ARKit uses machine learning to speed up surface detection and surface classification. It’s also the key technology behind ARKit’s object and image recognition and processing, as well as face and body detection.

If you’ve used any of these frameworks before, it’s easy to recognize how they work together to support ARKit’s advanced recognition and tracking capabilities.

Rendering integration with ARKit

Interestingly enough, ARKit doesn’t have its own graphics API; it only provides world-space-tracking and scene-understanding capabilities. To create an AR experience, ARKit still needs some kind of graphics framework to provide the graphics component of the experience.

Fortunately, ARKit provides easy integration for familiar graphics frameworks and popular game engines, including:

• SpriteKit: A high-level 2D graphics framework for drawing shapes, particle effects, text, animated sprites and video. It has its own built-in physics engine and is built on top of Metal to achieve the best possible performance.

• SceneKit: A high-level 3D graphics framework for creating 3D scenes and special effects with geometry, materials, lights, particle systems and cameras. It, too, has its own built-in physics engine and is built on top of Metal for the best possible performance.

• RealityKit: A high-level 3D graphics framework built from the ground up on top of Metal specifically for AR. It offers geometry, materials and lights along with its own built-in physics engine.

• Metal: Apple’s lowest-level graphics API, providing developers the fastest and most direct access to the Graphics Processing Unit.

• Unity & Unreal: Apple’s relationships with the gaming development community have resulted in some excellent plug-ins for popular game engines like Unity and Unreal.

ARKit makes it easy to integrate with graphics frameworks such as SpriteKit, SceneKit, RealityKit and Metal.

ARKit-powered app examples

To demonstrate ARKit’s capabilities, here are some examples of interesting ARKit-powered apps you can find for free on the Apple AppStore. Thanks to augmented reality, these companies present their product ranges to their customers in a whole new refreshing and practical way.

IKEA Place

IKEA Place is a digital catalog of lifelike, true-to-scale 3D furniture. Thanks to the power of ARKit, customers can preview IKEA’s furniture in their own homes, giving them a real sense of the furniture’s scale and ascetics.

Find it on the AppStore here: https://apple.co/31Zjoot

Warby Parker

Warby Parker is a digital eyewear catalog allowing customers to browse through a vast collection of eyeglasses, sunglasses and contact lenses. With ARKit’s face detection and tracking capabilities, customers can virtually try out and preview frames.

Find it on the AppStore here: https://apple.co/3fiHUVq

Minecraft Earth

Get to experience Minecraft in a whole new way with Minecraft Earth. With ARKit’s help, players can now enjoy this classic game in augmented reality. They can build amazing creations in tabletop mode or in life-size mode, and even collaborate and build together.

ARKit’s features

If you have prior experience with Apple’s other frameworks, you might expect ARKit to have a long and painful learning curve.

Fear not, because ARKit’s actually quite simple — it only takes a few lines of code to get things working. ARKit does most of the heavy lifting for you so you can focus on what’s important: Creating an immersive and engaging AR experience.

So what can ARKit do for you? Take a look and prepare to be blown away.

World space tracking

ARKit tracks the device’s position and orientation in real time through real-world space.

It uses sampled motion data from the device to automatically update the virtual content transforms. It then uses the captured motion data to move a virtual camera through a virtual scene.

This mimics the actual real-world position and orientation of the device in the virtual scene.

Scene understanding & persistence

To achieve markerless tracking, ARKit creates and manages its own map of the surfaces and feature points it detects, then stores that map in a point cloud. Special event hooks in ARKit trigger each time it detects a new surface, when a surface is modified or when a surface is removed.

This gives you incredible control over your scene, including how your content interacts with these surfaces.

ARKit can also load and save world maps for quick recognition of previously-scanned spaces. With the same world map loaded on multiple devices, you can easily create a shared experience with a basic network.

You can also keep virtual content persistent across multiple ARKit sessions. How about playing hide and seek with yourself, trying to find where you left your virtual coffee mug?! :]

Light estimation

ARKit can provide accurate real-world lighting conditions. It analyzes the active video feed frame data to calculate the estimated environmental lighting conditions, and then provides a light source to light your virtual scene.

If your virtual content uses physically-based materials, your content will blend into the real-world environment more realistically. This feature means that when you turn the lights off in your room, your virtual 3D content will dim, too.

Real-time environment map

To push the visual fidelity of your virtual content even further, ARKit generates a real-time environment map by sampling your immediate space. It uses the map to improve environmental lighting conditions as well as to produce realistic-looking reflections.

For example, when you place a shiny virtual bowl next to a juicy real-world apple lying on a table, you’ll see the apple’s reflection on the bowl.

Scene interaction

For unrestricted interaction with the 3D content within your AR scenes, use ARKit’s provided hit-test functions.

You can also couple interaction with traditional touch-based gestures, which Apple recommends in its Human Interface Guidelines, specifically for AR-based apps.

However, because you’re inside Apple’s ecosystem, nothing’s stopping you from using any of Apple’s other great frameworks like CoreML or Vision. You can even use SiriKit and command the AR experience with your voice.

Metric scaling system

To calculate distances and sizes in your scene, ARKit uses an accurate metric scaling system. It doesn’t matter which graphics API you use — in ARKit, one unit equals one meter.

This allows you to create 3D models that appear accurately scaled when you place them into the real-world.

Now, you can test if that big couch you always wanted will fit into the living room.

2D image detection & tracking

ARKit can detect and track up to 100 images and rectangular shapes at a time. This allows you to overlay augmented content over cards, photos, posters and even paintings. ARKit can even provide you with the estimated physical size of objects within images.

Bringing static images and paintings to life has never been easier. How about giving the Mona Lisa a grin?

3D object detection

Create your very own AR recognition models of physical 3D objects. Once you give it an AR model, ARKit can detect that physical 3D object, even in fairly complex environments. How about blowing some smoke out of a toy car exhaust?

Face detection & tracking

ARKit can do face detection and tracking on devices equipped with an A12 Bionic chip and later, which powers front-facing camera experiences. Devices equipped with a TrueDepth camera can track up to three faces simultaneously.

With this, you can create your own Animoji characters and have some fun with friends and family.

Multi-camera tracking

You can simultaneously use face and world tracking on the front and back cameras, opening up new possibilities.

Now you can puppeteer a virtual dancing character while controlling its facial movements with your own facial expressions.

Motion capture

Real-time motion capture allows you to track a person’s body position, movement and pose and produces a 3D skeleton consisting of joints and bones.

You can use this information as input into your own AR experiences, or you can store the captured information to animate your own game characters.

People occlusion

AR experiences are now much more immersive, thanks to people occlusion. This is a green-screen-style effect made possible by machine learning.

ARKit can let your AR content pass in front and behind of real-world human figures by using depth-based occlusion. This effect is vastly better on newer devices equipped with a LiDAR scanner.

Depth API

ARKit provides a Depth API for devices equipped with a LiDAR scanner. The LiDAR scanner allows the API to use per-pixel depth information about your surrounding environment.

Combining advanced scene understanding capabilities with 3D mesh data pushes them to the next level. This improves virtual object occlusion and placement, making objects blend into the physical surroundings much more realistically.

Scene geometry

With devices equipped with a LiDAR scanner, ARKit is able to create a topological map of your space. Thanks to machine learning, ARKit has a deep understanding of your environment, too. It can place labels next to identified content like floors, walls, ceilings, windows, doors and even seats. It can then use all this information to provide object occlusion and real-world physics for your virtual content.

Geo location anchors

ARKit allows you to place AR content based on geolocation. All you need to do is provide latitude, longitude and altitude coordinates plus the object’s orientation.

Now, you can place AR content anywhere in the world for users to experience. Anybody up for catching some Pokémon?

Instant AR

LiDAR-equipped devices are capable of incredibly fast plane detection, which allows ARKit to instantly place AR content into the real world without even having to scan the space.

Collaborative sessions

Create shared AR experiences with live collaborative sessions between multiple users. ARKit builds a collaborative world map between multiple users, allowing them to get into a shared AR experience much faster — and, ultimately, making the development of shared AR experiences easier and faster, too.

ARKit’s limitations

ARKit presents a new paradigm for user experiences on mobile devices. It does, however, have some limiting qualities that impact the user’s AR experience. As a developer, you should be aware of these limitations.

Surface detection takes time

If your user is using a device without a LiDAR scanner, surface detection and real-world understanding take a noticeable amount of time.

Sure, surface detection has come a long way since it was initially introduced. However, it’s still important to manage user expectations and guide them through the surface detection stage of your app.

To help with this, ARKit offers a standardized AR onboarding process that you can use.

Motion processing lag

With AR experiences, you’ll quickly notice that excessive device motion can cause stuttering and blurred imaging, which makes it difficult for ARKit to understand what it’s looking at. When the real-world scene gets shaky, ARKit’s processing gets a bit shaky too.

It’s important to notify the user of these issues. ARKit does provide you with various events that you can use to detect these issues and handle them proactively.

Low-light conditions

Low-light conditions are problematic. The on-device camera cannot discern low-light scenes as well as the human eye can, which means ARKit has trouble with scene detection when the lights are dim. It may even fail to recognize any surfaces at all.

As a rule of thumb, if it’s night bright enough for you to see clearly, ARKit is struggling even more.

Smooth & reflective surfaces

ARKit has difficulty detecting and tracking smooth, textureless and even reflective surfaces. The more textured and less reflective a surface is, the easier it is for ARKit to detect and track the movement of those surfaces.

As a developer, your job is to make sure your users understand these kinds of limitations before using your AR-enabled app. Otherwise, they’ll get frustrated when they can’t place virtual furniture onto shiny, polished, jet-black marble floors.

Ghost effect

After using AR for a while, you might notice some virtual content disobeys the laws of physics. For example, the virtual content might pass through real-world objects, much like a ghost would pass through a closed door.

Since situations like this can break the AR experience, it’s a good idea to encourage users to use your app in wide-open spaces or on top of a nice, clean table.

Apple is hard at work to improve the overall experience with the introduction of people and object occlusion. Who knows, perhaps Apple is already working on how to occlude those pesky pets that are always getting in the way? :]

Hardware limitations

As a developer, you have to be aware that some ARKit features are only available on certain Apple devices.

ARKit provides ways to check for feature availability. Make sure your app uses these methods to ensure that your app is stable and that the user has the optimum immersive AR experience.

ARKit resources

Apple provides a large collection of important resources in the form of documentation, videos and sample apps for you to learn from. Finding these resources can be challenging, so here’s a curated list for you:

Official ARKit documentation

This is Apple’s official site for ARKit documentation. Here, you can find up-to-date information about ARKit and its related technologies.

• Apple Developer Documentation for ARKit: http://apple.co/2sE0rUq

Human Interface Guidelines for augmented reality

Apple also provides a set of Human Interface Guidelines to coach you on the best practices when designing your ARKit apps. This guide shows you exactly what to do — and what not to do. Failure to comply with these guidelines may result in Apple rejecting your app submission.

• Human Interface Guidelines for AR: http://apple.co/2xOwp1Q

Official WWDC videos on ARKit

Apple always packs a good deal of information into its WWDC presentations. Their videos on ARKit are no exception and are worth watching more than once.

• ARKit video’s from WWDC 2017 to 2020: https://apple.co/2NxL2TX

Interactive content with ARKit

This is one of Apple’s official ARKit demos, and it’s a fun way to get some hands-on experience with ARKit.

Follow the link and look for the Download Sample Code button. Once you find it, download the .zip file, extract the contents and build the project using the latest version of Xcode.

You’ll need to run it on a physical device with an A9 processor or better (iPhone SE, iPhone 6s and later models). At this time, you can’t test AR apps on the simulator.

In this demo, you’ll get to meet Elon the chameleon, but be careful — he’s always watching!

• Interactive Content with ARKit: http://apple.co/2yI4gi2

Key points

You’ve reached the end of the chapter, but your journey of creating AR-enabled games and apps with ARKit has just begun.

Here’s a quick recap of some of the most important things you’ve learned about ARKit in this chapter:

• ARKit: You now know what ARKit is and the power it can add to your games and apps.

• Surface detection & tracking: ARKit makes it easy to detect viable real-world surfaces where you can place your AR content.

• Scene Understanding: Not only can ARKit track various types of surfaces, but with the power of machine learning, it’s also capable of understanding what it’s seeing and can even tell you what the types of surfaces are.

• Image detection & tracking: Detecting and tracking posters, paintings or even business cards is a breeze with ARKit.

• Object detection: ARKit can Detect and augment real-world objects.

• Face detection & tracking: ARKit’s the key technology behind features like Animojis and Memojis. You can harness the power of ARKit to create your own augmented face experiences.

• Occlusion: ARKit’s ability to occlude people and objects is slowly by surely improving. With capable devices, basic occlusion is standard, which has the power to make occlusion-related issues a thing of the past.

• Graphics Frameworks: ARKit isn’t a graphics library; instead it’s a framework that does the heavy lifting of AR for you. It offers easy integration for graphics frameworks and there are also plugins available for popular game engines.

In the next chapter, you’ll learn how to create your first ARKit app powered by SpriteKit using Xcode. See you soon!