Building a Museum App with ARKit 2

Have you ever stood at a museum exhibit and wanted to know more about the art or artifact than the little placard provides? There should really be an app for that. Well, you can make such an app with image and object detection and tracking in ARKit 2!

To make the experience fun and interesting, ARKit lets apps add dynamic virtual content to real-world objects. It allows you to build interactive guide apps for real-world places and things. ARKit makes it easy with built-in image and object tracking. This is perfect for companion apps at museums, galleries, amusement parks and colleges. It can bring to life any place that you want to provide a dynamic or user-specific experience.

In this tutorial, you’ll build out TriassicLoupe, a companion app for a dinosaur exhibit at a natural history museum. The concept is that of a jeweler’s loupe; this app reveals hidden details as a user points it around the exhibit. Don’t worry if you don’t have any dinosaurs lying around — you’ll be able to use regular household items as stand-ins.

The final app displays a short animation on top of an informational image. It also shows information about a dinosaur next to a free-standing replica object. The app will also add some scary sound effects to that object.

The app uses ARKit with SceneKit, the 3D graphics framework for iOS. You’ll see that ARKit does all the heavy lifting with SceneKit. For image and object tracking, you’ll use only very basic SceneKit capabilities, and nothing you learn about it from this tutorial is incidental. Learning more about SceneKit will enable you to build richer apps, but that’s outside the scope of this tutorial. See the Where to Go From Here? section at the end for further reading suggestions.

Getting Started

ARKit relies on the built-in functionality of the A9 or later processor. It uses machine learning, the rear camera, and video and image processing. This means that ARKit apps need to run on an iPhone 6s or newer, and they won’t work in the Simulator.

It’s handy to have an extra-long Lightning cable or for you to set up your device to connect to Xcode through Wi-Fi. ARKit requires moving around a little bit to get a good world map. A world map is ARKit’s awareness of physical space. It’s a collection of feature points, measurements, orientations and anchors.

To get started, use the Download Materials button at the top or bottom of this tutorial to download the starter project. The .zip file contains a helper project, assets to use later in the tutorial and the starter and final projects.

Open the starter project. The app itself is very simple with a single ViewController where you’ll add all the logic. There’s a helper struct, DinosaurFacts, which contains some basic information about a few dinosaurs.

If you build and run, you’ll see a black screen since you haven’t yet wired up the ARKit session.

Building Image Detection

The first thing to do is build the image detection. Image detection might be the simplest function of ARKit. To build an image detector, all you have to do is provide an image tracking session with a copy of the image. These provided images are called reference images.

Adding Reference Images

TriassicLoupe uses the artwork from informational signs for the reference images. When the user points the app at one of these signs, the app will add a dinosaur image overlay.

Like other app images, augmented reality (AR) reference images live in the asset catalog. Reference images are a little special because they need to be grouped specially for ARKit.

Open Assets.xcassets and click the + button at the bottom.

From the pop-up menu, select New AR Resource Group to create a new group. Rename it AR Images since this group will hold the reference images.

In finder, open the Dinosaur Images folder from the downloaded materials. Drag each image file one by one into the AR Images in Xcode. Once done, you should have three images with yellow warning triangles.

Xcode warns you if the reference images are not good enough to be a reference. This might be because they are too small or don’t have enough features or contrast. Images that have lots of empty space, few colors or lack distinctive shapes are hard to detect.

In this case, the warnings are “Unsupported Configuration” warnings. This is because reference images must have non-zero, positive width. AR reference images require you to specify their real-world sizes!

Select the stegosaurus image in the asset library. Then select the Attributes inspector.

Change the units to Inches. Next, enter a width of 4.15. When you do that, the height will automatically become 2.5711, based on the aspect ratio! These fields are where you bridge the virtual world and the real one.

For the other two images, use these values:

• trex: Inches, width: 6.3333, height: 4.75
• triceratops: Inches, width: 5, height: 2.8125

Once you’ve entered the sizes, the warnings disappear.

These images correspond to the slides of the included Dinosaurs.key Keynote file. Each slide represents an informational placard that would be next to a museum display. When printed out on U.S. letter-sized paper, the specified sizes are the physical image size.

These images are each a different style to demonstrate a small part of ARKit’s range. The two things that matter here: 1. There is a enough shape and contrast in the image. 2. The real-world version is flat, well-lit and non-reflective.

A book page, wallpaper or printing on a mirror are bad candidates. A photo, painting or illustration will work well. Xcode will warn you if the image isn’t good enough. No need to guess at runtime!

Now, it’s time to move on writing the code to look for those images.