Why I Love ARKit for Apple iOS 11

Other Augmented Reality Libraries

ARKit for Apple iOS 11 isn’t the first library created for augmented reality apps. Once iOS devices became capable of augmented reality, open source libraries weren’t far behind. The iphonearkit project on Github dates to 2009 when the iPhone 3GS was state-of-the-art. Our Augmented Reality iOS tutorial focuses on location-based AR and uses the HDAugmentedReality project.

The popular OpenCV project handles many computer vision aspects helpful in augmented reality. A tutorial on this site shows you how to make an augmented reality target shooter. It demonstrates this library’s capabilities.

Authors of open-source projects generally develop libraries for the author’s specific need. Most libraries support a specific usage and type of augmented reality application, and are seldom updated to take advantage of new iOS abilities and devices. They hide the complex calculations from the developer, but also make assumptions about how apps built on top of those libraries will work. If you find one that fits your need, it’s great. But for many apps, the right library doesn’t exist.

There are commercial libraries that help create augmented reality apps. These commercial libraries offer better support, but can have costs that are prohibitive for a single developer or small team. They’re often general purpose libraries written in C, C++, or other languages. This crossing of languages adds complexity, increases the learning curve for a library, and can cause integration headaches for developers not familiar with bridging libraries in other languages.

What ARKit Does

What makes ARKit special is that it reflects Apple at its best — it just works!

With ARKit, the creation of augmented reality apps no longer requires a custom engine or finding the perfect library. Apple built the engine to work on all existing devices with at least an A9 processor that run iOS 11. That covers the iPhone 6s or later, including the iPhone SE, all iPad Pros, and the 2017 iPad.

ARKit uses a process called visual-inertial odometry to build the correspondence between the real and virtual in your app. It combines the motion sensors in the device with analysis of the scene gathered through the device’s camera and produces a high-precision model of the device’s position and motion within the world for you. As a result, it recognizes notable features in the scene image, tracks differences in the positions of those features, and compares that information with motion sensing data during movement.

So ARKit doesn’t only determine that you’re pointing your phone to the east. It also analyzes and tries to understand the scene to the east. ARKit can find real-world surfaces that correspond to points in the camera image. It can detect flat surfaces (though not vertical ones) and provide information on the position and size of these surfaces. Your app can place virtual objects on and interact with these points and surfaces.

The tight integration with iOS means that you, as a developer, can leverage skills you’ve already gained in your years of iOS app development. You can build virtual objects using SceneKit and SpriteKit, or you can use Metal to develop a virtual scene with near complete control of the visual elements.

What can you do with ARKit?

Now that you understand a bit more about augmented reality you may be wondering, “What is this actually useful for?”

ARKit saw general release to the public in late September as part of the release of Apple’s iOS 11. In the months between, developers have already explored the technology and produced demos for inspiration. While ARKit is mere weeks old, developers have been considering how to use it for months. There seems to be a new ARKit demo on Twitter every day, and I’ve tried to highlight some of the best ones below.

Objects in World

The first demo I saw for ARKit used augmented reality to measure a physical space viewed through the phone. The ability to translate distances on a screen to distances in the real world provides a foundation for apps that need to place objects accurately onto a view.

Developers have expanded on this with apps that allow a user to determine a room’s dimensions and layout by moving a phone around the room. Other demos measure the distance between objects or along paths by drawing lines on a screen with a finger.

Ikea demonstrated an app allowing customers to preview furniture in a room before heading to the store. It’s a lot better to learn that a couch won’t fit in your room before you’ve had it delivered. Design and home decor companies likely will add similar abilities to their apps. This will make online shopping for these items feel a little less risky.

Games and Fun

Games in particular seem a natural fit for this functionality. It’s not a coincidence that both WWDC and the Apple iPhone launch events demonstrated augmented reality games. I expect game developers will embrace ARKit, as games are a good fit for new visual technologies. Even just having a glimpse at the demos during the Apple event, you can tell there’s something enticing about playing a game in augmented reality. The ability to move around and interact with a game as though it’s truly happening in the real world opens up a whole new category of gaming.

I suspect early games will likely resemble Pokémon Go, or the games demoed at Apple events like The Machines. In time, games that push the limits of augmented reality should arrive, and it’s likely that the most captivating games haven’t even been made public yet.

I’m most intrigued by the idea of games that bring people into a shared virtual environment. Instead of looking at a representation of a board on the screen, an augmented reality games can show the board on the desk in front of you. You could see avatars of other players around the board with you, much as if you were sitting in a room together. This would take the mildly-connected experience of multiplayer and make it much more immersive. The concept could also would work for card games, role playing games, and other shared game experiences.