Capturing Text From Camera Using SwiftUI

Learn how to capture text from the iPhone camera into your SwiftUI app so your users can enter data more quickly and easily. By Audrey Tam.

Leave a rating/review

Download materials

Save for later

Contents

Capturing Text From Camera Using SwiftUI

25 mins

Contents

Your iPhone camera lets you capture landscapes, people and events, but it’s also a useful information-gathering tool. You see a concert poster or an ad for a service you need or a restaurant that looks interesting, and you take a photo. Later, you type or speak some text from the photo — a URL, a date, a phone number — into a search engine or a new contact.

But keyboard or voice input is error-prone. What if you could just copy and paste text from a photo or directly from the camera view? Better still, what if you could scan text in a photo or camera view directly into your app? Give a big welcome to iOS 15 Live Text!

In this tutorial, you’ll learn how to capture text from the iPhone camera into your SwiftUI app to let your users enter data more quickly and easily.

Note: You should be comfortable with using SwiftUI, Swift and Xcode to develop iOS apps. You’ll need Xcode 13 and an iPhone running iOS 15. Your iPhone must have been manufactured in 2018 or later, so it has an A12 (or later) Neural Engine. Its list of preferred languages must include at least one of these: English, Spanish, Chinese, French, Italian, German, Portuguese. And check this list for your region.

iOS 15 Live Text

The wonderful new iOS 15 Live Text feature works right out of the box, but only for fairly new boxes. And only for the languages and regions listed in the note above.

You need a newish (2018+) iPhone, one with an A12 or later Neural Engine. Live Text uses Apple’s Vision machine learning model, which needs the Neural Engine.

Live Text works on iPhone XS, iPhone XR and later. These iPhones do not support Live Text: iPhone SE (first generation), iPhone 6S, iPhone 6S Plus, iPhone 7, iPhone 7 Plus and iPhone X.

Live Text sort of works in Photos on 2018-or-later iPads, but not iPad Camera. This article is about using Live Text with Camera, so it’s only about iPhones.

Now grab your iPhone and make sure Live Text is turned on: In Settings, turn on Camera ▸ Live Text and General ▸ Language & Region ▸ Live Text:

Turn on Live Text in Settings.

Note: If you don’t see these settings, your iPhone doesn’t have a Neural Engine. If you’re looking for an excuse to get an iPhone 13, you’re welcome!

Live Text in Photos

Live Text detects text in photos and in the Camera ▸ Photo view finder. In this tutorial, you’ll use your iPhone’s camera. But first, see how great it is on your existing photos. It’s also easier to practice using Live Text on an image that isn’t moving around in a hand-held camera view.

Open Photos and find a photo that has some text, especially a URL, phone number, email or street address.

I have this photo I took at a yarn expo, as a reminder about a vendor I wanted to look up later.

Live Text in Photos: Tapping a URL opens it in Safari.

I tapped the Live Text button (three lines in a viewfinder square); it turned blue. Then I tapped tarndie.com, and their web page opened in Safari!

If there’s a map address in your photo, tapping it opens Maps. Tapping a phone number displays the usual menu to Call, Send Message, FaceTime etc.

Live Text in Photos: Tapping address or phone number

And if there’s an email address, tapping it opens a new message in your email app.

If you want to copy text from an app that doesn’t let you select text, just take a screenshot and open the preview photo:

Live Text in screenshot

Note: Thanks to Harshil Shah for tweeting this. The screen is from Chris Wu’s Museum Shuffle.

Live Text works the same way in the Camera app, but you need a steady hand. If you can’t quite get it to focus on what you want, just take a photo, then use Live Text on that.

Now keep reading to see how you can use Live Text in your apps.

Getting Started

Download the project materials using the Download Materials button at the top or bottom of this tutorial.

Open the WaitForIt project in the starter folder. This is a simple app where you can keep track of how long you have to wait for someone’s birthday. It uses the new Date.RelativeFormatStyle method relative(presentation:unitsStyle:).

WaitForIt

To get camera input, you must run this app on your iPhone-with-Neural-Engine.

Build and Run on Your Device

Connect your iPhone to your Mac with a cable. In the target’s Signing & Capabilities tab, customize the Bundle Identifier and set a Team.

Customize bundle ID and set team.

Select your iPhone from the run destination menu, then build and run.

Write or type your name and birthday in this format:

Name and birthday sample

Camera input works with hand-writing but, in my experience, the writing needs to be really clear, more like careful printing than cursive.

Tap the + button. In the Add Person view, tap the Name text field then tap it a second time to show the Scan Text button:

Double-tap text field to show Scan Text button.

Note: You might see just the scan button icon, alongside the Paste button. This tends to happen after you’ve used the scan button a few times, and the system decides you don’t need the text label anymore.

Tap this button to open the camera and point the camera at your name and birthday text:

Point camera at text.

Brackets appear around detected text, and the detected text also appears in the text field. The brackets and text field text can change as your hand moves the camera, detecting different amounts of text.

You can tap to indicate where you want the camera to focus, and you can drag up on the camera view to enlarge it:

Drag up to enlarge camera view.