Testing with TestFlight

Of course, we’ve been testing our app all along; we had a whole chapter about testing our code early on. But developers never see their apps the same way users do; we come in with biases and assumptions, and with insider knowledge of how the app works. A typical user has none of these things.

We need some typical users!

Lucky for us, uploading our app gets us a lot closer to typical users. Previously, we could only run the app on a device directly connected via USB cable to our Mac running Xcode. But now, from iTunes Connect, we can send the app to testers all around the world and have them try it out.

To do that, we’re going to use a testing platform called TestFlight. This service allows us to send our app to testers of our choosing, lets them install it prior to its release on the App Store, and lets them send us their feedback. All they need is their own iOS devices.

Testing with Internal Testers

Let’s try it out. On the iTunes Connect page, go to My Apps and visit the Pragmatic Podcasts app. Among the tabs at the top of the page is TestFlight, so let’s go there.

On the left side of the page, there are three menu items at the top—Test Information, Internal Testing, and External Testing—followed by a list of platforms under the heading TestFlight Builds.

By default, we start on Test Information. Here we can enter a feedback email and (optionally) URLs for marketing and a privacy policy.

Let’s move on to Internal Testing. These are builds that go to trusted members of your own development team. We can have up to twenty-five internal testers, and there must be at least one. In the Internal Testers area, click the + to add yourself as a tester. More testers can be added later via iTunes Connect.

To send a build to yourself or other team members, click Select Version to Test and choose the version/build you just uploaded. This will also ask if the app uses cryptography, since that is subject to U.S. export controls. Ours does not, so click No, but take note of the Export Compliance information that appears—we will need to go back and add a key to our app’s Info.plist to declare that we don’t use crypto.

Then click Start Testing.

Beginning a test cycle sends out invitation emails to all members of your team. The invitation includes two important links: a download link to your app, and a link to the TestFlight iOS app. Once you receive the email, open the email on your device, and download TestFlight from the App Store. When you run it, you’ll also have to install a special profile; this is like the provisioning profiles that let Xcode put apps on your device, but in this case it lets the TestFlight app install the app to be tested.

Once the TestFlight app is set up, return to the email and click the Start Testing link. This will take you to a web page with a code you can redeem in the TestFlight app. When you redeem the code, you’ll see the test instructions and description from the TestFlight page, and an Install button to download the app. Once installed, you can run the app from TestFlight via the Open button or just from home screen.

Run the app as usual, and look for any bugs you missed. Back on the Mac, the internal testing page will show that you’ve installed and run the app.

This is all well and good, but our own developers and colleagues aren’t always going to be the most rigorous critics. For that, we need to go outside our organization.

Testing with External Testers

The opposite of internal testers is, of course, external testers. These are people who aren’t your fellow team members in iTunes Connect. You might never know them by anything more than an email and their feedback on your app.

External testing works like internal testing in a lot of ways: you choose a version and build to test, and an email is sent to all the testers, allowing them to install the app via the TestFlight app and try it out. There are two big differences:

• You can have up to 2,000 external testers of your choosing. These can include your mom, your college roommate, your lover, your worst enemy (hey, they’ll probably give honest feedback)…all they have to have is an iOS device and an email address.

• Apps sent out for external testing must pass a brief review by Apple before they’re made available.

Let’s try it out. Get one or more friends to agree to help test the app and collect their emails. Click External Testing from the left-side menu; this brings up a page showing which version/build combination we’re testing and who our testers are. For our screenshots, Janie isn’t on Chris’s team in iTunes Connect, so she’s the external tester. Click the + next to External Testers to add testers by name and email.

In the iOS section of the form, click the Add Build to Test link to choose any build we’ve uploaded (after a brief processing delay immediately following the upload). When we pick one, we go to a screen in which we describe what testers should focus on, a description of the app itself. Also, you must provide an email where testers can contact you, and a marketing URL. A second screen asks for information that Apple’s reviewers will need: your name, phone, and email, and whether a login is required to use the app (and, if so, a username and password they can use). Finally, there’s an Additional Info field where you can communicate any other information that Apple’s reviewers might need when looking at the app.

Approval can take as little as thirty minutes or as long as a few days if you submit on a weekend or holiday. At any rate, once approved, your external testers all receive an email linking them to the TestFlight app and our app to test, just as we saw with the internal testing.

Fixing Problems

And now we wait. Our testers will be notified by email that a new build is available, and by installing the TestFlight app, they can install our app. If they find bugs, they can return to the TestFlight app and click the Send Feedback button.

And chances are they will find bugs. It’s just a matter of time, so we wait and…oh look, there’s an email from Janie, with an attachment that gives us the specs of the device she was testing on. Let’s see what her feedback says.

Ah, she raises a good point. It’s hard to accurately enter URLs on iOS, and currently the app gives no feedback when the URL is junk or the download fails. We should at least show an alert or something in that case.

So let’s fix this and send out a new build. We need two things: a way to catch errors in our parser, and a way to show an error.

The first is easier: switch to PodcastFeedParser.swift. In this file, we already have an onParserFinished closure property to handle success cases, so let’s add a new property for errors:

Notice that this onParserErrored closure will pass an Error as a parameter, unlike onParserFinished, which passes no arguments.

Next, we’ll call this closure in the two places that can error: when loading data from the URL, and when parsing any data we received. In our init(contentsOf:), we creaste a URLSessionDataTask that can be passed the optionals dataMb, responseMb, and errorMb; although, the only one we’ve used so far is dataMb. So, after the curly brace that closes the if let data = dataMb condition, let’s call our closure if there’s an error:

As you can see, all this conditional needs to do is call the optional closure, passing in the error. It’ll be up to the caller to decide what actually happens in the closure.

Our other possible error case is when we get some data, but it’s garbage and the XMLParser produces an error. This calls back to the XMLParserDelegate with the parser(parseErrorOccurred:), a delegate method we haven’t needed until now. After the existing parserDidEndDocument implementation, add a simple error handler:

This allows the parser to tell us about an error, but what do we actually do about it? That decision belongs not in the parser, but in the UI. Specifically, we need to put it in the EpisodeListViewController, where unwindToEpisodeList sets up a PodcastFeedParser after the add-podcast view controller dismisses. After the close brace that completes the parser.onParserFinished = { [weak self] in declaration, add the following code to add an error-handling closure.

This creates a new kind of view controller, a UIAlertController, which gives us the modal error dialog we see throughout iOS. Its initializer takes a title, a message string, and a style, for which UIAlertControllerStyle.alert shows the typical middle-of-the-screen modal error dialog. The alert controller also lets us specify multiple buttons to be shown with the alert, each as a UIAlertAction. Each action takes a title for the button, a style, and a closure to be run when that button is tapped. We add the actions to the alert controller, and then our view controller shows the alert controller with presentViewController.

With this bug fixed—plus any others that come in—we need to kick off a new round of testing. Go to the target’s properties and increment the build number, since we will want to track this build separately. Then, as before, we archive, upload to Apple, go to TestFlight in iTunes Connect, mark this new build as the one to test, and click Start Testing to send the new and better build to our testers. Testers will get an email telling them about the new build, and the TestFlight app will let them install it over the old one and continue testing.