Media, Graphics, and Gaming

iOS devices shine when it comes to games, sound, and video, and there are a rich set of frameworks for working with static and time-based media.

Graphics

For 2D graphics, the essential tool is Core Graphics, sometimes also called Quartz. Core Graphics is a drawing library that lets us draw resolution-independent lines, shapes, patterns, and text to an arbitrary “context.” It doesn’t matter if the context is a UIView we’re drawing at runtime, or a PDF file stored on the filesystem—either way, the drawing commands are the same. This technology is what gives us user interfaces that look sharp at any size or resolution, from old non-Retina iPhones to the 3x display of the iPhone 7 Plus.

Quartz also gives us Core Animation, which allows apps to animate gradual changes like sliding their contents around, adjusting sizes, or fading in and out, rather than having UI elements just disappear or resize instantly. Many Core Animation features are built into UIKit components—we get smooth table scrolling for free—but in some cases, it’s useful to use Core Animation directly, particularly when creating our own custom components.

While Quartz works with the drawing and presentation of graphics, Core Image works with the content of images. Core Image offers hundreds of filters that operate on images, mostly for effects like color correction, image sharpening, cropping, scaling, distortion, blurring, and so on. These filters can be chained together, one after another, to produce interesting composite effects, like a “chroma key” that takes out all pixels of a certain color and replaces them with another image. Core Image is also the technology of choice for writing the Photo Editing type of app extension mentioned earlier.

Media

Media is one feature of the iOS SDK that we’ve used already: we imported AV Foundation to play our podcasts, with the convenient AVPlayer. AV Foundation goes far deeper than this, of course, and has one of the highest class counts of the iOS frameworks. With AV Foundation, you can capture media from the camera or microphones, edit together parts of different sources to create new audio-video presentations, and export these to files.

AV Foundation also offers some audio-only classes for capture and playback, as well as an “engine” API for mixing sources together and performing effects on the audio they process. For video, these kinds of mixing and effects techniques actually incorporate Core Animation and Core Image: you can use Core Animation to provide motion elements like scrolling text, and Core Image to perform visual filtering on video frames.

Games

It’s not often that a day goes by where at least one of the authors doesn’t put in some serious time playing a game on their iPhone or iPad. After all, iOS is one of the world’s most popular game platforms.

For 2D games, iOS offers SpriteKit, which lets us create flat “sprite” graphics, such as angry or flappy birds, and animate them over a background. This topic is so popular, it has its own Pragmatic title: Build iOS Games with Sprite Kit (out of print) [PS14].

For 3D games, there are a couple of options. The cross-platform 3D standard OpenGL is available on iOS, although it’s a C API that isn’t really practical to call from Swift. Apple also offers Metal, a similar 3D API that is meant only for use on the graphics processing units (GPUs) on iOS devices, and which can be faster by being purpose-built for these chips rather than hardware-agnostic. Both are very demanding and require advanced development skills to write the “shader” code to determine how to render 3D objects on a 2D screen.

For the logic of a game, whether 2D or 3D, GameplayKit offers help in modeling a game’s logic, providing things like path-finding, artificially intelligent opponent behavior, rule engines, randomization, and more.