What Is TensorFlow?

It’s recently become possible to solve a range of problems across a wide variety of domains using large neural networks. TensorFlow is a framework that lets you train and deploy these networks. It was originally created by Google as its main internal tool for deep learning, but it’s also available as open source with a large and active community.

The models (also known as graphs) are descriptions of neural networks. They consist of a series of operations, each connected to some other operations as inputs and outputs. TensorFlow helps you construct these models, train them on a dataset, and deploy them where they’re needed. What’s special about TensorFlow is that it’s built to support the whole process, from researchers building new models to production engineers deploying those models on servers or mobile devices.

This guide focuses on the deployment process, since there’s more documentation available already for the research side. The most common use case in production is that you have a pretrained model that shows promising results on test data, and you want to integrate it into a user-facing application. There are less-common situations in which you want to do training in production, but this guide won’t cover those.

The process of taking a trained model and running it on new inputs is known as inference, or prediction. Inference is particularly interesting because the computational requirements scale up with the numbers of users an application has, whereas the demands of training only scale with the number of researchers. As more uses are found for deep learning, the inference compute workload grows much more quickly than training. It also has a lot of opportunities for optimization, since the model you’re running is known ahead of time, and the weights are fixed.

The guide is specifically aimed at mobile and embedded platforms, since those are the environments most different from the kinds of machines that training is normally done on. However, many of the techniques also apply to the process of deploying on servers.

Mobile AI applications need to be small, fast, and easy to build to be successful. Here I’ll be explaining how you can achieve this on your platform with TensorFlow.

What Level of Knowledge Do You Need?

There are examples in this guide that don’t require any machine learning experience, so don’t be put off if you’re not a specialist. You’ll need to know a bit more once you start to deploy your own models, but even there we hope that the demands won’t be overwhelming.

What Hardware and Software Should You Have?

TensorFlow runs on most modern Linux distributions, Windows 10, and macOS. The easiest way to run the examples in this guide is to install Docker and boot up the official TensorFlow image by running:

docker run -it -p 8888:8888 tensorflow/tensorflow

This method does have the disadvantage that any local files (such as compilation caches) are lost when you close the container. If you’re running outside of Docker, we recommend using virtualenv to keep your Python dependencies clean.

Some of the scripts require you to compile TensorFlow, so you’ll need more than the pip install to work through all the sample code.

In order to try out the mobile examples, you’ll need a device set up for development, using Android Studio for Android, or Xcode for iOS.

What Is TensorFlow Useful for on Mobile?

Traditionally, deep learning has been associated with data centers and giant clusters of high-powered GPU machines. So why does it make sense to run it on mobile devices? The key driver is that it can be very expensive and time-consuming to send all the data a device has access to across a network connection. Deep learning also makes it possible to deliver very interactive applications, in a way that’s not possible when you have to wait for a network round-trip.

In the rest of this section, we cover common use cases for on-device deep learning. Chances are good you’ll find something relevant to the problems you’re trying to solve. We also include links to useful models and papers to give you a head start on building your solutions.

What Should You Do Before You Get Started?

Once you have an idea of the problem you want to solve, you need to make a plan to build your solution. The most important first step is making sure your problem is actually solvable, and the best way to do that is to mock it up using humans in the loop. For example, if you want to drive a robot toy car using voice commands, try recording some audio from the device. Play it back to see if you can make sense of what’s being said. Often you’ll find there are problems in the capture process, such as the motor drowning out speech or not being able to hear at a distance. You should tackle these problems before investing in the modeling process.

Another example might involve showing people photos taken from your app to see if they can classify what’s in the photos in the way you’re expecting. If they can’t (for example, when trying to estimate calories in food from photos, because all white soups look the same), you need to redesign your experience to cope with that uncertainty. A good rule of thumb is that if a human can’t handle the task, it will be hard to train a network to do better.

After you’ve solved any fundamental issues with your use case, create a labeled dataset to define what problem you’re trying to solve. This step is extremely important, even more than picking which model to use. You want it to be as representative as possible of your actual use case, since the model will only be effective at the task you teach it.

It’s also worth investing in tools to make labeling the data as efficient and accurate as possible. For example, if you’re able to switch from the necessity to click a button on a web interface to simple keyboard shortcuts, you may be able to speed up the generation process a lot. We also recommend doing the initial labeling yourself so that you can learn about the difficulties and likely errors and then adjust your labeling or data capture process to avoid them. Once you and your team are able to consistently label examples (that is, once you generally agree on the same labels for most examples), you can then try and capture your knowledge in a manual and teach external raters how to run the same process.

The next step is to pick an effective model to use. If you’re lucky, there’s already a trained model out there you can do fine-tuning on, see “What Is TensorFlow Useful for on Mobile?”; there might be something similar to what you need at TensorFlow’s model garden. Lean toward the simplest model you can find, and try to get started as soon as you have even a small amount of labeled data, since you’ll get the best results when you’re able to iterate quickly. The shorter the amount of time it takes to train a model and run it in the real application, the better overall results you’ll see.

It’s common for an algorithm to get great training accuracy numbers but fail to be useful within a real application because there’s a mismatch between the dataset and real usage. To combat such a mismatch, it’s vital to prototype end-to-end usage as soon as possible. I like to say that the only metric that matters is app store ratings, since the user experience is the end goal for everything we’re doing.

Common Model Patterns

There are lots of different ways of using neural network models to solve problems, and it’s worth having a high-level understanding of some of these patterns to help you decide how to integrate a model into your application:

Bare model

If the data you want to work with comes as a plain array of numerical values, and all you want as output is the same, you may just be able to use a model on its own, with minimal pre- or post-processing. A good example of this is image recognition, where the input is an array of pixel values, and the result is a score for each category. You need to resize the input to fit the expected dimensions the model was trained with, scale the pixel values to a float range instead of 0 to 255, and then sort the outputs to find the highest score. Other than that, getting useful results requires very little additional code beyond running the model.

Sliding window

If you want to find the location of an object in an image, the simplest way is to run many small tiles of the image through a neural network, at many possible locations, and pick the tiles that give the highest response as the likely location. Because you can think of this as running a rectangular window from left to right in rows, gradually moving downward, this is known as sliding window. It’s effective, but because there are so many windows needed to cover a typical image, it’s often far too expensive to use in practical applications. The numbers grow even more if you’re looking for variably sized objects, or those at different rotations, since you have to run the same exhaustive search with all those possible parameters, too.

Box proposals

To optimize the sliding window approach, you can use some other technique to come up with proposals for likely boxes to test, rather than going through every possible combination. These proposals are often generated using traditional computer vision techniques. A big advantage of this approach compared to sliding windows is that you can trade off sensitivity with latency by varying the number of box proposals fed into the model.

Cascades

Another common optimization pattern is to use a simple and cheap neural network to decide whether it’s worth running a more costly model on the input. For example, a system detecting a spoken hotword might run a fast but inaccurate model continuously on all the audio that comes in, and only invoke a more precise and battery-draining model when it thinks there’s a good chance the audio does represent the wanted word. This can also be applied to image models, where you might run an extremely cheap model across the whole image with a sliding window, and then only do an in-depth analysis of tiles that scored higher than a threshold. One disadvantage of this approach is that the latency of the system is unpredictable, since it depends on how many of the early models trigger more processing. If you imagine a face-detecting cascade that is run on a photo of a large crowd, you can see how it would end up doing dramatically more work than on a landscape without people. This is something you need to watch out for if you’re planning an application, since large delays can lead to a very poor user experience if they’re unexpected.

Single-shot detection

It’s also possible to produce both a class and bounding box from running a single model, in a similar way to the basic image classification approach. You can look at examples like the Android person detector or YOLO to see how this works. It’s conceptually a lot simpler than other localization approaches, since it only involves running a single model; but it can struggle to achieve the same results as other techniques when there are a lot of classes involved. It’s an active area of research, though, so keep an eye on the latest model releases to see what improvements are available.

Visualizations

The success of deep learning for vision problems has led to a lot of other domains recasting their tasks as ones that can be tackled by image networks. For example, speech recognition often takes raw PCM audio samples and produces a spectrogram, essentially a visual fingerprint of the frequencies over time, which can then be fed into standard image classification models. If you have time-based data (for example, accelerometer or vibration signals), then this can be an easy way to get started with a neural network solution.

Embeddings

There are some domains, like text, where there’s no obvious way to turn the natural input data into the numbers that a neural network requires. This is where embeddings come in handy. These take a non-numerical input, like a word, and assign it an n-dimensional number. To be useful, these numbers are usually chosen so that objects that have properties in common have numbers that are nearby in that n-dimensional space. A classic example of this is word2vec, where words with similar meanings are usually close to each other. These are called embeddings because all of the concepts can be seen as being embedded in an n-dimensional space represented by the numbers assigned. The idea isn’t restricted to words, however; you can use embeddings for anything you want to convert into useful numbers, even unlikely things like photos of landmarks. They can be useful as the output of networks, too, since the calculations on typical, final, fully connected layers scale linearly with the number of classes being detected; whereas if you output an embedding, it’s possible to do a nearest-neighbor lookup on a very large set of candidates with much lower latency.

Language models

A final pattern to know about is where the output of a neural network is just an input to a much larger system. A good example of this is a language model, in which a speech recognition neural network might output likely phonemes for some audio, but then some custom code that understands likely words and sentences will try to make sense of them overall.

Android

TensorFlow’s first release came with support for Android, and it’s been a priority for the team to keep improving the experience. There are a variety of ways to run TensorFlow on Android, from a prepackaged binary installation to compiling it yourself from scratch.

allprojects {
 repositories {
 jcenter()
 }
}
dependencies {
 compile 'org.tensorflow:tensorflow-android:+'
}

bazel build -c opt //tensorflow/examples/android:tensorflow_demo

bazel build tensorflow/examples/android:tensorflow_demo
adb install -r 
 bazel-bin/tensorflow/examples/android/tensorflow_demo.apk

Android Inference Library

Because Android apps need to be written in Java, and core TensorFlow is in C++, we provide a JNI library to interface between the two. Its interface is aimed only at inference, so it provides the ability to load a graph, set up inputs, and run the model to calculate particular outputs. You can see the full documentation for the minimal set of methods here: http://bit.ly/TensorFlowInferenceInterface-java. The demos applications use this interface, so they’re a good place to look for example usage. You can download prebuilt binary jars at https://ci.tensorflow.org/view/Nightly/job/nightly-android/.

iOS

We’ve seen a lot of great internal and external applications launch using TensorFlow on iOS, so supporting the platform is important to us. We’ve now got both a prebuilt approach as well as a recipe for building it from source if you need more customization.

tensorflow/contrib/makefile/build_all_ios.sh

tensorflow/contrib/makefile/download_dependencies.sh

tensorflow/contrib/makefile/compile_ios_protobuf.sh

make -f tensorflow/contrib/makefile/Makefile 
 TARGET=IOS 
 IOS_ARCH=ARM64

compile_ios_tensorflow.sh "-Os"

mkdir -p ~/graphs
curl -o ~/graphs/inception5h.zip 
 https://storage.googleapis.com/download.tensorflow.org/ 
 models/inception5h.zip
unzip ~/graphs/inception5h.zip -d ~/graphs/inception5h
cp ~/graphs/inception5h/* 
 tensorflow/examples/ios/benchmark/data/
cp ~/graphs/inception5h/* 
 tensorflow/examples/ios/camera/data/
cp ~/graphs/inception5h/* 
 tensorflow/examples/ios/simple/data/

Raspberry Pi

The TensorFlow team is working on providing an official pip install path for getting the framework running easily on the Pi with pre-built binaries. At the time of writing it’s not yet available (check https://www.tensorflow.org/install for the latest details), so here I’ll cover how to build it from source. Building on the Raspberry Pi is similar to a normal Linux system. First, download the dependencies, install the required packages, and build protobuf:

tensorflow/contrib/makefile/download_dependencies.sh
sudo apt-get install -y 
 autoconf automake libtool gcc-4.8 g++-4.8
cd tensorflow/contrib/makefile/downloads/protobuf/
./autogen.sh
./configure
make
sudo make install
sudo ldconfig # refresh shared library cache
cd ../../../../..

Once that’s done, you can use make to build the library and example:

make -f tensorflow/contrib/makefile/Makefile HOST_OS=PI 
 TARGET=PI OPTFLAGS="-Os" CXX=g++-4.8

If you’re only interested in building for Raspberry Pi’s second and third generations, you can supply some extra optimization flags to give you code that will run faster:

make -f tensorflow/contrib/makefile/Makefile HOST_OS=PI 
 TARGET=PI OPTFLAGS="-Os -mfpu=neon-vfpv4 
 -funsafe-math-optimizations -ftree-vectorize" CXX=g++-4.8

One thing to be careful of is that the GCC version 4.9 currently installed on Jessie by default will hit an error mentioning __atomic_compare_exchange. This is why the examples in this section specify CXX=g++-4.8 explicitly, and why we install it using apt-get. If you have partially built using the default GCC 4.9, hit the error and switch to 4.8. You need to do a make -f tensorflow/contrib/makefile/Makefile clean before you build. If you don’t, the build will appear to succeed, but you’ll encounter malloc(): memory corruption errors when you try to run any programs using the library.

curl https://storage.googleapis.com/download.tensorflow.org/
models/inception_dec_2015_stripped.zip 
-o /tmp/inception_dec_2015_stripped.zip
unzip /tmp/inception_dec_2015_stripped.zip 
 -d tensorflow/contrib/pi_examples/label_image/data/

Linking the Library

After you’ve managed to build the examples, the next step is to call the code from your own application. This means you need to break out TensorFlow as a framework, include the right header files, and link against the built libraries and dependencies. Unfortunately, there’s no clear separation between implementation and API headers in the C++ core of TensorFlow, so you’ll need to pull in quite a few things to be able to call it.

Here is a checklist of what you’ll need to do, based on the iOS build:

• Link against tensorflow/contrib/makefile/gen/lib/libtensorflow-core.a, usually by adding -L/your/path/tensorflow/contrib/makefile/gen/lib/ and -ltensorflow-core to your linker flags.

• Link against the generated protobuf libraries by adding -L/your/path/tensorflow/contrib/makefile/gen/protobuf_ios/lib and -lprotobuf and -lprotobuf-lite to your command line.

• For the include paths, you need the root of your TensorFlow source folder as the first entry, followed by tensorflow/contrib/makefile/downloads/protobuf/src, tensorflow/contrib/makefile/downloads, tensorflow/contrib/makefile/downloads/eigen, and tensorflow/contrib/makefile/gen/proto.

• Make sure your binary is built with -force_load (or the equivalent on your platform), aimed at the TensorFlow library to ensure that it’s linked correctly. More detail on why this is necessary can be found in the next section, “Global Constructor Magic”. On Linux-like platforms, you’ll need different flags, more like -Wl, --allow-multiple-definition -Wl, --whole-archive.

• On iOS, you’ll also need to link in the Accelerator framework, since this is used to speed up some of the operations.

You may find Android a bit easier to work with, since you’ll only need to pull in a Java library contained in a jar, and you can download nightly precompiled versions.

Global Constructor Magic

One of the subtlest problems you may run up against is the “No session factory registered for the given session options” error when trying to call TensorFlow from your own application. To understand why this is happening and how to fix it, you need to know a bit about the architecture of TensorFlow.

The framework is designed to be very modular, with a thin core and a large number of specific objects that are independent and can be mixed and matched as needed. To enable this, we needed a coding pattern in C++ that easily let modules notify the framework about the services they offer, without requiring a central list that would have to be updated separately from each implementation. We also needed a way for separate libraries to add their own implementations without needing a recompile of the core.

To achieve this capability, we ended up using a registration pattern in a lot of places. In the code, it looks something like this:

class MulKernel : OpKernel {
  Status Compute(OpKernelContext* context) { … }
};
REGISTER_KERNEL(MulKernel, “Mul”);

This would be in a standalone .cc file that linked into your application, either as part of the main set of kernels or as a separate custom library. The magic part is that the REGISTER_KERNEL() macro is able to inform the core of TensorFlow that it has an implementation of the Mul operation so that it can be called in any graphs that require it.

From a programming point of view, this setup is very convenient. The implementation and registration code live in the same file, and adding new implementations is as simple as compiling and linking it in. The difficult part comes from the way that the REGISTER_KERNEL() macro is implemented. C++ doesn’t offer a good mechanism for doing this sort of registration, so we have to resort to some tricky code. Under the hood, the macro is implemented so that it produces something like this:

class RegisterMul {
 public:
  RegisterMul() {
    global_kernel_registry()->Register(“Mul”, [](){
      return new MulKernel()
    });
  }
};
RegisterMul g_register_mul;

What this is saying is that there’s a class RegisterMul with a constructor that tells the global kernel registry what function to call when somebody asks it how to create a Mul kernel. Then there’s a global object of that class, and so the constructor should be called at the start of any program.

If you’ve followed that, hopefully it sounds sensible, right? The unfortunate part is that the global that’s defined is not used by any other code, so linkers not designed with this in mind will decide that it can be deleted. As a result, the constructor is never called, and the class is never registered. All sorts of modules use this pattern in TensorFlow, and it happens that Session implementations are the first to be looked for when the code is run, which is why it shows up as the characteristic error when this problem occurs.

The solution is to force the linker to not strip any code from the library, even if it believes it’s unused. On iOS, this step can be accomplished with the -force_load flag, specifying a library path, and on Linux you need --whole-archive. These persuade the linker to not be as aggressive about stripping, and they should retain the globals.

The actual implementation of the various REGISTER_* macros is a bit more complicated in practice, but they all suffer the same underlying problem. If you’re interested in how they work, https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/framework/op_kernel.h#L1091 is a good place to start investigating.

Protobuf Problems

TensorFlow relies on the Protocol Buffer library, commonly known as protobuf. This library takes definitions of data structures and produces serialization and access code for them in a variety of languages. The tricky part is that this generated code needs to be linked against shared libraries for the exact same version of the framework that was used for the generator. This can be an issue when protoc, the tool used to generate the code, is from a different version of protobuf than the libraries in the standard linking and include paths. For example, you might be using a copy of protoc that was built locally in ~/projects/protobuf-3.0.1.a, but you have libraries installed at /usr/local/lib and /usr/local/include that are from 3.0.0.

The symptoms of this issue are errors during the compilation or linking phases with protobufs. Usually, the build tools take care of this, but if you’re using the makefile, make sure you’re building the protobuf library locally and using it, as shown in https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/makefile/Makefile#L18.

Another situation that can cause problems is when protobuf headers and source files need to be generated as part of the build process. This process makes building more complex, since the first phase has to be a pass over the protobuf definitions to create all the needed code files, and only after that can you go ahead and do a build of the library code.

The other thing about protobufs that’s tricky is that they generate headers that are needed as part of the C++ interface to the overall TensorFlow library. This complicates using the library as a standalone framework, as described in the next section.

Protobuf Compatibility Issues

If your application is already using version 1 of the protocol buffers library, you may have trouble integrating TensorFlow because it requires version 2. If you just try to link both versions into the same binary, you’ll see linking errors because some of the symbols clash. To solve this particular problem, we have an experimental script at http://bit.ly/rename-protobuf-sh.

You need to run this as part of the makefile build, after you’ve downloaded all the dependencies:

tensorflow/contrib/makefile/download_dependencies.sh
tensorflow/contrib/makefile/rename_protobuf.sh

Calling the TensorFlow API

Once you have the framework available, you then need to call into it. The usual pattern is that you first load your model, which represents a preset set of numeric computations, and then you run inputs through that model (for example, images from a camera) and receive outputs (for example, predicted labels).

On Android, we provide the Java Inference Library that is focused on just this use case, while on iOS and Raspberry Pis you call directly into the C++ API.

Here’s what a typical Inference Library sequence looks like on Android:

// Load the model from disk.
TensorFlowInferenceInterface inferenceInterface = 
	new TensorFlowInferenceInterface(assetManager, 
	modelFilename);


// Copy the input data into TensorFlow.
inferenceInterface.feed(inputName, 
	floatValues, 1, inputSize, inputSize, 3);


// Run the inference call.
inferenceInterface.run(outputNames, logStats);


// Copy the output Tensor back into the output array.
inferenceInterface.fetch(outputName, outputs);

You can find the source of this code in the Android examples at http://bit.ly/TensorFlowImageClassifier-java.

Here’s the equivalent code for iOS:

// Load the model.
PortableReadFileToProto(file_path, &tensorflow_graph);


// Create a session from the model.
tensorflow::Status s = session->Create(tensorflow_graph);
if (!s.ok()) {
  LOG(FATAL) << "Could not create TensorFlow Graph: " << s;
}


// Run the model.
std::string input_layer = "input";
std::string output_layer = "output";
std::vector<tensorflow::Tensor> outputs;
tensorflow::Status run_status = session->Run(
	{{input_layer, image_tensor}},
                               {output_layer}, {}, &outputs);
if (!run_status.ok()) {
  LOG(FATAL) << "Running model failed: " << run_status;
}


// Access the output data.
tensorflow::Tensor* output = &outputs[0];

This is all based on the iOS sample code at the iOS sample code at https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/ios/simple, but there’s nothing iOS-specific; the same code should be usable on any platform that supports C++.

C++ op API

On the desktop side, there’s a great API that autogenerates classes for every op at http://bit.ly/api-guides based on the op definitions. This is very handy if you need to create graphs dynamically from C++ rather than loading them, and you can see this in practice in the label_image example at http://bit.ly/label-image. Unfortunately, this automatic code generation adds extra complexity to the build process, so it’s not currently supported on mobile devices.

Training On-Device

Most applications of TensorFlow on mobile and embedded devices are focused on inference, taking a model that’s been trained in the cloud and running it locally with unchanging parameters. There are some interesting use cases for on-device training emerging, though, such as federated learning. The implementation of federated learning that is shipped with Google Keyboard uses TensorFlow under the hood, so it’s definitely possible to use it like for on-device training. It’s not a heavily used path, however, so you’ll find there are some challenges:

• The C++ API doesn’t yet support automatically figuring out gradient ops for a forward model pass. Instead, you’ll need to generate your training graph in Python to use automatic differentiation and export it, or manually add the right ops by manually building a GraphDef from NodeDefs in C++.

• You’ll need to make sure you include the right training ops, since these are typically not included in mobile builds because we expect most applications of TensorFlow on mobile will only use inference. See “What Ops Are Available on Mobile?” for more information on adding op implementations.

What Are the Minimum Device Requirements for TensorFlow?

You need at least one megabyte of program memory and several megabytes of RAM to run the base TensorFlow runtime, so it’s not suitable for DSPs or microcontrollers. Other than those, the biggest constraint is usually the calculation speed of the device, and if you can run the model you need for your application with a low enough latency. It’s a good idea to use the benchmarking tools described earlier to get an idea of how many FLOPs are required for a model, and then use that to make rule-of-thumb estimates of how fast they will run on different devices. For example, a modern smartphone might be able to run 10 GFLOPs per second, so the best you could hope for from a 5 GFLOP model is 2 frames per second, although you may do worse, depending on what the exact computation patterns are.

This model dependence means that it’s possible to run TensorFlow even on very old or constrained phones, as long as you optimize your network to fit within the latency budget, and possibly within limited RAM, as well. For memory usage, you mostly need to make sure that the intermediate buffers that TensorFlow creates aren’t too large, which you can examine in the benchmark output too.

What’s Up with All the Different Saved File Formats?

You may find yourself confused by all the different ways TensorFlow can save out graphs. Here’s a rundown of some of the different components and what they are used for. The objects are mostly defined and serialized as protocol buffers:

NodeDef

Defines a single operation in a model. It has a unique name, a list of the names of other nodes it pulls inputs from, the operation type it implements (for example, Add or Mul), and any attributes that are needed to control that operation. This is the basic unit of computation for TensorFlow, and all work is done by iterating through a network of these nodes, applying each one in turn. One particular operation type that’s worth knowing about is Const, since this type holds information about a constant. This may be a single, scalar number or string, but it can also hold an entire multidimensional tensor array. The values for a Const are stored inside the NodeDef, and so large constants that contain a lot of values can take up a lot of room when serialized.

Checkpoint

Another way of storing values for a model is by using Variable ops. Unlike Const ops, these don’t store their content as part of the NodeDef, so they take up very little space within the GraphDef file. Instead, their values are held in RAM while a computation is running, and then saved out to disk as checkpoint files periodically. This typically happens as a neural network is being trained and weights are updated, so it’s a time-critical operation and may happen in a distributed fashion across many workers. As a result, the file format has to be both fast and flexible. They are stored as multiple checkpoint files, together with metadata files that describe what’s contained within the checkpoints. When you’re referring to a checkpoint in the API (for example, when passing a filename in as a command-line argument), you’ll use the common prefix for a set of related files. If you had these files:

/tmp/model/model-chkpt-1000.data-00000-of-00002
/tmp/model/model-chkpt-1000.data-00001-of-00002
/tmp/model/model-chkpt-1000.index
/tmp/model/model-chkpt-1000.meta

you would refer to them as /tmp/model/chkpt-1000.

GraphDef

GraphDef has a list of NodeDefs, which together define the computational graph to execute. During training, some of these nodes will be variables, so if you want to have a complete graph you can run, including the weights, you’ll need to call a restore operation to pull those values from checkpoints. Because checkpoint loading has to be flexible to deal with all of the training requirements, this can be tricky to implement on mobile and embedded devices, especially those with no proper filesystem available, like iOS. This is where the freeze_graph.py script comes in handy. As mentioned, Const ops store their values as part of the NodeDef, so if all the Variable weights are converted to Const nodes, then we only need a single GraphDef file to hold the model architecture and the weights. Freezing the graph handles the process of loading the checkpoints, and then converts all Consts to Variables. You can then load the resulting file in a single call, without having to restore variable values from checkpoints. One thing to watch out for with GraphDef files is that sometimes they’re stored in text format for easy inspection. These versions usually have a .pbtxt filename suffix, whereas the binary files end with .pb.

FunctionDefLibrary

FunctionDefLibrary appears in GraphDef and is effectively a set of subgraphs, each with information about their input and output nodes. Each subgraph can then be used as an op in the main graph, allowing easy instantiation of different nodes, in a similar way to how functions encapsulate code in other languages.

MetaGraphDef

A plain GraphDef only has information about the network of computations but doesn’t have any extra information about the model or how it can be used. MetaGraphDef contains a GraphDef defining the computation part of the model, but also includes information like signatures, which are suggestions about which inputs and outputs you may want to call the model with, data on how and where any checkpoint files are saved, and convenience tags for grouping ops together for ease of use.

SavedModel

It’s common to want to have different versions of a graph that rely on a common set of variable checkpoints. For example, you might need a GPU and a CPU version of the same graph, but keep the same weights for both. You might also need some extra files (like label names) as part of your model. The SavedModel format addresses these needs by letting you save multiple versions of the same graph without duplicating variables and by storing asset files in the same bundle. Under the hood, it uses MetaGraphDef and checkpoint files, along with extra metadata files. It’s the format that you’ll want to use if you’re deploying a web API using TensorFlow Serving, for example.

How Do You Get a Model You Can Use on Mobile?

In most situations, training a model with TensorFlow gives you a folder containing a GraphDef file (usually ending with the .pb or .pbtxt extension) and a set of checkpoint files. What you need for mobile or embedded deployment is a single GraphDef file that’s been “frozen” or had its variables converted into inline constants so everything’s in one file. To handle the conversion, you’ll need the freeze_graph.py script that’s held in tensorflow/pythons/tools/freeze_graph.py.

Run it like this:

bazel build tensorflow/python/tools:freeze_graph
bazel-bin/tensorflow/python/tools/freeze_graph 
 --input_graph=/tmp/model/my_graph.pb 
 --input_checkpoint=/tmp/model/model.ckpt-1000 
 --output_graph=/tmp/frozen_graph.pb 
 --input_node_names=input_node 
 --output_node_names=output_node

The input_graph argument should point to the GraphDef file that holds your model architecture. It’s possible that your GraphDef has been stored in a text format on disk, in which case it’s likely to end in .pbtxt instead of .pb, and you should add an extra --input_binary=false flag to the command.

The input_checkpoint should be the most recent saved checkpoint. As previously mentioned, you need to give the common prefix to the set of checkpoints here, rather than a full filename. output_graph defines where the resulting frozen GraphDef will be saved. Because it’s likely to contain a lot of weight values that take up a large amount of space in text format, it’s always saved as a binary protobuf. output_node_names is a list of the names of the nodes that you want to extract the results of your graph from. This list is needed because the freezing process must understand which parts of the graph are actually needed, and which are artifacts of the training process, such as summarization ops. Only ops that contribute to calculating the given output nodes will be kept. If you know how your graph is going to be used, these should just be the names of the nodes you pass into Session::Run() as your fetch targets. If you don’t have this information handy, you can get some suggestions on likely outputs by running the summarize_graph tool.

Because the output format for TensorFlow has changed over time, there are a variety of other less commonly used flags available too, such as input_saver, but hopefully you shouldn’t need these on graphs trained with modern versions of the framework.

Using the Graph Transform Tool

A lot of the things you need to do to efficiently run a model on-device are available through the Graph Transform Tool interface. This command-line tool takes an input GraphDef file, applies the set of rewriting rules you request, and writes out the result as a GraphDef.

Removing training-only nodes

TensorFlow GraphDefs, produced by the training code, contain all the computation that’s needed for back-propagation and updates of weights, as well as the queuing and decoding of inputs and the saving out of checkpoints. None of these nodes are needed during inference, and some of the operations, such as checkpoint saving, aren’t even supported on mobile platforms. To create a model file that you can load on devices, delete those unneeded operations by running the strip_unused_nodes rule in the Graph Transform Tool.

The trickiest part of this process is figuring out the names of the nodes you want to use as inputs and outputs during inference. You’ll need these anyway once you start to run inference, but you also need them here so the transform can calculate which nodes are not needed on the inference-only path. These nodes may not be obvious from the training code because inputs are often fed through queues and decoding operations, and outputs feed into loss calculations, so they’re not the final op in the graph. Unfortunately, there’s no automatic way to discover these, and I often end up visualizing the graph and doing some detective work to figure out the right values.

Remember that mobile applications typically gather their data from sensors and have it as arrays in memory, whereas training involves loading and decoding representations of the data stored on disk. In the case of Inception v3, for example, there’s a DecodeJpeg op at the start of the graph that’s designed to take JPEG-encoded data from a file retrieved from disk and turn it into an arbitrary-sized image. After that there’s a BilinearResize op to scale it to the expected size, followed by a couple of other ops that convert the byte data into floats and and scales their values to the range that the rest of the graph expects. A typical mobile app skips most of these steps because it’s getting its input directly from a live camera, so the input node you actually supply will be the output of the Mul node in this case. Figure 1-1 shows a diagram of an Inception input module.

Figure 1-1. Diagram of an Inception input module

You’ll need to do a similar process of inspection to figure out the correct output nodes.

If you’ve just been given a frozen GraphDef file, and you’re not sure about the contents, we recommend using the summarize_graph tool to print out information about the inputs and outputs it finds from the graph structure. Here’s an example with the original Inception v3 file:

bazel run tensorflow/tools/graph_transforms:summarize_graph -- 
 --in_graph=tensorflow_inception_graph.pb

Once you have an idea of what the input and output nodes are, you can feed them into the graph transform tool as the --input_names and --output_names arguments, and call the strip_unused_nodes transform, like this:

bazel run tensorflow/tools/graph_transforms:transform_graph --
--in_graph=tensorflow_inception_graph.pb
--out_graph=optimized_inception_graph.pb --inputs='Mul'
--outputs='softmax' --transforms= 
 ' strip_unused_nodes(type=float,
shape="1,299,299,3") fold_constants(ignore_errors=true) 
 fold_batch_norms
fold_old_batch_norms'

One thing to look out for is that you need to specify the size and type that you want your inputs to be. This is because any values you’re going to be passing in as inputs to inference need to be fed to special Placeholder op nodes, and the transform may need to create them if they don’t already exist. In the case of Inception v3, for example, a Placeholder node replaces the old Mul node that used to output the resized and rescaled image array, since we’re going to be doing that processing ourselves before we call TensorFlow. It keeps the original name, though, which is why we always feed in inputs to Mul when we run a session with our modified Inception graph.

After you’ve run this process, you’ll have a graph that only contains the actual nodes you need to run your prediction process. This is the point where it becomes useful to run metrics on the graph, so it’s worth running summarize_graph again to understand what’s in your model.

Quantized Representation

Because we’re dealing with large arrays of numbers where the values are usually distributed within a common range, encoding those values linearly into eight bits using the minimum and maximum of the float values as the extremes works well. In practice, this looks a lot like a block floating-point representation, though it’s actually a bit more flexible. To understand how this works, here’s an example of encoding a floating point array, with the original values:

[-10.0, 20.0, 0]

By scanning the array, you can see that the minimum and maximum values are -10.0 and 20.0. Take each value in the array, subtract the float minimum from it, divide it by the difference between the minimum and the maximum to get a normalized value between 0.0 and 1.0, and then multiply by 255 to fit it into 8 bits. This gives us:

[((-10.0 - -10.0) / 30.0) * 255, ((20.0 - -10.0) / 30.0) * 255,
  ((0.0 - -10.0) / 30.0) * 255]

which resolves into:

[0, 255, 85]

The crucial thing when dealing with this representation is to remember that it’s meaningless without also knowing the min and max that it’s based on. It’s best to think of it as a compression scheme for real numbers, where the range is needed to make sense of every value. Within TensorFlow, this means every time a quantized tensor is passed through the graph, you need to make sure that two auxiliary tensors holding the minimum and maximum float values for the main tensor are always wired in. All the operations that accept quantized buffers as inputs require them, and each quantized output always has two associated scalar outputs producing the range for the output.

Representation Drawbacks

The nice thing about this representation is that it’s very general. You can use it to hold traditional, fixed-point values if you set the min and max to powers of two, the range doesn’t have to be symmetrical as in typical signed representations, and it can be adapted to hold almost any scale of values.

These properties were important when we first started working with quantization, since we didn’t have a clear understanding of what constraints we could place on the values without losing overall precision. As we’ve gained more experience, we’ve realized that we can be more restrictive without significantly affecting accuracy, and there are disadvantages to having such a flexible format.

One of the fundamental problems is that the range doesn’t necessarily have to include zero. For example, you can validly specify the minimum as 10.0 and the maximum as 20.0. This makes implementing a lot of algorithms unnecessarily complicated; for example, there’s no easy way to express adding zero onto a number with that representation. It also turns out that zero is an unusually common number in neural networks, since it’s used in the padding for convolutions beyond the edges of an image and is the output for any negative numbers from the Relu activation function. This creates the subtle problem that if zero doesn’t have an exact representation—for example, if its closest encoded value actually decodes to 0.1 rather than 0.0—the error introduces a bias that hurts the overall accuracy of the network.

Quantization works on neural networks when the errors introduced by rounding to eight bits look similar to the kind of noise that they’re trained to cope with anyway. This means the quantization errors must be roughly uniform, or at least average out to zero over large enough runs. This uniformity holds true if every number in the encoded range comes up with the same frequency, but when zero is present much more often than any other number, than whatever quantization error is present for that value will be amplified.This results in a systematic bias that can skew the final results of the network.

Another drawback is that it’s possible to create nonsensical or invalid representations, for example, where the min and max are equal or the minimum is greater than the maximum. This kind of representation is probably a sign that the format is a flawed way to represent what we’re trying to hold. Yet another issue is that when we’re trying to use the same format for 32-bit numbers, the float values for the ranges become very large and unwieldy.

A possible solution might be to switch to a representation where we just express the real value of an incremental increase of 1 in the code. This should remain small enough to avoid some of these problems. It also might be worth just restricting the offset choices to just signed or unsigned so that the ranges are just from zero to max, or symmetrical so that the minimum of the range is always the negative of the maximum. Even with symmetrical ranges, there’s still the challenge that a twos-complement signed 8-bit value has a minimum of -128 but a maximum of 127. That means if you assign your float ranges naively to those coding values you’ll end up with the real value of zero not falling exactly on the zero encoding.

To address some of these issues, we’ve ended up constraining what values the min and max can actually be. For example, we always try to produce ranges that include zero; and if min and max are too close together, we nudge them apart by a small amount. In the future, we may enforce symmetrical or positive ranges too. Luckily, we’re able to handle this without changing the representation; instead, we just enforce these constraints whenever the code calculates ranges for quantized buffers.

Using Quantized Calculations in TensorFlow

The usual process is to take a model that’s been trained in floating point and run it through the quantize_nodes conversion process using the graph transform tool. What this does is replace normal float operations with their quantized equivalents, where those exist. Because the implementations of 8-bit algorithms are so different from floating-point versions, we’ve focused on ops that are commonly used in popular models to start with. You can find the most up-to-date list here; but at the time of writing, here are the operations that will run natively in 8-bit:

• BiasAdd

• Concat

• Conv2D

• MatMul

• Relu

• Relu6

• AvgPool

• MaxPool

• Mul

These operations are enough to implement the Inception networks and many other convolutional models. For any sets of adjacent nodes that belong to these types, 8-bit quantized buffers will be passed between them. If an unsupported operation is encountered, any quantized tensors will be converted to floats and fed in as normal, with a conversion back to quantized happening just before the next 8-bit operation.

Let’s look at an example of the Relu operation. All Relu does is take a tensor array of values and output a copy of its input. Any negative numbers are replaced by zeros. An initial graph might look like Figure 1-2.

Figure 1-2. Simple float Relu graph

The yellow boxes are tensors, and the square blue box is the operation. The first thing the quantize_nodes rewriting rule does is replace the Relu with the equivalent 8-bit version (called Quantized​Relu). At this point, you’re not looking at what ops are surrounding it in the graph though, so make sure the inputs and outputs are converted to float, as in Figure 1-3.

Figure 1-3. Quantized Relu graph

Handle this conversion for the inputs using the Quantize op. As we mentioned previously, quantized buffers only make sense when we also know what range was used to encode them, so this op outputs the float minimum and maximum as well as as the coded 8-bit values. This is then operated on by the QuantizedRelu op, which outputs 8-bit values together with the range. In fact, for this implementation the output range is the same as the input, so we could wire the min/max from Quantize directly as inputs to Dequantize, but to keep the implementation consistent it’s better to always have range outputs for every quantized one as a convention.

The encoded 8-bit values from QuantizedRelu are fed into a Dequantize op together with the range, which produces a final float output. All in all, this probably seems a very convoluted (if you’ll excuse the pun) way to handle 8-bit calculations. The important part though is that this is a generic way to substitute any individual op in the graph with an 8-bit equivalent without needing to understand any of the larger picture. We can implement these substitutions as a first pass, and then go through the resulting graph and remove inefficiencies. Figure 1-4 shows how that works if you have a pair of 8-bit operations with a dequantize/quantize stage between them.

Figure 1-4. Graph showing the removal of unneeded quantization ops

Here the graph transform is able to spot that the subgraph of operations marked in red on the left all cancel each other out, producing the same output as its original input. By spotting that pattern, the transform tool can remove those unnecessary ops and produce the simplified version of the graph on the right.

Why Does Quantization Removal Matter?

Quantization removal is important because it means the performance of 8-bit graphs depends a lot on how many of the operations in the model have quantized equivalents, and if unconverted ops cause a lot of conversions back and forth between float and 8-bit. A few conversions at the beginning or end of a graph won’t matter too much; for example, SoftMax usually works with a very small amount of data and comes as the final step of a graph, so it’s not usually a bottleneck. However, having a mix of float and 8-bit in the heart of your graph can squander any advantages of moving to the lower bit depth for calculations.

Another problem to watch out for is that running the algorithms efficiently for 8-bit requires architecture-specific SIMD code. We use the gemmlowp library to implement the matrix multiplies that make up the bulk of neural network calculations, but currently that’s only optimized for ARM NEON and mobile Intel chips. That means desktop performance for 8-bit on x86 is actually worse than float, because we use floating-point libraries like Eigen that have been highly optimized for those chips, whereas the 8-bit code hasn’t been.

Activation Ranges

One challenge we haven’t talked about is that some operations that take in 8-bit inputs actually produce 32-bit outputs. For example, if you’re doing a matrix multiply, each output value will be the sum of a series of 8-bit input numbers multiplied with each other. The result of multiplying 2 8-bit inputs is a 16-bit value, and then to accurately accumulate a number of them, you need something larger than 16 bits, which on most chips means a 32-bit value. Subsequent quantized operations that use this result as an input don’t want 32 bits of data, though, because that’s no more efficient than a float value and is considerably harder to deal with. Instead, we convert those 32 bits into 8-bit equivalents.

You could imagine just calculating the smallest and largest possible values that could be produced from a particular matrix multiplication, and using those as the range to extract the highest eight bits from the wider output values. This encoding would be very inefficient, however, because the actual inputs to most neural network operations don’t have extreme distributions, so the everyday smallest and largest values that will be encountered are much tinier than their theoretical limits. Using the extreme ranges would mean that most of the bits in the coding would be wasted.

To address that issue, we need to know what the extremes will be for commonly encountered data. Unfortunately, this information has proved to be very hard to calculate analytically, so we’ve ended up having to empirically observe the statistics while running real examples through the entire network. There are three main ways of handling this, described in the following sections.

Dynamic Ranges

The easiest way to get started is to insert an op that runs through the output from a 32-bit-producing operation just after it’s been generated and figures out what the current range of those values actually is. This range can then be fed into a Requantization operation that converts 32-bit tensors into 8-bits, given a target range for the output.

The big advantage of this method is that it doesn’t need any extra data or user intervention, so it’s the default way that the quantize_nodes transform uses. This method makes it straightforward to take a float network, convert it to eight bits, and then start checking the accuracy and performance.

The downside is that the range calculation has to be run every time inference is performed, which means looking through every output value and figuring out the minimum and maximum across each buffer. This is extra work that reduces performance (usually by a fairly small amount) on the CPU; but for specialized hardware platforms it’s even worse, because they may not be able to handle this sort of dynamic rescaling.

Observed Ranges

The next-easiest approach is to run a representative set of example data through the network, track what the ranges are for each op over time, and then use a statistical methodology to estimate reasonable values to cover all those ranges without wasting too much precision. Unfortunately this approach is hard to do automatically, since the idea of what constitutes representative data is very application-dependent. For example, with an image-recognition network, plugging in random noise or extreme values as inputs would only exercise a few of the pattern recognition paths, so the resulting ranges wouldn’t be very useful. Instead, you need to have inputs that represent the kind of data that’s expected, like training data.

To make this possible, we offer a multistage approach. By using the insert_logging rule, you can add debug ops that output the values of the ranges every time the model is run. Here’s a complete example of how to do everything you need on the pretrained InceptionV3 graph.

First, download and decompress the model file:

mkdir /tmp/model/
curl
 "https://storage.googleapis.com/download.tensorflow.org/ 
 models/inception_dec_2015.zip" 
 -o /tmp/model/inception_dec_2015.zip
unzip /tmp/model/inception_dec_2015.zip -d /tmp/model/

Then quantize the graph to use 8-bit calculations:

bazel build tensorflow/tools/graph_transforms:transform_graph
bazel-bin/tensorflow/tools/graph_transforms/transform_graph 
 --in_graph="/tmp/model/tensorflow_inception_graph.pb" 
 --out_graph="/tmp/model/quantized_graph.pb" --inputs='Mul:0' 
 --outputs='softmax:0' --transforms='add_default_attributes
strip_unused_nodes(type=float, shape="1,299,299,3")
remove_nodes(op=Identity, op=CheckNumerics)
fold_old_batch_norms
quantize_weights
quantize_nodes
Strip_unused_nodes'

Once that’s complete, run the label_image example to make sure the model is still giving the expected results. It runs on a picture of Grace Hopper by default, so you should see Uniform as the top result:

bazel build tensorflow/examples/label_image:label_image
bazel-bin/tensorflow/examples/label_image/label_image 
 --input_mean=128 --input_std=128 --input_layer=Mul 
 --output_layer=softmax --graph=/tmp/model/quantized_graph.pb 
 --labels=/tmp/model/imagenet_comp_graph_label_strings.txt

With that working, next you’ll append log operations onto the outputs of all the RequantizationRange nodes:

bazel-bin/tensorflow/tools/graph_transforms/transform_graph 
 --in_graph=/tmp/model/quantized_graph.pb 
 --out_graph=/tmp/model/logged_quantized_graph.pb 
 --inputs=Mul 
 --outputs=softmax 
--transforms='insert_logging(op=RequantizationRange, 
 show_name=true, message="__requant_min_max:")'

Now you can run the graph, and stderr should contain log statements showing what values the ranges have on that run:

bazel-bin/tensorflow/examples/label_image/label_image 
 --input_mean=128 --input_std=128 
 --input_layer=Mul --output_layer=softmax 
 --graph=/tmp/model/logged_quantized_graph.pb 
 --labels=/tmp/model/imagenet_comp_graph_label_strings.txt 2> 
 /tmp/model/logged_ranges.txt
cat /tmp/model/logged_ranges.txt

You should see a series of lines like this:

;conv/Conv2D/eightbit/requant_range__print*; 
  *requant_min_max:[-20.887871] [22.274715]

Each one of these encodes the range values for a given operation. In this case, we’re just running the graph once on a single image. In a real application, we’d want to run hundreds of representative images to get a good sample of all the usual ranges.

Finally, we want to take the information we’ve gathered and replace the dynamic range calculations with simple constants, using the freeze_requantization_ranges transform:

bazel-bin/tensorflow/tools/graph_transforms/transform_graph 
  --in_graph=/tmp/model/quantized_graph.pb 
  --out_graph=/tmp/model/ranged_quantized_graph.pb 
  --inputs=Mul 
  --outputs=softmax 
--transforms='freeze_requantization_ranges 
  (min_max_log_file=/tmp/model/logged_ranges.txt)'

If you run the label_image example on this new network, you should see uniform as the top result still, though the exact numbers may be slightly different from before:

bazel-bin/tensorflow/examples/label_image/label_image 
 --input_mean=128 --input_std=128 --input_layer=Mul 
 --output_layer=softmax 
 --graph=/tmp/model/ranged_quantized_graph.pb 
 --labels=/tmp/model/imagenet_comp_graph_label_strings.txt

Trained Ranges

The final way to figure out good ranges for the activation layers is to integrate the calculations into the training process. You can do this using the FakeQuantWithMinMaxVars node. This operation can be placed at various points in the graph to simulate quantization inaccuracies by rounding its float inputs to a fixed number of levels (typically 256), within a range set by two Variable inputs representing the minimum and maximum. These range inputs are updated during the learning process based on the minimums and maximums actually required, as the gradient values are passed through to these inputs.

Unfortunately, there aren’t any convenience functions to add these ops to your models in Python, so if you do go down this route you’ll need to manually insert the ops and variables anywhere a range is needed. This will typically be between weights and the ops they’re used in, and on the outputs of Conv2D or MatMul nodes.

There are a couple of big advantages to this approach that make up for the inconvenience. When we take a pretrained float model and simply convert it to 8-bit, there’s typically a small loss in accuracy—for example, a drop in top-1 precision on InceptionV3 from 78% to 77%. Training with the quantization baked in usually shrinks that loss dramatically, sometimes to the point where the networks perform as well as float. Having the ranges known ahead of time also helps latency during inference, since we don’t have to do runtime calculations to determine the minimum and maximum.