Chapter 3. Making Your Strategy Observable

In this chapter, we’ll cover the terms observability and monitoring and the role they play in establishing a reliable way to “see” how developers are finding and using your API product. By collecting data on developers as they explore your API products, you can discover where you need to improve your developer experience and what features really matter to your audience, and you can uncover bugs and problems your developers are dealing with as they try to use your API.

Above all, it is important that you get in the habit of “designing in” observability and monitoring metrics when you create your API products. As Geoffrey Moore, author of Crossing the Chasm (HarperCollins), put it: “Without big data, you are blind and deaf and in the middle of a freeway.”

First, let’s talk about the terms observability and monitoring, and then we can explore the notion of product analytics and how you can use it to improve your API products.

What Is Observability?

Observability is a method of determining the internal condition of a system through external outputs—in other words, having the ability to “see” into a running system to determine its overall health and efficiency. Launching an online service means you are responsible for maintaining that service—keeping it up and running—in order to meet the needs of users.

The challenge of creating and maintaining any API product is that, since it is virtual, it is hard to “see” what is going on from moment to moment. Is the service up and running? Are customers able to successfully complete an order? Is that a machine online that is running out of memory?

To answer these questions, we need to observe external parts of the system (number of completed orders, wait times for requests, etc.) and derive meaning from those observations (e.g., it is taking too long to read a record from the database) in order to, if needed, do something to improve the efficiency of the product.

Monitoring and Observability

It is worth pointing out the difference between monitoring and observability. In Site Reliability Engineering (O’Reilly), the word monitoring is defined as “Collecting, processing, aggregating, and displaying real-time quantitative data about a system.” This is, basically, the work of identifying key metrics, in real time, and making them available for review.

As we mentioned earlier in this chapter, observability is the process of determining the internal state of a system based on external outputs. We can observe the health of a system by inspecting and analyzing the results of monitoring. Quality observations are dependent on solid monitoring output.

You can greatly improve the observability of your API by paying close attention to the values you monitor or publish. You can’t easily determine the time it takes to complete a user order if you don’t monitor the latency of the checkout and payment steps in your product.

Levels of Observability

Just as there are different points of view, there are different types of observability. Successful API products support observability for multiple points of view. For this report, we’ll focus on three observability types:

  • Machine

  • Service

  • Product

Machine-level observability

At the lowest level, you have machine-level observability. Typically machine-level observations include things like CPU, memory, disk access speeds, and other similar values. When you deploy your API product into production, it will run on individual physical or virtual machines (usually in clusters), and each machine has its own observable health status. For any API product, you should be able to view—at any time—the machine-level status of each running instance in your service cluster.

Service-level observability

The next level is service-level observability. Services are the dedicated operations that run on various machines. The ability to read and write data from the database, process an API request, validate inputs for an order, and the like. These are all service-level metrics. Essentially, you need to be able to view the service-level status of your system at all times. Often this is provided through a dashboard that lists the rate of object reads/writes, backlog of processing requests, and so on. Service-level metrics focus on that status of individual URL endpoints (e.g., /shopping/computeTaxes, customer/update) instead of that condition of machines (see above).

Product-level observability

At the top of the stack is product-level observability. At this level, you are monitoring the rate of key business metrics such as completed orders, added users, and abandoned shopping carts. Product-level metrics for your API are tied directly to your business goals (net new revenue, increased customers, etc.). It is essential that your API product implementation make it easy to track the status of these business goals. Usually these are not tied to any single endpoint but are, instead, derived from a series of steps (e.g., selecting an item, placing it in the cart, and checking out).

Successful API products support observability at all three levels. By combining metrics on machine performance, service reliability, and business goals, you can get a comprehensive view of the health and success of your API products.

Implementing API Product Analytics

API product analytics is the process of identifying important business-level goals (e.g., onboarding new developers), selecting monitoring values for that metric (count of added developers per day), and then inferring the success level for that product metric.

For this report, we’ll review three important product metrics since they relate directly to keeping a close eye on your API-first strategy (see Chapter 2):

  • Onboarding

  • Time to first hello

  • API depth

These are just examples of the kind of project-level observability you’ll need in order to create a successful API product.1

Onboarding Metrics

A key metric to track for API products is the onboarding of new developers. As discussed in “Land Developers First”, developers are often the first target audience for any API, and understanding the status of your onboarding experience is critical to the success of your API.

Example metrics you can set up as monitoring output in order to gain observability into your API onboarding are:

SignedUp

The act of opening up or creating an API account for your product. Usually some exchange of a developer’s identifying information for access to a subset of your API product.

APIKey

The moment a SignedUp developer creates an API key that allows them to start making API calls on your platform.

FirstCall

The first API call made using the APIKey for that SignedUp developer.

The above list is a skeleton of the onboarding process. By monitoring (and dashboarding) these values, you will get a view into the process and can make some decisions on how you can improve your overall onboarding experience.

Some questions to ask yourself as you review the data:

  • What is the average time from SignedUp to APIKey to FirstCall?

  • What percentage of developers who SignedUp reach FirstCall?

  • Is there a big fall-off between SignedUp and APIKey or between APIKey and FirstCall?

By analyzing the data, you may find you’re getting a high SignedUp rate but a low APIKey rate. Maybe it is too cumbersome to generate an APIKey. What if you made that easier or maybe automatically generated an API key for them?

Note

This onboarding example assumes there are three steps (SignedUp, APIKey, and FirstCall). If your product onboarding has more steps, you need to be sure to include all of them in your Onboarding Product metric.

At this stage, your goal is to turn prospects into customers as quickly and easily as possible. Maybe developers get an API key but aren’t motivated to make that first API call. The next product metric can help you with that.

Time-to-First-Hello Metrics

The time-to-first-hello (TTFH) metric (see “First Hello World”) is the one that tracks the developer making the very first API call on your product platform. This is a critical measure of the value of your API product. And the quicker you can get a developer to make that first call, the better. Jarkko Moilanen, coauthor of the book API Economy 101, says that “developers should be able to test APIs using code examples within just 180 seconds of engaging with the product.”2

There are a handful of values you can monitor in order to gain observability into your developers’ TTFH experience. For example:

APIErrors

Are developers experiencing errors when they try to use your API?

SDKBugs

Are there bugs or unexplained errors coming from SDKs your developers are using to access your APIs?

ImplementationFlaws

Sometimes developers end up using one or more of your APIs in ways you didn’t anticipate. For example, they need some bit of data about their customers, but it takes three or four API calls to get that data. These repeated calls might be more error-prone or introduce performance problems and even increase the likelihood of experiencing API call errors.

Paying attention to the API calls your developers are making gives you a pretty good view of what they understand about your API from documentation, examples, and any other sources like peer-level support from other developers. The most common barrier to TTFH is usually client-side API errors.

Poor documentation

Maybe developers can’t figure out the proper URL or arguments or HTTP methods to use when calling your API. If yes, maybe you need to modify your API product documentation.

Unexpected usage

Maybe they are using your API in ways you didn’t anticipate and are getting unexpected results. If this is true, you may need to offer more, clearer example calls to help developers get started.

Troublesome SDKs

Maybe they are using your own SDK (or ones created by other developers) and the SDK itself has bugs. You might need to encourage new developers to skip the SDKs to start to make sure they get a quality TTFH experience.

It is a good idea to treat every attempt by your developers as a kind of gift—one that tells you a bit more about the developer as well as your API. Learning from their experience gives you an opportunity to improve your product as well as their experience.

But TTFH is not the end of the line. It is also important to pay attention to how seasoned developers use your product.

API Engagement Metrics

Once you get developers signed up and past the first API call, you want to pay attention to what they do next. Do they explore other APIs in your product? Do they just keep making the same couple of API calls? Do they just disappear altogether? The long-term success of your API depends on building a strong relationship with your developers and helping them learn how to use more and more of your APIs. You want to get them to go “deeper” into your product.

Often developers start using your product because they have a particular problem they needed to solve and your API does that for them (see “Understanding API Product Strategy”). However, it is to your advantage to help developers discover more features of your API, too. It could turn out they have other problems that your product solves and they haven’t figured that out yet.

Monitoring for API Depth

Possible monitoring values to help gauge developers’ API engagement are:

  • A list of the APIs most commonly used (API-MCU)

  • The number of different API endpoints called by a single developer

  • Which SDKs are used and how often (if you support API SDKs)

Knowing the most commonly used APIs (API-MCU) can give you a valuable view of which parts of your product are seen by developers as most beneficial. You might be surprised by what you find. Sometimes the most popular APIs are not the ones you expected. Developers might be telling you they need more APIs in a domain, that they want easy access to data properties that are currently difficult to reach, and so on. In some cases, developers may be giving you hints on additional API products you can sell them.

Once you have good numbers for API-MCU, you can also get a direct count of the number of different endpoints called by each developer. For example, if developers typically only call a small subsection of your API, you might have an opportunity to expand their use of your product. Or developers might be telling you which APIs are not needed. This can help you focus your bug-fixing and feature-building activities for future releases.

Also, if you provide API SDKs for your developers, you can track which releases (by language) are more popular, or which ones are not used at all. Just like the API-MCU case, getting a handle on SDK usage can help you focus your efforts for future releases.

Increasing Developer’s API Engagement

A good way to increase your developer’s API engagement is to publish emails, blogs, and videos that cover select API features and/or SDKs (see “Introduction to Inbound Marketing”). For example:

Segment marketing

Initiating a mini-marketing campaign that targets a segment of your users with nudges to try out new features can be a great way to track the success of both your marketing efforts as well as the value of particular API features.

Feature onboarding

You can create dedicated “onboarding” experiences, or “pilots,” for new endpoints too. Write up a short tutorial with multiple steps developers can take to interact with a new API and set up metrics to track each step along the way. This gives you a chance to see if there are places where developers experience problems or just abandon the tutorial due to lack of interest. Using an account-based dashboard can help track onboarding goals and ensure pilots are successful.

Developer advisors

You can identify “heavy users” of your APIs and invite these developers to try out new experimental features or early updates to SDKs. Essentially, you’re inviting developers to tell you what they like, what they wish was different, and how to improve your product in future releases.

Increasing API depth is a great way to improve your relationship with developers and learn more about the API product space and how you can help solve your customers’ problems.

Summary

In this chapter we covered the terms observability and monitoring and talked about how you can use monitored metrics to observe the way developers are using your API. We also talked about the idea of API product analytics, including tracking API onboarding, time to first hello, and increasing the API depth of your developer community.

1 For an in-depth view on API product analytics, check out Moesif’s blog post “API Analytics Across the Developer Journey”.

2 Art Anthony, “API Onboarding Is Broken (and How to Fix It)”, Nordic APIs blog, January 2022.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset