Chapter 5. Guiding principles for API design

This chapter covers

  • API design principles for successful APIs
  • Cautionary notes on design antipatterns
  • API examples highlighting these principles

The technical aspects of REST APIs are only part of the puzzle. This chapter and the next few focus on the process of creating an API that’s not only functional but also delightful and usable.

Before I get to the meat of the process, I want to give you some advice to keep in mind during the process. This is definitely not a checklist; even when you’ve already done one of these things, you need to continue doing it. Think of it as learning to drive a car—even though you’ve checked your blind spot once, you need to continue doing so as you move forward. Similarly, this chapter is designed to help you create a mind-set while you go through the API design and development process that will help you make the right decisions to achieve success. I discuss the guiding principles of API creation and ideas to keep at the front of your mind while going through the steps to create your API.

The guiding principles in this chapter are as follows:

  • Don’t surprise your users.
  • Focus on use cases.
  • Copy successful APIs.
  • REST is not always best.
  • Focus on the developer experience, not the architecture:

    • Communication and consistency are critical.
    • Documentation should tell a story.

Following these principles will give you a great head start on building an API that’s not only irresistible for your customer developers, but fun to develop and easy to support.

5.1. Don’t surprise your users

First off, always remember that developers are people too. In fact, in the case of your API, developers are your customers, even if they’re in the same company using internal APIs. In the case of an API, your customer and partner developers should be treated with the same respect and consideration given to users of your revenue-producing products, even when the API doesn’t itself provide revenue for your organization. APIs that don’t produce money directly can enhance the products that end users do pay for—your main website, the mobile applications, or integrations—and make those products more compelling.

When designing an API, you’ll frequently encounter situations where it’d be easier to develop the API by cutting corners or moving away from standards. Using POST for all write actions, including updates and deletes, would be easier for the API development staff, but would also create more work and confusion for the developer customers of the API. Any decision of this type should be made to ease the burden on your customers even if it takes extra work on the development side. In short, put your customers ahead of your API development team, and make sure the experience is as smooth as possible. Whether you decide to use different verbs than usual for your methods, use nonstandard response and request formats, or organize your schema in a nonintuitive way, you should carefully consider the experience of the developers using the API. It’s often tempting to try to focus on scalability and performance, but the basic truth of this is that if you don’t focus on usability, you’re not going to create a usable API, and nobody is going to use it—at which point scalability and performance aren’t going to matter to anyone.

Any time you go off the beaten path, you create a “surprise” for your users, and they have to adapt their normal development style to suit the changes you’ve made. This design choice can result in a support burden in a couple of ways. First, users may be confused about how they’re supposed to use your nonstandard system and need assistance in learning how to work with it. Second, and even more important, using a nonstandard approach creates a situation where developers are quite likely to use the API in a way you don’t expect or support as they try to figure out the right way to interact with your system. Sticking with the standard interaction model wherever possible means that you and your users share the context of what that interaction should look like, whereas using a different approach means that the user is starting at square one and likely to make unexpected choices resulting in errors, or worse, code that works in one version and doesn’t work in the next because they misunderstood the logic behind your API. In any case where you’re considering extra work for your team versus ease of use for your developer customers, you should always perform the extra work on your side to reduce the support burden and increase developer engagement.

5.1.1. Flickr API example

An example of a company that didn’t follow REST best practices is Flickr. When it created its web API, Flickr described it as RESTful (the company has since changed the description to REST-like). Rather than exposing data as objects, staying within the usual guidelines of REST, Flickr exposed methods (see table 5.1). Flickr’s choice to make method-based calls limits the types of actions that can happen with the items in their system. The left column in table 5.1 shows what a method-based API looks like. Instead of operating on specific items within the system, each call is a command to do a specific thing. When compared to the RESTful calls on the right-hand side, the Flickr calls are much more difficult to understand as a complete system.

Table 5.1. Flickr calls and RESTful calls

Flickr call

RESTful call
GET /services/rest/?method=flickr.activity.userPhotos GET /services/rest/activity/userPhotos
POST /services/rest/?method=flickr.favorites.add POST/services/rest/favorites
POST /services/rest/?method=flickr.favorites.remove DELETE /services/rest/favorites/:id
POST /services/rest/?method=flickr.galleries.editPhoto PUT/services/rest/galleries/photo/:id

Flickr took a SOAP API—an API focused on actions, not resources—and tried to make it work in a more REST-like way, without the changes to the structure needed to make it work appropriately. That meant the API didn’t have a contract to tell developers how they were supposed to interact with the system—and no objects.

The Flickr API accepts only two methods: GET and POST. This means that it supports clients who are using limited frameworks, but it also means that a developer with the full methods available has to think carefully about which method to use for a call. Additionally, the URL isn’t used to describe the resource being accessed; rather, a method is called, and all calls are made to the same URL.

For instance, this is the easiest call in the Flickr API:

https://api.flickr.com/services/rest/?method=flickr.test.echo&name=value

This call is clearly not a REST call (even though it has rest right in the path). The “method” is the biggest clue that this system is working with verbs (methods) and not nouns (resources). But most developers can work with this, even though most libraries designed to work with REST expect a different URL for each specific resource.

Deleting a photo should be a DELETE to the photo resource, but instead Flickr uses the following call: 

https://api.flickr.com/services/rest/?method=flickr.photos.delete&photo_id=value

To call this, a POST must be made (which in the usual case would create a new resource). Once again a method is specified for the deletion, and the photo_id, which should be part of the URL as a resource, is instead specified as a parameter.

The error codes used are also nonstandard and aren’t passed back in the usual way. Figure 5.1 shows the list of error codes sent back for API calls that fail.

Figure 5.1. As with the resources and methods described in this figure, using inconsistent error codes means developers have to hunt through documentation to learn what the errors mean. Using the standard HTTP status codes helps alleviate this problem and saves times for your client developers.

Instead of using HTTP status codes, as is the usual pattern, Flickr is using an antipattern where a failed call returns a 2XX status code (indicating success) and then includes the error as part of the response. Using the photosets.delete method within the API Explorer yields the response shown in figure 5.2.

Figure 5.2. Even using the API Explorer for Flickr, you can see that there’s no standard HTTP status code (although this API is using HTTP) and that the message format doesn’t follow any standard industry pattern.

Again, when faced with this situation developers are unable to use the standard HTTP libraries, with their usual error handling, to write their code. The libraries designed to work with the Flickr API handle these methods, but when interacting with multiple APIs, the developer has to have special code targeted at the Flickr API.

All of Flickr’s methods follow this pattern. It’s a good thing that once you start working with its API you can get the hang of it and work through the issues caused by trying to get existing REST consumer libraries to work with a nonstandard setup. I know many developers who have decided not to use this API because of its poor design and implementation, who use other photo services because they trust an API more when it behaves as they expect. Flickr hasn’t changed its API, probably at least in part because switching to a true REST system would require a complete overhaul of the API, which would break existing clients. This situation is a shame, because Flickr is one of the most popular photo-sharing applications out there, but the API it has makes it challenging to recommend Flickr as a good platform on which to build.

In learning from this example, when designing your API you want to stick to standards wherever possible, both the explicitly described standards and the best practices for API services. Don’t stray from the path unless you must do so, and strive for consistency across your API endpoints in terms of organization, layout, behavior, and status codes.

5.1.2. Don’t make me think

One of the mottoes for excellent web design is Don’t make me think. This concept can be applied directly to API design as well. The flip side of not surprising your users is presenting your information in a clear and consumable way. Don’t make your users think. Make sure that your API behaves as they expect wherever possible. In particular, don’t expose your back-end database schema as the API, because it’s highly unlikely to be the right structure for end-user developers. You need to create an API that’s optimized to be an excellent interface to the front-end developer and avoid making decisions for back-end development that reduce this usability. Use cases are one great way to make sure you know which actions and workflows should be easy and fast. APIs that are created organically tend to end up being quite unusable, and developers attempting to integrate the API get frustrated or give up.

This is particularly true when you’re developing external web APIs for your organization, but it’s also true for internal systems. Remember that those internal APIs may well become external APIs, and they should be designed and developed so that they can easily be opened up to external users, consistent with the APIs you currently have externally. A huge part of the developer experience with an API is trust; developers have to trust that the system is reliable and that you’ve created an API mindfully and skillfully. It may not seem like a big deal if your server sends responses that are nonstandard or formatted incorrectly. It’s easier to develop systems that use methods rather than objects even when you’ve defined your API as RESTful. But this type of mistake will make your users suspicious that you threw together the system; they won’t know if you’re going to “fix” it later, requiring them to change the code consuming the API, or if you’re going to make other decisions that are nonstandard. They may even wonder if your poorly designed API will be deprecated because it doesn’t have the focus to thrive.

Sometimes it’s necessary to make exceptions, because your system needs to process information in a different way, or because some of your targeted users are working with platforms that can’t interface in the usual way. For instance, in the past there were frameworks such as Flash that weren’t able to send methods other than GET and POST, so in this case, an API designer who was targeting these users would need to make sure there was still a way to support these frameworks as clients. In such cases, most API developers chose to create headers to define exactly what the correct method was, and they documented it well. Because this was a common problem across APIs for this subset of users, a consistent method of working around it was extremely helpful for the client developers.

In a more general sense, when you need to move away from standards in order to support the needs of your system or users, how can you overcome the issues we’re discussing? You can avoid many of these problems with the judicious use of documentation, communication, and sample code. You need to tell the users exactly what to expect and how to use the system, and provide them with as many examples of successful interaction as possible. It’s important to create excellent documentation for any API, but when your API behaves differently than others, it’s vital to provide extra documentation that helps the user understand the context correctly, above and beyond the normal documentation and example code required to get your users pointed in the right direction for using your system.

One temptation is to cover up these eccentricities by providing client libraries that mask the unusual interface. These libraries are frequently referred to as software development kits (SDKs). SDKs go only so far in “fixing” the problem, because you’re requiring that users rely on these libraries in order to use your system, and if they step beyond the libraries, they’ll discover those oddities you were trying to cover up. When you’ve built SDKs for various languages to work with your API, you’ve added a great deal of technical debt: every time the API changes, all SDKs need to change as well. Additionally, creating heavy SDKs adds an unnecessary layer of abstraction between your client developers and the API itself. This makes it more difficult to triage problems—there’s one more point of potential failure. You’ve created a black box around something that shouldn’t be opaque. In general, if your API requires an SDK more complicated than an authentication library, your API needs work.

Exercise 1

Look for additional APIs that don’t follow standard rules. If you needed to develop an API that required nonstandard practices, how could you make it easier for your customer developers to use and implement?

5.2. Focus on use cases

If you take nothing else away from this chapter, understanding the value of use cases is the key to creating an API that’s successful and engaging. Without use cases to define the actions that should be easy in your API, it’s far too easy to create an API that’s technically sound but difficult to use. For instance, if you have a social application and you want users to be able to write to their activity stream, you should define the steps developers would need to follow and make sure that workflow flows smoothly for clients using the API. A use case is a set of steps describing the experience that an end user wants—a story-based workflow, if you will.

Use cases are vital throughout the process of creating an API. Once you’ve defined the business value you want to address with the API and how you’re going to measure success, the resulting use cases drive the rest of the process. During the process of defining your API using a modeling language, make sure you can see how each piece works together to support each required use case. In the case of a social application, for instance, you’d need to make sure that the login/authentication system works seamlessly with the activity—and that reading back the activity stream is similarly easy. When developing your API, this focus gives you the opportunity to release sections of your API incrementally, guiding you to releases that enable specific use cases. Once the API has been released, you can create tutorials based on the use cases, and communicate clearly to developers what kinds of clients you want them to make.

Assuming you already have a product, that’s a fantastic use case to consider. You already know what the user experience should be and how the pieces tie together. Creating a use case supporting your product can sometimes be unnecessarily complex, so break down the functionality into smaller standalone pieces that make sense, which can be combined together to build the full functionality (or whatever functionality you want the API to support). Going through the exercise of breaking down your main product into individual use cases is a powerful way to discover what kinds of functionality your developers might need to leverage.

In the agile world, use cases are like user stories. “As an X, I want to Y, so that I can Z.” A use case generally lists the steps a user should be able to follow to create a desired outcome.

5.2.1. Use case: mobile

One use case that’s important to consider is mobile (see figure 5.3). Mobile developers have needs that will have a huge impact on the behavior of your API.

Figure 5.3. In a mobile interface, the main goals include logging in and user authentication, a performant and efficient platform, and an exact and complete response. The illustration on the right demonstrates what an ideal mobile/platform interaction looks like.

For a mobile application, the use case will almost certainly include:

  • Logging in with the platform
  • Avoiding unexpected application crashes
  • Ensuring a quick response time

To support this kind of application, the platform must support the following developer needs:

  • Single call per mobile screen
  • Minimal data size
  • Ability to explicitly specify which sections of data are needed

Creating a system that supports mobile developers adds a lot of extra work on the API development side, but it’s likely unavoidable. Although it may be tempting to avoid this use case at the start, eventually your system will almost certainly need to support these users, and if your system hasn’t been designed with this use case in mind, you may find yourself in a situation where you can’t reasonably add this functionality. Here are the reasons for these requirements:

  • Mobile devices have limited bandwidth.
  • Mobile users frequently lose connectivity by walking into an elevator or driving into a tunnel (or hiking in the woods).
  • Most mobile devices don’t do parallel processing well, so a platform requiring several calls to obtain the information for a screen generates a poorly performing application.

If your API doesn’t support this use case because it’s hard to create, mobile developers will either work around the limitation or choose a different organization’s API that supports their needs more completely and clearly. When they work around your system by, for instance, creating a caching server of their own, or scraping your website to avoid interacting with the API, there’s a strong possibility that their system will get out of sync with your platform, creating a poor user experience. Although this seems like it’s the mobile developer’s problem, it’ll come back to you because users tend to look to the main platform owner as the owner of this kind of problem.

5.2.2. Use case: application integration

Whatever kind of system you have, one of the most valuable use cases it can provide for you and your customers—or partners—is the ability to integrate your data into the developer’s systems, and vice versa. Whether this is a shared login experience such as the ones provided by Twitter, GitHub, LinkedIn, and others, or a stream of activity that can be aggregated into a larger activity stream within their application, this kind of integration is an important use case to consider. If you don’t plan for this up front, it may be difficult to present this information in a way that’s easily used by your consumers. Remember, you should build all APIs as if they’re likely to become external APIs at some point. Even if they don’t, the consistency will improve the overall quality of your system, reducing your support burden at the same time.

Twitter example

The first company we’ll focus on is Twitter, because the functionality of this system is widely understood (see figure 5.4).

Figure 5.4. The interaction between a client and Twitter can include multiple types of interactions. The initial request and response is usually logging in, or client authentication. From that point, a client can either add new updates to the system or read the current feed of updates in the system. Each of these transactions works the same way: a request is sent from the client, and the server processes the request and sends a response back to the client.

The most basic use cases for Twitter are adding to and reading from the activity stream, which could be done with a standalone application or as an integration with another product such as an online magazine or separate social system. For this, an agile user story might be, “As a Twitter user, I want to be able to post updates and view my message stream so that I can share information and keep up with updates from the people I follow.” The use case could be expressed slightly differently. Viewing the action stream of the agile user story, a development-focused API use case could be the following:

  • Log in with the platform.
  • Post a message to the user’s stream.
  • Read the user’s message stream to display the new content in context.

To support this use case, the platform must make it easy and fast to perform these actions. The documentation should include sample code showing how to make these calls in order so that it’s easy and clear to developers. This sample code needs to be written by a developer or developer advocate and should include all potential client types: a native mobile application, a website application integrating with multiple facets of Twitter, or a website wanting to add the ability to share content on Twitter. Additionally, a narrative should be written by the technical team within the documentation describing the workflow for this interaction.

Twitter has also added new functionality based on use cases requested by its third-party developers. For instance, polling the API can be extremely expensive for a client and the server; making a request frequently is inefficient, and users don’t understand why their updates don’t appear immediately (see figure 5.5). To support this need, Twitter created streaming resources to which clients could “subscribe,” letting them know when an update has happened.

Figure 5.5. When waiting for changes from a platform, a client can continue asking the server whether there’s been any change, but this is expensive in terms of time and resources on the client and server side both. A subscription model, where the client “subscribes” to receive updates from the system, is much more efficient.

Twitter is also an example of a platform where a particular use case was initially supported, but usage of the system indicated that it wasn’t necessary. The API originally gave developers a choice between XML-formatted data and JSON-formatted data. Reviewing the usage of their API, the company discovered that fewer than 5% of clients were using the XML-formatted data, so it removed that option from the system and used its engineering resources for other, more popular features.

As a company, Twitter has generally been responsive to use cases requested by developers. Although it initially focused on one product without consideration for how it would be used to integrate with other systems, it quickly moved to a use case model to make it easy to target development and new features toward integration, as this became a large part of the reason for Twitter’s success. The hundreds of millions of users are definitely a compelling reason to use the API, but many of those users are participating in the system because of the large number of places they can leverage a connection to Twitter. As a company, it focuses quite heavily on developer experience, which makes it possible for them to support the vast number of application and integration clients out in the ecosystem.

Netflix example

Netflix is an example of an API that follows use cases well. Although the Netflix API was originally designed for third-party developers, it has evolved to be targeted to devices such as consoles, television sets, and DVD players. These devices tend to have a large amount of bandwidth available in order to support streaming of movies or TV shows, but the end users are fairly impatient, and the device developers want to be able to grab all the information they need to display the information quickly (see figure 5.6).

Figure 5.6. Common API interaction between Netflix and a client device or application

Here’s what this use case looks like:

  • Retrieve all information about a movie and related items in a single call
  • Respond quickly to requests for this information (some devices, such as mobile devices, do have bandwidth limitations)
  • Easily navigate to similar content

To support this, Netflix created a highly performant REST-based platform with the ability to expand similar information inline in a single call. A device can request information about a movie, including data about cast members, directors, genres, and similar movies. In addition, Netflix uses hypermedia within the calls to make it possible to programmatically find similar information dynamically within a call, so that subsequent calls can follow the chain to the related data a user wants to see. Adding information inline means that the payload is larger, but it’s the client’s decision as to whether to expand the information out to include more information or stick with the smaller, more basic default call.

Netflix has another specific challenge: many of the devices using the API are set in stone. Some TVs and DVD players, for instance, never update the firmware, so their use of the platform never changes. This has caused problems when Netflix wanted to add new functionality. In response, it encouraged devices to move to a system where the application itself was a shell into which the functionality could be injected. In this way, it became possible to add new functionality even when the system itself remained unchanged. Nevertheless, there are still old Blu-ray players out there that are chugging along using the old interface and will continue to do so on into the future.

When the API was designed, Netflix didn’t know which way the usage was going to guide it. It targeted several use cases including these devices. When it turned out that their business model led them largely to devices as clients, it was relatively simple to continue tuning the platform to that use case, and the company didn’t need to re-create the system from scratch.

5.3. Copy successful APIs

Once you understand how you want people using your APIs, copying from other existing APIs is often a good idea. Chapter 1 briefly described the API Commons, a project designed to facilitate the sharing of API schemas between companies, where a new API can choose to implement some or part of existing APIs. This project is evolving slowly, as many existing API providers feel as if they’ll be giving away their competitive advantage if they share the schema behind their API. This is an unfortunate choice by API producers, based on the unfounded fear of giving up a competitive advantage. The theory is that any information related to your company’s intellectual property and plans must be held close to the chest, lest other people get the jump on you. Remember that once your API is released, the schema can be determined by using the API itself. The advantage for your API should be in the quality of the data, the algorithms crafting the responses, and the integration with your presumably excellent product.

Using someone else’s API as a starting point is relatively easy even without a schema—a blueprint or definition of the API in a more easily understandable format—but starting from an API definition can help you see which of the resources in the collection will work for you. For instance, if you’re creating a new API based around fitness, you’ll want your API to include resources for weight, steps, and calories burned. If a well-known API like Fitbit has placed its schema in the Commons, adding those items to your schema so that they’re compatible for clients who want to implement both APIs would be simple. An API in the API Commons indicates that the organization owning that platform has explicitly invited others to use its API definition as a springboard for design. There’s a huge upside to placing your schema in the Commons—and little downside. All schemas in this system have a Creative Commons license, so you can leverage them without fear of legal ramifications.

When you’re creating a new API, you want to reduce the learning curve as much as possible for your users. Looking at schemas for existing APIs can help you create a system that can be integrated with other APIs quickly and painlessly. If your “User” model matches exactly with the schema of a complementary platform, a developer can tie those resources together with minimum effort. All fitness trackers keep track of the steps a user takes daily. If you’re creating a platform for a system like this, with many existing models out there, pick one of the most successful platforms and model your system after it.

When organizations share design resources in this way, it speeds the process of arriving at the best practices for a specific industry or API type. Without a system like the API Commons, each platform is designed in an echo chamber, without learning from the experience of other existing systems. Because there are so many existing APIs in the ecosystem already, new APIs need to excel at creating APIs that make sense in the context of other platforms already being used by developers. Remember that having a schema isn’t the end of the design phase—it’s the beginning of the next stage in the process. If your development isn’t guided by the schema you’ve created, the schema isn’t doing its job.

API Commons, as of this writing, is relatively new. This means that organizations choosing to share their schema models in this way are creating a de facto standard describing the best practices for creating APIs using the resources they define. By sharing their model, they can help create a world where their API is easy to integrate with similar APIs, without changing their schema themselves.

How does the Commons work? An API is designed using a schema modeling system, or an existing API with no existing schema model can use a modeling language to describe the API. For this purpose, any of the existing schema modeling languages is fine to use—Swagger, RAML, Blueprint, or any other format that may be created. These modeling languages are covered in depth in chapter 7, but for now, know that there are several formats to describe what an API will do before it has been developed.

Once the schema has been modeled, an API Commons manifest is created, following the format shown in figure 5.7.

Figure 5.7. Manifest format for API Commons

This relatively simple format allows the API owner to describe the schema that they used for their platform. As you can see, this document describes the API, an icon to use for the definition, and the format used for the API definition (or schema model). The resulting information is used to add the schema to the API Commons index, which is shown on the API Commons website (see figure 5.8).

Figure 5.8. API Commons top-level definitions

Although it’s not necessary to use the same definition language as the platform(s) you’re using as the basis for your API, doing so reduces the amount of work needed to create your own schema if you stick with the language used by the base platform. Chapter 7 discusses each of the schema modeling languages so that you can understand the pros and cons of each. For now, it’s sufficient to know that they all have the functionality needed to model an API, even if it’s quite complex.

The API Commons also supports versioning, so when your schema model is based on an API in the Commons, you can specify exactly which pieces you’re using from several different APIs. The definitions are stored in GitHub so there’s history available—which means versioning is possible.

Imagine you had a social event application and wanted to make an API so developers could create integrations with other systems and applications to improve your user experience. You could define this by choosing a People API from one API Commons definition (or schema), the Video API from YouTube, and the Events API from Google Calendar—each at the specific version you’re using. By doing this, you make it possible for developers to explore your API even more quickly. Although no tools exist for building API consoles or data explorers out of the data in the Commons, these tools will no doubt become available as more APIs are included.

All that said, remember the Flickr example from earlier: you want to carefully evaluate the APIs you’re considering using to make sure you feel they’ll support the best developer experience possible. There’s no reason you can’t look at all the models out there, decide they’re not going to work, and create your own—but the very exercise of looking through existing models should help quite a bit when creating your own API schema.

Exercise 2

Look through the API Commons at www.apicommons.org. Imagine that you are making a contact management system. Which of these APIs would you use to bootstrap your API model? Are they sufficient for your needs, or would you need to create a slightly different version—or another version entirely—for the system you’re making? What APIs would you like to see represented in the Commons? Reach out to the API providers you’d like to see there and ask them to participate.

5.4. REST is not always best

You’ve probably noticed that rather than always describe the web APIs I’m discussing as REST APIs, I tend to soften the description. REST-based is an excellent way of describing most successful web APIs. For various reasons, such as the mobile case described earlier, you’ll probably need to move away from the strict REST philosophy in order to best serve the needs of your users. Remember, the usability of your API is of paramount importance. Your users need an API that will meet their goals easily and efficiently, so follow the best practices listed here but remember the number one priority is usability.

The first guiding principle, Don’t surprise your users, seems to indicate that you shouldn’t move away from this philosophy, but you may remember that I’ve indicated, multiple times, that you shouldn’t move away from existing best practices unless you have a good reason. Anytime there’s a conflict between strict REST and making a use case easy, you should always err on the side of usability for all the use cases you’re working with. Again, though, don’t make the mistake of creating an API that’s awesome for a single use case but useless for any others.

I discussed one use case you’re likely to encounter that will make you move away from the model of a single resource per call: mobile. There are various ways to support the mobile use case, but none of them is particularly RESTful. That’s because mobile applications need to be able to make a single call per screen, even if that screen demonstrates multiple types of resources. But if you don’t implement something to enable your mobile developers to create a performant application, you’re going to create heartache for them, and for yourself.

5.4.1. Expand and include related resources

One way to approach this problem is to do what Netflix did: allow developers to specify whether they want to expand related resources inline with the call. Figure 5.9 shows the difference in calls when requesting a movie plus related information in a strict REST API versus an expandable resource API.

Figure 5.9. Strict REST versus API with expansion

In this case, the strict REST system requires that the client make four calls to show the movie with some related information. Because a mobile device isn’t likely to have parallel processes, that means each call must complete before the next call can be made. If the mobile phone user walks into an elevator or tunnel, causing a call to fail, that call must be made again until it succeeds. This results in a nonperformant application in one of the places where people are least patient (mobile users expect quick responses from requests on their phones). Even if the calls all succeed, the time needed to wait for all the responses will result in a poorly performing application.

With the expanded API, though, the call is made to the movie, requesting expansion for the three other items desired for the page. It’s a single call to the API instead of four separate calls, and though it might still fail, one retry on one call will still result in a more responsive experience for the mobile user than if all of the calls to the strict REST API had succeeded.

In addition to supporting expansion, Netflix uses hypermedia within results—which are covered in chapter 6 when we discuss business value and decisions, and in chapter 7 during the discussion about schema modeling—to help developers discover what related information can be expanded when making calls. Including this information makes it much easier for developers to understand the capabilities they have when using an API.

5.4.2. Create a query language

To solve the same problem, LinkedIn went with a slightly different approach. Its data structure is much more complicated than Netflix’s, so a simple expansion wasn’t likely to cover the needs of all its developers. Instead, LinkedIn created a query language to make it possible for developers to explicitly express what they wanted to see, what related information they wanted, and which fields they wanted within all the resources (figure 5.10).

Figure 5.10. Complex LinkedIn request: specific attributes for particular fields

The LinkedIn API was designed to encourage developers to explicitly specify exactly which fields they want in every single call, so the default representation of each resource was a minimal list with few fields included.

Here’s an example of what you could request with a single request to the API:

  • User’s information
  • User’s contacts

    • School attended

      • Name of school
      • City of school
    • Organizations
    • Shared contacts

      • Name

This query language made it possible to say that you wanted all of a user’s information, contacts, and the name and city of the last school each contact attended. The downside of this system was that the learning curve was somewhat steep. Because the resources aren’t RESTful—in order to get a reasonable representation of an object, the developer needs to specify more fields than the default—users of the API must learn how to determine which fields are available, how to access them, and what the restrictions are for use within the system. The LinkedIn API addressed this issue by spending time and resources creating example code and documentation describing how to make common requests. The consistency across the various endpoints made it easier to learn once a developer had made a single successful call to the system.

Unfortunately, due to business decisions, neither of these APIs is currently available for the public to use or test. But the general idea of creating query languages to allow client developers to express precisely what information they need, and in what format, is a good design model.

5.4.3. Create a comprehensive data transfer scheme

Freebase (recently deprecated) was a system not many people were familiar with. It was, in my opinion, the best web API out there. Freebase was a structured, user-editable, graph database. It included information from Wikipedia, MusicBrainz, and multiple other sources. It represented items as nodes (things) and edges (links). In this case, it wasn’t a REST API at all. All queries hit the same endpoint, and users created an object expressing exactly what they wanted to know, receiving back exactly that object. Because Freebase was a graph database, extremely complex queries took only 50 milliseconds to return.

Here’s an example of a Freebase Metabase Query Language (MGL) request. As you can see, it’s in JSON format, which was the language used by Freebase.

Listing 5.1. Freebase MGL: searching for Tom Cruise and Katie Holmes 
[{
  "type": "/film/actor",
  "ns0:type": "/film/producer",
  "/people/person/religion": "Scientology",
  "/people/person/height_meters<=": 2,
  "/people/person/spouse_s": [{
    "spouse": [{
      "name": null,
      "/people/person/religion": "Scientology",
      "type": "/film/actor"
    }]
  }]
}]

The request in listing 5.1 is asking for people who are film producers and film actors, and who are Scientologists less than 2 meters tall. Additionally, it wants to know about any Scientologists this person has married.

The response from the system (in the following listing) is a similar JSON object, containing the things that were requested.

Listing 5.2. Freebase response 
{
  "result": [
    {
      "/people/person/religion": "Scientology",
      "ns0:type": "/film/producer",
      "/people/person/spouse_s": [
        {
          "spouse": [
            {
              "name": "Katie Holmes",
              "/people/person/religion": "Scientology",
              "type": "/film/actor"
            },
            {
              "name": "Tom Cruise",
              "/people/person/religion": "Scientology",
              "type": "/film/actor"
            }
          ]
        },
...

The response had other groupings, but this is the one I was looking for, and the response was almost immediate. For most systems, this kind of complex query would be time consuming, but even if I had asked for the films they had in common, or the name of their daughter, the response would still be almost immediate.

Although a graph database may not be the right back end for you, this example should show you that there are different extremes in requests to web APIs. The main constraint you should follow is that it’s easy for your users to implement your use cases.

5.4.4. Create a separate batching system

When the developers at Etsy refactored their API, they wanted to stick with strict REST representations for their resources but understood that this wouldn’t work for third-party developers creating mobile applications. For this reason, they created a system called BeSpoke, which batches commands and uses a separate address (figure 5.11).

Figure 5.11. Etsy reworked its platform to be API First and strongly RESTful and decided to address the issue of mobile integration using the BeSpoke system, which batches calls together on the server end and sends back the aggregated information to the caller.

Etsy’s goal was to create endpoints that were combinations of calls to its APIs. The company made the decision that it wanted to own the specific structure of each of these endpoints. If a client developer needs a new batched endpoint, they must request help from Etsy to get it implemented.

The batched-up calls are run in parallel wherever possible (sometimes the response for one changes the request for the next) and those calls are bundled up into a single response that heads back to the client. In this way, a mobile application can ask for information about sellers, their activity list, and the current things they have up for sale.

This system meets both of the company’s requirements: it has a solid REST API, but it has built a separate system that allows its client developers to quickly request larger amounts of data as needed.

5.4.5. RESTful decisions

It should now be clear that although it’s important for your system to behave as predictably as possible, you need to make sure that your system is usable. If that can’t happen within the context of strict REST (or whatever other model you’re using), then you need to step to the side and find something that does work.

It’s vital in these cases that you communicate clearly to your customers exactly how your system does work so they can get up to speed as quickly as possible. If you use a separate system like BeSpoke, make sure that mobile developers can find their way to that information easily so they don’t try to implement their application using the bare REST API.

5.5. Focus on the developer experience

I’ve said many times that you need to focus on the developer and use cases during design, development, and documentation, but your API still won’t be successful if you don’t pay close attention to the developer experience once you’ve released it. From the developer portal to the support you provide when people are struggling, everything you do for your users after you release the API is as important as the work you did to create it.

Developer experience for an existing API covers several different realms. Reference documentation should be solid and consistent across all of your API endpoints. It should be easy to get a list of endpoints for a particular API without searching through documentation. One of the advantages of using a modeling language is that these systems are designed to help you create automatic reference documentation for your API.

Developer experience, including documentation, is the heart and soul of a successful API. Chapter 9 explores this topic in depth.

5.5.1. Communication and consistency are critical

Throughout the process outlined in the following chapters, I want you to think critically about which pieces of information you want to share with your users. It’s always best to assume you’ll be sharing everything. Sharing your business value with your customers helps them to better understand what kind of clients they should build that will help your API to remain successful—and that will help them stay successful as well, because they’re supporting your financial reason for having the API. Giving them the metrics you’re using helps too; adding this to the context of the business value means the users have a great idea of what looks like success for your API, and they’ll strive to help you with that, because it means they have a stronger position for their particular client or clients.

The use cases you create should be turned directly into tutorials for your client developers, to give them working code with an understandable story to follow. Sharing your schema model (even when the API is under construction) opens up the ability to have conversations with your customers about how the design meets their use cases. And they can use the blueprint to identify potential error cases they should handle and understand better how to implement their code most efficiently. Too often the client developers are seen as intruders into the system, and API providers try to hide away their business choices from those developers—but treating these developers as partners means telling them everything you can.

Whenever you make a decision, whether it’s in line with everyone else or a new path you’re blazing, it’s vital to communicate with your users. Too often developers are kept in the dark about important decisions being made by the platform organization, which leads to frustration and mistrust. When you treat your developer clients as partners, keeping them in the loop wherever possible, you’ll find that they’re forgiving, helpful, and as interested in your success as you’re interested in theirs.

It’s tempting to treat these people as competitors because they’re implementing functionality that may compete directly with your main product. But providing a different user experience for a subset of users is sometimes exactly what your user base needs to feel more excitement about the whole organization. If you’re willing to let developers use your system, then you should be willing to be transparent with them wherever possible.

Sometimes you can’t share the business reason for decisions you need to make. In those cases, it’s still important to tell your client developers that there has been a change and that it was made because of an underlying business decision. If you don’t tell them this, they’re likely to think the decision was arbitrary and can be changed, or will feel slighted by the decision. As a company, Google has always been transparent with the developers utilizing its APIs. It does get a lot of flak for the choices it makes, but developers using its system can be assured that it will give notice before the API is changed drastically or deprecated.

Relying on a platform for functionality within your application or website is a frightening prospect for anyone. Reaching out to your customers to keep them in the loop when changes are made is one of the best ways to build trust within the developer community. Remember, these developers are giving you their time and skills to integrate your system into theirs, and you need to treat them with respect.

These guiding principles are important to keep in mind while building your API, but the most critical thing in implementing your API is consistency. Your API needs to be consistent in terms of resource formatting, status codes, formats, and any other design choices you make. If a developer gets a user by drilling down through an organizational group, that user should be identical to the user when found via a search. Status codes for users should behave the same way as status codes for locations. If you choose to use an expansion or query method to allow for more complex queries, it needs to be implemented consistently throughout the API. Your documentation should also be consistent across the system. It’s likely that different teams will be building different parts of your API, so you may need a guiding team to help each group create resources and documentation that speak directly to the use cases and flow naturally from one section to another. Consistency can be a challenge, but as with everything else, when you have a choice between extra effort for the developers of the API and the consumers of the API, you should always make the experience as easy as possible for the users of the API.

5.5.2. Documentation should tell a story

When developers are initially exploring your platform, the questions they’re likely to ask are “What can I do with this?” and “How do I do something?” Unfortunately, most documentation for APIs focuses on “How does this work?” That’s a question that generally doesn’t come until after the developer is already engaged with the platform. The pretty example in figure 5.12 demonstrates what most API resource documentation looks like.

Figure 5.12. This diagram shows what reference documentation can look like in a system that has been designed by modeling a schema. The documentation you see here, which is dynamic and allows for exploration, is created entirely from an OpenAPI specification.

The problem with this type of documentation is that there’s no story or context for developers to use to understand why they should care about this API. Sometimes the thought is that if a developer finds her way to the API, she already knows what she wants to do with it, but it’s not a good idea to make that assumption for several reasons.

First, developers who are new to your system and find it through spelunking on the web may well not know what they want to do with the API. Making the assumption that they do know means that developers who aren’t already familiar with the data space and why they want to use it will likely give up and leave. This seems like it would pertain only to external or open APIs, but for developers starting out with your API, you need at least a “Getting Started” story, or they’re more likely to fumble around trying to figure out what to do.

Second, tutorials that read like stories tell your users a tale about how you expect them to use the platform. When developers can get started with an existing example of a flow through the system, they can more easily create applications that use the API in a way that’s supported. If you leave developers to try to find something that works without guiding them, they’re more likely to work around the functionality that you want them to use. It may seem like all you’re doing with use cases and story-based tutorials is teaching people how to do specific things with your platform, but that’s not true. Particularly when you provide some structure to the stories you’re providing, as the client developers work through them they gain a greater understanding of how your system works and are more likely to have their creativity sparked by the examples you’ve given.

Third, stories help users to better understand how the different parts of your system interact together so they can be successful more quickly. I’ve seen many cases of developers who cobble together a mashup between different resources to create a particular function, when that data is available directly from another resource. It may seem obvious to you how your data is structured, but without that context developers will do the best they can.

Fourth, and most important, developers love to get up and running quickly. Taking the time to provide them with documentation that helps them solve their problem effectively communicates to them that you value their time. It shouldn’t be an arduous chore to figure out how to combine your resources to create a meaningful application or integration. Don’t make them reinvent the wheel. Give them examples of successful stories they can use to inspire their own ideas. These developers are your partners, so treat them with the consideration they deserve, and they’ll reward you by creating awesome applications with your API.

It’s challenging to find examples of story-type documentation in the wild, but at the least many API providers have started providing a “Getting Started” story to help developers make the critical first call to become engaged in your API. As the creator of an API, challenge yourself and your documentation writers to create documentation that helps users make the first call in the minimum amount of time.

Here are some questions that your documentation should answer in a “Getting Started” guide:

  • Do you require authentication?
  • How does a developer quickly get the pieces he needs to make the authentication work?
  • What does an example call look like?
  • How can a developer make a call to the system using a common library or on her own?

Don’t assume that your developers already know anything about using your API. If you’re worried about scaring developers away by seeming condescending, don’t. They’re more likely to be grateful that you took the time to guide them to a successful call, and trust me, many of your users won’t be familiar with systems like yours and will be happy to see documentation that guides them through the process of making a call and teaches them what they can do with the API. There are many, many skilled developers out there who haven’t ever interacted with a web API before; this has been clearly demonstrated to me by the enormous number of people who attend my “REST Demystified” talk at conferences. Great developers are not all client developers . . . yet. Help them get there.

Once you’ve started making documentation that’s story-based, don’t stop there. Give some use cases to your developers to show what kinds of things are possible given the way your system works. Spark their creativity with ideas.

Remember that new users want to know what they can do with the system and how they’d do that. It should be as straightforward as possible to find this information in the front of your documentation. A new user doesn’t know where to look, so the most obvious call to action needs to be a button leading developers to the documentation they need to figure out the answers to these questions.

5.6. Summary

Even before you start creating an API, you need to get into the mind-set of the people who will be using your system. All of these guiding principles will help you to put yourself in the shoes of the people you want to succeed. Keep these principles in mind as you work through the process of creating your API, and the system you build will be more compelling, successful, and easy to use.

This chapter covered several guiding principles for creating a successful API:

  • Don’t surprise your users. Be mindful of the decisions you make and make sure to communicate your intent clearly and consistently. Arbitrary decisions made in a rush frequently come back to bite you when those decisions lead to developer confusion.
  • Focus on use cases. If you can’t describe what you want developers to do with your API, they won’t know what you’re expecting, and you won’t have any guiding vision to drive the development of the API.
  • Copy successful APIs. Stand on the shoulders of giants. There’s no shame in cribbing from a successful API to make the experience of the application developers that much more consistent and easy.
  • REST is not always best. Although the focus of this book is on REST APIs, it’s important to keep a critical eye on the development to make sure that idealism isn’t trumping usability.
  • Focus on the developer experience. Again, this is the focus of chapter 9, but it’s worth reiterating here that a great developer experience is the number one way to ensure success for your API.

The next chapter covers the process of determining business value, establishing metrics, and designing use cases for your API.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset