Preface

In this book, we introduce some patterns that should help with serving machine learning models, along with some popular model serving tools. A lot of machine learning projects do not see the light of day due to the technical gap between model creation and model deployment. This book will help machine learning engineers and scientists to get a good understanding of model serving. We have used simple examples and models throughout the book to demonstrate the serving mechanisms instead of building complicated models. This will help you focus on the serving patterns and using the serving tools.

Who this book is for

This book is for data scientists, machine learning engineers, and anyone who wants to work on serving machine learning models.

What this book covers

Chapter 1, Introducing Model Serving, introduces model serving and why model serving is important to the success of data science and machine learning projects.

Chapter 2, Introducing Model Serving Patterns, describes how patterns in model serving can be of great help to easily identify the best serving approach for a particular problem following the best practices. We also introduce you to different types of serving patterns.

Chapter 3, Stateless Model Serving, discusses how stateless model serving can help improve customer experiences, and the advantages of stateless serving in resilient and scalable model serving.

Chapter 4, Continuous Model Evaluation, introduces you to continuous model evaluation after serving and why it is important. We also discuss some techniques to evaluate the model continuously.

Chapter 5, Keyed Prediction, introduces you to keyed prediction patterns and discusses how passing keys can be beneficial during returning inference to the clients. We also discuss some ideas to generate keys.

Chapter 6, Batch Model Serving, discusses batch and offline model serving and how the inference can be updated during batch serving. We also discuss different techniques for updating the model periodically in batch serving.

Chapter 7, Online Learning Model Serving, discusses how can we serve models where real-time inferences are needed and some of the techniques and challenges in online serving.

Chapter 8, Two-Phase Model Serving, discusses serving two models in parallel, where one model is strong and the other model is weak. This chapter also discusses the necessity of two-phase serving and some ideas and challenges related to it.

Chapter 9, Pipeline Pattern Model Serving, introduces how models can be served using pipelines using directed acyclic graphs.

Chapter 10, Ensemble Model Serving Pattern, introduces the idea of combining multiple models in serving. It also shows how we can ensemble models in different ways and how the response given to the client is sent as a combined outcome from multiple models.

Chapter 11, Business Logic Pattern, discusses how different business logics are used along with inference codes to serve models.

Chapter 12, Exploring TensorFlow Serving, gives a high level introduction to using TensorFlow Serving tool to serve a model.

Chapter 13, Using Ray Serve, introduces the Ray Serve tool for serving machine learning models with of how to use the tool for serving model following few patterns we have discussed.

Chapter 14, Using BentoML, introduces the BentoML tool for serving models, with examples of using BentoML in ensemble pattern and business logic pattern.

Chapter 15, Serving ML Models using a Fully Managed AWS Sagemaker Cloud Solution, discusses how we can serve models using fully managed cloud solution. We use Amazon SageMaker to show you at the high-level how you can serve models using the built-in services provided by a fully managed cloud solution.

To get the most out of this book

This book does not demonstrate creation of the best performing ML models. We only intend to introduce the readers with different model serving patterns and tools and to demonstrate the use of these patterns and tools we have used very basic models. We assume the readers are already familiar with machine learning and know how to create their models. To get best out of the book try touse the patterns and run the examples provided thruoughout the book and use your own models to serve following the patterns discussed.

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Machine-Learning-Model-Serving-Patterns-and-Best-Practices. If there’s an update to the code, it will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “ In this chapter, we will use the TensorFlow library.”

A block of code is set as follows:

train_images = train_images.astype(np.float32) / 255.0
test_images = test_images.astype(np.float32) / 255.0

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

[[1.5 6.5 7.5 8.5]
[1.5 6.5 7.5 8.5]
[1.5 6.5 7.5 8.5]
[1.5 6.5 7.5 8.5]]
240

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: “Select System info from the Administration panel.”

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [email protected] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you’ve read Machine Learning Model Serving Patterns and Best Practices, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere? Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application. 

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

1. Scan the QR code or visit the link below

https://packt.link/free-ebook/9781803249902

2. Submit your proof of purchase
3. That’s it! We’ll send your free PDF and other benefits to your email directly