Model Serving

Because TFServing only has production-level support for TensorFlow, it does not have the desired flexibility which you would expect from a framework-agnostic inference service. It does, however, support REST; gRPC; GPU acceleration; mini-batching; and “lite” versions for serving on edge devices. Regardless of the underlying hardware, this support does not extend to streaming inputs nor built-in auto scaling ⁶. Furthermore, the ability to extend the inference graph—beyond a Fairness Indicator—to include more advanced ML insights isn’t supported in a first-class way. Despite providing basic serving and model analysis features for TensorFlow models, this inference solution does not satisfy your more advanced serving requirements.

Model Monitoring

TFServing supports traditional monitoring via its integration with Prometheus. This exposes both system information—such as CPU, memory, and networking—and TFServing specific metrics; unfortunately, there is very little documentation (see the best source found here). Also, there is no first-class integration with data visualization tools like Kibana or distributed tracing libraries like Jaeger. As such, TFServing does not provide the managed network observability capabilities you desire.

When it comes to advanced model serving insights, including model drift and explainability, some of them are available in TensorFlow 2.0. Furthermore, the vendor lock-in to a proprietary serving solution complicates the plugability of model insight components. Since the deployment strategy of TFServing uses Kubeflow’s infrastructure stack, it leverages a microservice approach. This allows TFServing deployments to be easily coupled with auxiliary ML components.

Model Updating

TFServing is quite advanced in that it enables canary, pinned, and even rollback deployment strategies ⁷. However, the strategies are limited to the manual labeling of existing model versions and do not include support for the introduction of in-flight model versions. So version promotion does not have a safe-rollout guarantee. Lastly, the strategies are embedded in the server and aren’t extensible for other deployment strategies that might exist outside of TFServing.

Summary

TFServing provides extremely performant and sophisticated out-of-the-box integration for TensorFlow models, but it falls short on enabling more advanced features like framework extensibility, advanced telemetry, and plugable deployment strategies. Seeing these requirements unsatisfied, we will now look at how Seldon Core attempts to fill these gaps.

Setting Up Seldon Core

Seldon Core 1.0 comes prepackaged with Kubeflow so it should already be available to you. The Seldon Core installation will create a Kubernetes Operator which will watch for SeldonDeployment resources that describe your inference graph. However, you can install a custom version of Seldon Core as per the installation instructions with:

helm install seldon-core-operator 
    --repo https://storage.googleapis.com/seldon-charts  
    --namespace default 
    --set istio.gateway=istio-system/seldon-gateway 
    --set istio.enabled=true

You must ensure that the namespace where your models will be served has an Istio gateway and an InferenceServing namespace label. An example label application would be:

kubectl label namespace kubeflow serving.kubeflow.org/inferenceservice=enabled

An example Istio gateway would be:

kind: Gateway
metadata:
  name: seldon-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP

Save the above in a file and apply it using kubectl.

Packaging your model

As mentioned before, to run a model with Seldon Core you can either package it using a pre-packaged model server⁹ or using a Language Wrapper.¹⁰

Creating a SeldonDeployment

After packaging your model you need to define an inference graph that connects a set of model components into a single inference system. Each of the model components can be one of the two options outlined in “Packaging your model”.

Some example graphs are shown in Figure 8-3.

Figure 8-3. Seldon graph examples

The example inference graphs (a) to (e) shown in Figure 8-3 are expanded on below:

• (a) A single model

• (b) Two models in sequence. The output of the first will be fed into the input of the second.

• (c) A model with input and output transformers. The input transformer will be called, then the model and the response will be transformed by the output transformer.

• (d) A router that will choose whether to send to model A or B.

• (e) A combiner that takes the response from model A and B and combines into a single response.¹¹

In addition, SeldonDeployment can specify methods for each component. When your SeldonDeployment is deployed, Seldon Core adds a service orchestrator to manage the request and response flow through your graph.

An example SeldonDeployment (for inference graph (a) in Figure 8-3 above) is shown here as an example of what a pre-packaged model server looks like.

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
 name: seldon-model
spec:
 name: test-deployment
 predictors:
 - componentSpecs:
   graph:
     name: classifier
     type: SKLEARN_SERVER
     modelUri: gs://seldon-models/sklearn/income/model
     children: []
   name: example
   replicas: 1

In the example you see that the SeldonDeployment has a list of predictors, each of which describes an inference graph. Each predictor has some core fields:

componentSpecs

A list of Kubernetes PodSpecs each of which will be created into a Kubernetes Deployment

graph

A representation of the inference graph containing the name of each component, its type, and the protocol it respects The name must match one container name from the componentSpecs section, unless it is a prepackaged model server (see subsequent examples)

Name

The name of the predictor

Replicas

The number of replicas to create for each Deployment in the predictor

Type

The detail on whether it is a prepackaged model server or a custom language wrapper model

modelUri

A url where the model binary or weight are stored, which would be relevant for the respective pre-packaged model server

The example for SeldonDeployment for (a) is shown again in the following, however using a custom language wrapper model this time:

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
 name: seldon-model
spec:
 name: test-deployment
 predictors:
 - componentSpecs:
   - spec:
       containers:
       - image: seldonio/mock_classifier_rest:1.3
         name: classifier
   graph:
     children: []
     endpoint:
       type: REST
     name: classifier
     type: MODEL
   name: example
   replicas: 1

In this example you have a small set of new sections:

Containers

This is your Kubernetes container definition where you are able to provide overrides to the details of your container, together with your docker image and tag

Endpoint

In this case you can specify if the endpoint of your model will be REST or GRPC

The definition of your inference graph is now complete. We will now discuss how to test your components individually or in unison on the cluster.

Python Client for Python Language Wrapped Models

When you create your Python model, such as a file called MyModel.py with the contents:

class MyModel:
    def __init__(self):
      pass
    def predict(*args, **kwargs):
      return ["hello, "world"]

You are able to test your model by running the microservice CLI that is provided by the Python module. Once you install the Python seldon-core module you will be able to run the model above with the following command:

> seldon-core-microservice MyModel REST --service-type MODEL
...
2020-03-23 16:59:17,366 - werkzeug:_log:122
- INFO: * Running on http://0.0.0.0:5000/
(Press CTRL+C to quit)

Now that your model microservice is running, you can send a request using curl:

> curl -X POST 
> 	-H 'Content-Type: application/json' 
> 	-d '{"data": { "ndarray": [[1,2,3,4]]}}' 
>     	http://localhost:5000/api/v1.0/predictions
{"data":{"names":[],"ndarray":["hello","world"]},"meta":{}}

We can see that the output of the model is returned through the API.¹²

Local testing with Docker

If you are building language models with other wrappers, you can run the containers you build through your local Docker client. A good tool for building Docker containers from source code is S2I. For this, you just have to run the Docker client with the following command:

docker run --rm --name mymodel -p 5000:5000 mymodel:0.1

This will run the model and export it on port 5000, so we can now send a request using CURL:

> curl -X POST 
> 	-H 'Content-Type: application/json' 
> 	-d '{"data": { "ndarray": [[1,2,3,4]]}}' 
>     	http://localhost:5000/api/v1.0/predictions

{"data":{"names":[],"ndarray":["hello","world"]},"meta":{}}

With this environment, you can rapidly prototype and effectively test, before serving your model in a live cluster.

Model Explainability

Model explainability algorithms seek to answer the question: “Why did my model make this prediction on this instance?” The answer can come in many shapes, i.e., the most important features contributing to the model’s prediction or the minimum change necessary to features to induce a different prediction.

Explainability algorithms are also distinguished by how much access to the underlying model they have. On one end of the spectrum there are “black box” algorithms that only have access to the model prediction endpoint and nothing else. In contrast, you have “white box” algorithms that have full access to the internal model architecture and allow for much greater insight (such as taking gradients). In the production scenario, however, the black-box case is much more prominent, so we will focus on that here.

Before discussing an example, we will describe the integration patterns that would arise from the use of black-box explanation algorithms. These algorithms typically work by generating a lot of similar looking instances to the one being explained and then send both batch and sequential requests to the model to “map out” a picture of the model’s decision-making process in the vicinity of the original instance. Thus, an explainer component will communicate with the underlying model, as the explanation is being computed. Figure 8-4 shows how this pattern is implemented. A model configured as a SeldonDeployment sits alongside an explainer component, which comes with its own endpoint. When the explainer endpoint is called internally the explainer communicates with the model to produce an explanation.

Figure 8-4. Seldon explainer component

To illustrate these techniques we will show a couple of worked examples.

Sentiment Prediction Model

One example is a sentiment prediction model that is trained on movie reviews. You can launch this with an associated Anchors explainer using the following SeldonDeployment:

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: movie
spec:
  name: movie
  annotations:
    seldon.io/rest-timeout: "10000"
  predictors:
  - graph:
      children: []
      implementation: SKLEARN_SERVER
      modelUri: gs://seldon-models/sklearn/moviesentiment
      name: classifier
    explainer:
      type: AnchorText
    name: default
    replicas: 1

Once deployed, this model can be queried via the Istio ingress as usual. You will send the simple review “This film has great actors” to the model.

curl -d '{"data": {"ndarray":["This film has great actors"]}}' 
   -X POST http://<istio-ingress>/seldon/seldon/movie/api/v1.0/predictions 
   -H "Content-Type: application/json"

You will see a response like:

{
  "data": {
    "names": ["t:0","t:1"],
    "ndarray": [[0.21266916924914636,0.7873308307508536]]
  },
  "meta": {}
}

The model is a classifier and it is predicting with 78% accuracy that this is a positive review, which is correct. We can now try to explain this request:

curl -d '{"data": {"ndarray":["This movie has great actors"]}}' 
   -X POST http://<istio-ingress>/seldon/seldon/movie/explainer/api/v1.0/explain 
   -H "Content-Type: application/json"

You should see a response like the following (curtailed without the examples section):

{
  "names": [
    "great"
  ],
  "precision": 1,
  "coverage": 0.5007,
  ...
  "instance": "This movie has great actors",
  "prediction": 1
  },
  "meta": {
    "name": "AnchorText"
  }
}

The key element in this example is that the explainer has identified the word “great” as being the reason the model predicted positive sentiment and suggests that this would occur 100% of the time for this model if a sentence contains the word “great” (reflected by the precision value).

US Census Income Predictor Model Example

Here is a second example, trained on the 1996 US Census data, which predicts whether a person will have high or low income.¹⁵ For this example, you also need to have an Alibi explainer sample the input dataset and identify categorical features. to allow the explainer to give more intuitive results. The details for configuring an Alibi explainer can be found in the Alibi documentation along with an in-depth review of the following data science example.

The SeldonDeployment resource is as follows:

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: income
spec:
  name: income
  annotations:
    seldon.io/rest-timeout: "100000"
  predictors:
  - graph:
      children: []
      implementation: SKLEARN_SERVER
      modelUri: gs://seldon-models/sklearn/income/model
      name: classifier
    explainer:
      type: AnchorTabular
      modelUri: gs://seldon-models/sklearn/income/explainer
    name: default
    replicas: 1

Once deployed, you can ask for a prediction:

curl -d '{"data": {"ndarray":[[39, 7, 1, 1, 1, 1, 4, 1, 2174, 0, 40, 9]]}}' 
   -X POST http://<istio-ingress>/seldon/seldon/income/api/v1.0/predictions 
   -H "Content-Type: application/json"

You should see a response like:

{
    "data": {
      "names":["t:0","t:1"],
      "ndarray":[[1.0,0.0]]
     },
     "meta":{}
 }

The model is predicting low income for this person. You can now get an explanation for this prediction:

curl -d '{"data": {"ndarray":[[39, 7, 1, 1, 1, 1, 4, 1, 2174, 0, 40, 9]]}}' 
   -X POST http://<istio-ingress>/seldon/seldon/income/explainer/api/v1.0/explain 
   -H "Content-Type: application/json"

You should see the following response, which we have shortened to not show all the examples returned:

{
  "names": [
    "Marital Status = Never-Married",
    "Occupation = Admin",
    "Relationship = Not-in-family"
  ],
  "precision": 0.9766081871345029,
  "coverage": 0.022,
  ...
}

The key takeaway is that this model will predict a low income classification 97% of the time if the input features just were "Marital Status = Never-Married", "Occupation = Admin", and "Relationship = Not-in-family". So these are the key features from the input that influenced the model.

Outlier and Drift Detection

ML models traditionally do not extrapolate well outside of the training data distribution and that impacts model drift. In order to trust and reliably act on model predictions, you must monitor the distribution of incoming requests via different types of detectors. Outlier detectors aim to flag individual instances that do not follow the original training distribution. An adversarial detector tries to spot and correct a carefully crafted attack with the intent to fool the model. Drift detectors check when the distribution of the incoming requests is diverging from a reference distribution, such as that of the training data. If data drift occurs, the model performance can deteriorate, and it should be retrained. The ML model predictions on instances flagged by any of the above detectors should be verified before being used in real-life applications. Detectors typically return an outlier score at the instance or even feature level. If the score is above a predefined threshold, the instance is flagged.

Outlier and Drift detection are usually done asynchronously to the actual prediction request. In SeldonCore you can activate payload logging and send the requests to an external service that will do the outlier and drift detection outside the main request/response flow. An example architecture is shown in Figure 8-5 where Seldon Core’s payload logger passes requests to components that process them asynchronously. The components that do the processing and alerting are managed via Knative Eventing, which is described in “Knative Eventing”. The use of Knative Eventing here is to provide late-binding event sources and event consumers, enabling asynchronous processing. The results can be passed onwards to alerting systems.

Figure 8-5. Data science monitoring of models with Seldon Core + Knative

Model Serving

Seldon Core clearly provides the functionality to extend an inference graph and support advanced ML insights in a first-class way. The architecture is also flexible enough to leverage other advanced ML insights outside of its managed offering. And Seldon Core is quite versatile, providing the expected serving flexibility because it is framework agnostic. It provides support for both REST and gRPC, and GPU acceleration. It also can interface with streaming inputs using Knative Eventing. However, because the SeldonDeployment is running as a bare Kubernetes Deployment, it does not provide GPU autoscaling, which we expect from hardware agnostic auto-scaling.

Model Monitoring

Seldon Core seems to satisfy all of your model monitoring needs. Seldon Core’s deployment strategy also uses Kubeflow’s infrastructure stack, so it also leverages a microservice approach. This is especially noticeable with Seldon Core’s Explainers and Detectors being represented as separate microservices, within a flexible inference graph. Secondly, Seldon Core makes monitoring first-class by enabling monitoring, logging, and tracing with its support of Prometheus and Zipkin.

Model Updating

Seldon Core is advanced, in that it supports a variety of deployment strategies, including canary, pinned, and even multi-armed-bandit. However, similar to TFServing, revision or version management isn’t managed in a first-class way. This, again, means that version promotion does not have a safe-rollout guarantee. Lastly, as you can see by the options available for graph inferencing, in Figure 8-3, Seldon Core provides complete flexibility in growing your inference graph to support more complex deployment strategies.

Summary

Seldon Core works to fill in the gaps by providing extensibility and sophisticated out-of-the-box support for complex inference graphs and model insight. But it falls short with regards to the auto-scaling of GPUs, its scale-to-zero capabilities, and revision management for safe model updating—features that are common to serverless applications. We will now explore how KFServing works to fill this gap, by adding some recent Kubernetes additions, provided by Knative, to enable serverless workflows for TFServing, Seldon Core, and many more serving solutions.

Setting Up KFServing

KFServing provides InferenceService, a serverless inference resource that describes your static graph, by providing a Kubernetes Custom Resource Definition (CRD) for serving ML models on arbitrary frameworks. KFServing comes prepackaged with Kubeflow, so it should already be available. The KFServing installation¹⁸ will create a Kubernetes Operator in the kubeflow namespace, which will watch for InferenceService resources.

kubectl label namespace 
my-namespace serving.kubeflow.org/inferenceservice=enabled

To check if the KFServing Controller is installed correctly, please run the following command:

kubectl get pods -n kubeflow | grep kfserving

Confirm that the controller running. There is also a detailed troubleshooting guide you can follow on this GitHub site.

Simplicity and Extensibility

KFServing was fashioned to be simple to use for day-one users and customizable for seasoned data scientists. This is enabled via the interface that KFServing designed.

Now we will take a look at three examples of InferenceService:

One for sklearn:

apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
  name: "sklearn-iris"
spec:
  default:
    predictor:
      sklearn:
        storageUri: "gs://kfserving-samples/models/sklearn/iris"

One for tensorflow:

apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
  name: "flowers-sample"
spec:
  default:
    predictor:
      tensorflow:
        storageUri: "gs://kfserving-samples/models/tensorflow/flowers"

And one for pytorch:

apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
  name: "pytorch-cifar10"
spec:
  default:
    predictor:
      pytorch:
        storageUri: "gs://kfserving-samples/models/pytorch/cifar10/"
        modelClassName: "Net"

Each of these will give you a serving instance—with a HTTP endpoint—that serves a model using a requested framework server type. In each of these examples, a storageUri points to a serialized asset. The interface is mostly consistent across different models. The differences are in the framework specifications, i.e., tensorflow and pytorch. These framework specifications are common enough in that they share information like storageUri and Kubernetes resources requests but are also extensible in that they can enable framework specific information like PyTorch’s ModelClassName.

Clearly this interface is simple enough to get started quite easily, but how extensible is it towards more complex deployment configurations and strategies? Now we will look at an example where we can exhibit some of the features that KFServing has to offer.

apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
  name: "my-model"
spec:
  default:
    predictor:
      # 90% of traffic is sent to this model
      tensorflow:
        storageUri: "gs://kfserving-samples/models/tensorflow/flowers"
        serviceAccount: default
        minReplicas: 2
        maxReplicas: 10
        resources:
          requests:
            cpu: 1
            gpu: 1
            memory: 8Gi
  canaryTrafficPercent: 10
  canary:
    predictor:
      # 10% of traffic is sent to this model
      tensorflow:
        storageUri: "gs://kfserving-samples/models/tensorflow/flowers-2"
        serviceAccount: default
        minReplicas: 1
        maxReplicas: 5
        resources:
          requests:
            cpu: 1
            gpu: 1
            memory: 8Gi

The first extension is the ServiceAccount, which is used for authentication in the form of managed identities. If you wish to authenticate to S3 because your S3 should not be public, you need an identity attached to your InferenceService that validates you as a user. KFServing allows you to pass an identity mounted on the container and wires up the credentials through the ServiceAccount in a managed way. For example, say you are trying to access a model that may be stored on Minio. You would use your Minio identity information to create a secret, beforehand, and then attach it to the service account. If you recall, we created a secret in “Minio”, we just need to include KFServing related annotations:

apiVersion: v1
data:
 awsAccessKeyID: xxxx
 awsSecretAccessKey: xxxxxxxxx
kind: Secret
metadata:
 annotations:
   serving.kubeflow.org/s3-endpoint: minio-service.kubeflow.svc.cluster.local:9000
   serving.kubeflow.org/s3-verifyssl: "0"
   serving.kubeflow.org/s3-usehttps: "0"
   serving.kubeflow.org/s3-region: us-east-1
 name: minioaccess
 namespace: my-namespace

And attach it to the service account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: my-namespace
secrets:
- name: default-token-rand6
- name: minioaccess

The second extension to notice is the min and max replicas. You would use these to control provisioning to allow you to meet demand, by not dropping requests, while not over-allocating.

The third extension is resource requests, which have preset defaults that you will almost always need to customize for your model. As you can see, this interface enables the use of hardware accelerators, like GPUs.

The last extension showcases the mechanism that KFServing uses to enable canary deployments. This deployment strategy assumes that you only want to focus on a two-way traffic split, as opposed to an n-way traffic split. In order to customize your deployment strategy, do the following:

• If you use just the default, like in your initial template, you get a standard blue-green deployment that comes with a Kubernetes Deployment resource.

• If you include a canary, with canaryTrafficPercent == 0, you get a pinned deployment where you have an addressable default and canary endpoint. This is useful if you wish to send experimental traffic to your new endpoint, while keeping your production traffic pointed to your old endpoint.

• If you include canary, with canaryTrafficPercent > 0, you get a canary deployment that enables you to slowly increment traffic to your canary deployment, in a transparent way. In the above example, you are experimenting with flowers-2, and as you slowly increment this canaryTrafficPercentage you can gain confidence that your new model will not break your current users.¹⁹ Eventually, you would go to 100, thereby flipping the canary and default, and you should then delete your old version afterward.

Now that we understand some of the powerful abstractions that KFServing offers, let’s use KFServing to host your Product Recommender example.

Recommender Example

We will now put your Product Recommender example, from “Building a Recommender with TensorFlow”, behind an InferenceService.

First you will define your InferenceService with the following 11 lines of yaml:

apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
 name: "recommender"
 namespace: my-namespace
spec:
 default:
   predictor:
     tensorflow:
       serviceAccount: default
       storageUri: "s3://models/recommender"

After running kubectl apply and waiting until your InferenceService is Ready, you should see:

$ kubectl get inferenceservices -n my-namespace
NAME          URL                                                                 READY   DEFAULT TRAFFIC   CANARY TRAFFIC   AGE
recommender   http://recommender.my-namespace.example.com/v1/models/recommender   True    100                                59s

You can then curl your InferenceService with the following:

kubectl port-forward --namespace istio-system 
 $(kubectl get pod --namespace istio-system --selector="app=istio-ingressgateway" 
 --output jsonpath='{.items[0].metadata.name}') 
 8080:80

curl -v -H "Host: recommender.my-namespace.example.com" 
http://localhost:8080/v1/models/recommender:predict -d 
'{"signature_name":"serving_default","inputs": {"products": [[1],[2]],"users" : [[25], [3]]}}'

As an alternative to curl, you may also use the KFServing PythonSDK to send requests in Python.²⁰.

In addition to an HTTP endpoint, this simple interface also provides all the serverless features that come with Kubeflow’s Stack + Knative, including, but not limited to:

• Scale-to-zero

• GPU Autoscaling

• Revision Management (Safe Rollouts)

• Optimized Containers

• Network Policy and Authentication

• Tracing

• Metrics

As such, with only a few lines of yaml, KFServing provides production ML features, while also allowing Data Scientists to scale their deployments into the future. But how does KFServing enable these features in such an abstracted way?

We will now look at KFServing’s underlying infrastructure stack and see how it promotes serverless, how its layers can be further customized, and what additional features exist.

Going layer by layer

Hardware that runs your compute cluster is the base-building block for all the layers above. Your cluster could run a variety of hardware devices including: CPUs, GPUs, or even TPUs. It is the responsibility of the layers above to simplify the toggling of hardware types and to abstract as much complexity as possible.

Kubernetes is the critical layer, right above the compute cluster, that manages, orchestrates, and deploys a variety of resources—successfully abstracting the underlying hardware. The main resources we will focus on are Deployments, Horizontal Pod Autoscalers (HPA), and Ingresses. And since Kubernetes abstracts the underlying hardware, upon which deployments are run, this enables you to use hardware optimizers like GPUs within the upper levels of the stack.

Istio has been alluded to through this book, but we will talk about a few of its features that are particularly relevant to KFServing. Istio is an open source service mesh that layers transparently onto the Kubernetes cluster. It integrates into any logging platform, or telemetry or policy system and promotes a uniform way to secure, connect, and monitor microservices. But what is a service mesh? Traditionally, each service instance is co-located with a sidecar network proxy. All network traffic (HTTP, REST, gRPC, etc.) from an individual service instance flows via its local sidecar proxy to the appropriate destination. Thus, the service instance is not aware of the network at large and only knows about its local proxy. In effect, the distributed system network has been abstracted away from the service programmer. Primarily, Istio expands upon Kubernetes Resources, like Ingresses, to provides service mesh fundamentals like:

• Authentication/Access control

• Ingress and Egress policy management

• Distributed Tracing

• Federation via multi-cluster ingress and routing

• Intelligent traffic management

These tools are all critical for production inference applications that require administration, security, and monitoring.

The last component of the KFServing infrastructure stack is Knative, which takes advantage of the abstractions that Istio provides. The KFServing project primarily borrows from Knative Serving and Eventing, the latter which will be expanded upon in “Knative Eventing”. As we described in “Knative”, Knative Serving builds on Kubernetes and Istio to support deploying and serving serverless applications. By building atop Kubernetes resources, like Deployments and HPA`s, and Istio resources, like `virtual services, Knative Serving provides:

• An abstracted service mesh

• CPU/GPU autoscaling: either QPS (Queries per second) or metric based

• Revision management for safe rollouts and canary/pinned deployment strategies

These offerings are desirable for data scientists who want to limit their focus and energy to model development, and have scaling and versioning be handled for them in a managed way.

Escape Hatches

KFServing’s extensibility features escape hatches to the underlying layers of its stack. By building escape hatches into the InferenceService CRD, data scientists can further tune their production inference offering for security at the Istio level, and performance at the Knative level.

We will now walk through one example of how you can leverage these escape hatches, by tuning the autoscaling of your InferenceService.

To understand how to use this escape hatch, you need to understand how Knative enables autoscaling. There is a proxy in the Knative Serving Pods called the (queue-proxy) that is responsible for enforcing request queue parameters (concurrency limits), and reporting concurrent client metrics to the autoscaler. The autoscaler, in turn, reacts to these metrics by bringing pods up and down. Every second, the queue proxy publishes the observed number of concurrent requests in that time period. KFServing by default sets the target concurrency (average number of in-flight requests per pod) to be one. If we are to load the service with five concurrent requests, the autoscaler would try to scale up to five pods. You can customize the target concurrency by adding the example annotation autoscaling.knative.dev/target.

Here is the example Recommender InferenceService from earlier:

apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
 name: "recommender"
 namespace: my-namespace
spec:
 default:
   predictor:
     tensorflow:
       serviceAccount: default
       storageUri: "s3://models/recommender"

If you test this service by sending traffic in 30-second spurts while maintaining five in-flight requests, you will see that the autoscaler scales up your inference services to five pods.²¹.

By applying the annotation autoscaling.knative.dev/target:

apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
 name: "recommender"
 namespace: my-namespace
 annotations:
   autoscaling.knative.dev/target: "5"
spec:
 default:
   predictor:
     tensorflow:
       serviceAccount: default
       storageUri: "s3://models/recommender"

This will set the target concurrency to five. Which means, that if you load the service with five concurrent requests, you will see that you only need one pod for your inference service.

Debugging an InferenceService

With a fully abstracted interface, InferenceService enables many features while giving minimal exposure to the complexity under the hood. To properly debug your InferenceService, let’s look at the request flow upon hitting your InferenceService.

The request flow when hitting your inference service, illustrated in Figure 8-8, is as follows:

1. Traffic arrives through the Istio Ingress Gateway when traffic is external and through the Istio Cluster Local Gateway when traffic is internal.

2. KFServing creates an Istio VirtualService to specify its top-level routing rules for all of its components. As such traffic routes to that top-level VirtualService from the Gateway.

3. Knative creates an Istio Virtual service to configure the gateway to route the user traffic to the desired revision. Upon opening up the destination rules, you will see that the destination is a Kubernetes Service for the latest ready Knative Revision.

4. Once the revision pods are ready, the Kubernetes Service will send the request to the queue-proxy.

   • If the queue-proxy has more requests than it can handle, based on the concurrency of the kfserving container, then the autoscaler will create more pods to handle the additional requests.

5. Lastly, the queue-proxy will send traffic to the kfserving controller.

Figure 8-8. KFSeving request flow

Where does this come in handy? Well say you create your InferenceService but the Ready status is false:

kubectl get inferenceservice -n my-namespace recommender
NAME          URL   READY   DEFAULT TRAFFIC   CANARY TRAFFIC   AGE
recommender         False                                      3s

You can step through the resources that are created in the request flow and view each of their status objects to understand what is the blocker.²².

Debugging Performance

What if you deployed your InferenceService but its performance does not meet your expectations? KFServing provides various dashboards and tools to help investigate such issues. Using Knative, KFServing has many resources in its “debugging performance issues” guide. You can also follow this Knative guide to access Prometheus and Grafana. Lastly, you can also use request tracing, also known as distributed tracing, to see how much time is spent in each step of KFServing’s request flow in Figure 8-8. You can use this Knative guide to access request traces.

Knative Eventing

By bringing in Knative into its stack, KFServing enabled serverless via Knative Serving and the use of event-sources and event-consumers via Knative Eventing.²³

We will take a look at how Knative Eventing works, and how you can extend you inference service with an event source.

Knative Eventing enforces a lambda-style architecture of event-sources and event-consumers with the following design principles:

• Knative Eventing services are loosely coupled.

• Event producers and event consumers are independent. Any producer or Source, can generate events before there are active event consumers listening. Any event consumer can express interest in an event before there are producers that are creating those events.

• Other services can be connected to any Eventing system that:

  • Creates new applications without modifying the event producer or event-consumer.

  • Selects and targets specific subsets of events from their producers.

Knative Eventing delivers events in two flavors: direct delivery from a Source to a single service and fan-out delivery from a Source to multiple endpoints using Channels and Subscriptions.

There are a variety of Sources²⁴ that come out of the box when installing Knative Eventing, one of which includes the KafkaSource.²⁵ We will take a look at an example specification for how you would use a KafkaSource to send events, received by Kafka, to your Recommender example:

apiVersion: sources.knative.dev/v1alpha1
kind: KafkaSource
metadata:
  name: kafka-source
spec:
  consumerGroup: knative-group
  # Broker URL. Replace this with the URLs for your kafka cluster,
  # which is in the format of my-cluster-kafka-bootstrap.my-kafka-namespace:9092.
  bootstrapServers: my-cluster-kafka-bootstrap.my-kafka-namespace:9092.
  topics: recommender
  sink:
    ref:
      apiVersion: serving.kubeflow.org/v1alpha2
      kind: InferenceService
      name: recommender

As you can see by the simplicity of this specification, after setting up your Kafka resources, hooking up Kafka into your InferenceService is as simple as 13 lines of yaml. You can also find a more advanced end-to-end example with Minio and Kafka this GitHub site.

Additional Features

KFServing contains a host of features that are continuously being improved. A comprehensive list of its capabilities can be found here.

API Documentation

KFServing Kubernetes APIs

KFServing Python APIs

Model Serving

KFServing makes graph inference and advanced ML insights first-class while also defining a Data Plane that is extremely extensible for pluggable components. This flexibility allows data scientists to focus on ML insights without having to strain over how to include it in the graph.

KFServing is not only versatile in that it provides serving flexibility for a variety of frameworks, but it also standardizes the Data Plane across differing frameworks to reduce complexity in switching between model servers. It codifies the Kubernetes design pattern by moving common functionalities like request-batching, logging, and pipe-lining into a sidecar. This, in turn, slims down the model server and creates a “separation of concerns,” as model services without these features can immediately benefit from deploying onto KFServing. It also provides support for both REST, gRPC, and GPU acceleration and can interface with streaming inputs using Knative Eventing. And lastly, thanks to Knative Serving, KFServing provides GPU autoscaling, which you expect from a hardware agnostic auto-scaling.

Model Monitoring

By taking from Seldon Core and its infrastructure stack, KFServing meets all of your model monitoring needs. KFServing leverages the sophisticated model explainers and drift detectors of Seldon Core in a first-class way, while also paving a way for developers to define their own monitoring components in a highly flexible yet powerful Data Plane.

Furthermore, with all the networking capabilities enabled by having Istio and Knative in its infrastructure stack, KFServing provides extensible network monitoring and telemetry with support for Prometheus, Grafana, Kibana, Zipkin, and Jaeger, to name a few. These all satisfy your needs to monitor for Kubernetes metrics, i.e., memory/CPU container limits, and server metrics, i.e., qps and distributed tracing.

Model Updating

KFServing’s use of Knative was strategic in providing sophisticated model updating features. As such, KFServing satisfies all of your requirements regarding deployment strategies and version rollouts.

By leveraging Istio’s virtual services and the simplicity of an abstracted CRD, KFServing makes the toggling of deployment strategies simple. It makes the flow from blue-green → pinned → canary as simple as changing a few lines of yaml. Furthermore, with the diverse and ever-expanding feature of its underlying stack, KFServing is easily extensible to support more complicated deployment strategies like Multi-Armed Bandit.²⁶

By using Knative Serving, KFServing adopts revision management that makes Kubernetes deployment immutable. This ensures safe rollout by health checking the new revisions pods before moving over the traffic. A revision enables:

• Automated and safe-roll rollouts

• Bookkeeping for all revisions previously created

• Rollbacks to known, good configurations

This sufficiently satisfies your versioning requirements for models in development, in flight, and in production.

Summary

KFServing has developed a sophisticated inference solution that abstract its complexity from day-one users while also enabling power users to take advantage of its diverse feature set. Building cloud-native, KFServing seamlessly sits atop Kubeflow and finalizes the MDLC life cycle with its inference solution.