Programmatic Query

The following functionality is available for programmatic query.

First, we list all the models in the workspace:

pandas.DataFrame.from_dict(ws1.list(metadata.Model.ARTIFACT_TYPE_NAME))

In our code we created only a single model, which is returned as a result of this query (see Figure 6-1).

Figure 6-1. List of models

Next, we get basic lineage. In our case we created a single model, so, the returned lineage will contain only ID of this model.

print("model id is " + model.id) 

Then we find the execution that produces this model. In our toy application we created a single execution. An ID of this execution is returned as a result of this query.

output_events = ws1.client.list_events2(model.id).events
execution_id = output_events[0].execution_id
print(execution_id) 

Finally, we find all events related to that execution:

all_events = ws1.client.list_events(execution_id).events
assert len(all_events) == 3
print("
All events related to this model:")
pandas.DataFrame.from_dict([e.to_dict() for e in all_events])

In our case we used a single input which was used to create a model and metrics. As a result, the result of this query looks as shown in Figure 6-2.

Figure 6-2. Query result as table

Kubeflow Metadata UI

In addition to providing APIs for writing code to analyze metadata, Kubeflow Metadata tool provides a UI, which allows you to view metadata without writing code. Access to Metadata UI is done through the main Kubeflow UI, as seen in Figure 6-3.

Figure 6-3. Accessing metadata UI

Once you click on Artifact Store you should see the list of available artifacts (logged metadata events), as in Figure 6-4.

Figure 6-4. List of artifacts in Artifact Store UI

From this view we can click on the individual artifact and see its details, as shown in Figure 6-5.

Figure 6-5. Artifact view

Kubeflow Metadata provides some basic capabilities for storing and viewing of machine learning metadata, its capabilities however are extremely limited, especially in terms of viewing and manipulating stored metadata. A more powerful implementation of machine learning metadata management is implemented by MLflow. Although mlflow is not part of Kubeflow distribution, it’s very easy to deploy it alongside Kubeflow and use it from Kubeflow based applications, as described below.

Creating and Deploying an MLflow Tracking Server

MLflow tracking server allows you to record MLflow runs to local files, to a SQLAlchemy compatible database, or remotely to a tracking server. In our implementation we are using a remote server.

An MLflow tracking server has two components for storage: a backend store and an artifact store.The backend store is where MLflow Tracking Server stores experiment and run metadata as well as parameters, metrics, and tags for runs. MLflow supports two types of backend stores: file store and database-backed store. For simplicity we will be using a file store. In our deployment, this file store is part of the docker image, which means that this data is lost in the case of server restart. If you need longer term storage, you can either use an external file system, like NFS server, or a database. The artifact store is a location suitable for large data (such as an S3 bucket or shared NFS file system) and is where clients log their artifact output (for example, models). To make our deployment cloud independent, we decided to use Minio (part of Kubeflow) as an artifact store. Based on these decisions, a docker file for building of the MLflow tracking server look like follows (similar to implementation here).

FROM python:3.7

RUN pip3 install --upgrade pip && 
   pip3 install mlflow --upgrade && 
   pip3 install awscli --upgrade  && 
   pip3 install boto3 --upgrade

ENV PORT 5000
ENV AWS_BUCKET bucket
ENV AWS_ACCESS_KEY_ID aws_id
ENV AWS_SECRET_ACCESS_KEY aws_key
ENV FILE_DIR /tmp/mlflow

RUN mkdir -p /opt/mlflow
COPY run.sh /opt/mlflow
RUN chmod -R 777 /opt/mlflow/

ENTRYPOINT ["/opt/mlflow/run.sh"]

Here we first load MLflow code (using PIP), set environment variables, and then copy and run the startup script. The startup script used here looks like Example 6-10.⁵

#!/bin/sh
mkdir -p $FILE_DIR

mlflow server 
   --backend-store-uri file://$FILE_DIR 
   --default-artifact-root s3://$AWS_BUCKET/mlflow/artifacts 
   --host 0.0.0.0 
   --port $PORT

This script sets an environment and then verifies that all required environment variables are set. Once validation succeeds, an MLflow server is started. Once the docker is created the following Helm command (the Helm chart is located on GitHub.) can be used to install the server

helm install <location of the Helm chart>

This Helm chart installs three main components implementing the MLflow tracking server:

Deployment

Deploying MLflow server itself (single replica). The important parameters here are the environment, including minio endpoint, credentials, and bucket used for artifact storage.

Service

Creating a kubernetes service exposing MLflow deployment.

Virtual service

Exposing MLflow service to users through the Istio ingress gateway used by Kubeflow.

Once the server is deployed, we can get access to the UI, but at this point it will say that there are no available experiments. Let’s now look at how this server can be used to capture metadata.⁶

Logging Data on Runs

As an example of logging data, let’s look at simple code below.⁷ We will start by installing required packages:

!pip install pandas --upgrade --user
!pip install mlflow --upgrade --user 
!pip install joblib --upgrade --user
!pip install numpy --upgrade --user
!pip install scipy --upgrade --user
!pip install scikit-learn --upgrade --user
!pip install boto3 --upgrade --user 

import time
import json
import os
from joblib import Parallel, delayed

import pandas as pd
import numpy as np
import scipy

from sklearn.model_selection import train_test_split, KFold
from sklearn.metrics
        import mean_squared_error, mean_absolute_error, r2_score, explained_variance_score
from sklearn.exceptions import ConvergenceWarning

import mlflow
import mlflow.sklearn
from  mlflow.tracking import MlflowClient

Here again, os and the last three imports are required for mlflow logging, while the rest are used for machine learning. Now we need to define environment variables required for proper access to Minio server for storing artifacts:

os.environ['MLFLOW_S3_ENDPOINT_URL'] = 'http://minio-service.kubeflow.svc.cluster.local:9000'
os.environ['AWS_ACCESS_KEY_ID'] = 'minio'
os.environ['AWS_SECRET_ACCESS_KEY'] = 'minio123'

Note here that in addition to the tracking server itself MLFLOW_S3_ENDPOINT_URL is defined not only in the tracking server definition, but also in the code, that actually captures the metadata. This is because, as we mentioned above, user code writes to the artifact store directly, bypassing the server.

Here we skip the majority of the code (the full code can be found at GitHub) and concentrate only on the parts related to the MLflow logging. The next step is connecting to the tracking server and creating an experiment:

remote_server_uri = "http://mlflowserver.kubeflow.svc.cluster.local:5000"
mlflow.set_tracking_uri(remote_server_uri)
experiment_name = "electricityconsumption-forecast"
mlflow.set_experiment(experiment_name)

Once connected to the server and creating (choosing) an experiment, we can start logging data. As an example let’s look at the code for storing KNN regressor information.

def train_knnmodel(parameters, inputs, tags, log = False):
    with mlflow.start_run(nested = True):

……………………………………………….
        # Build the model
        tic = time.time()
        model = KNeighborsRegressor(parameters["nbr_neighbors"],
                                weights = parameters["weight_method"])
        model.fit(array_inputs_train, array_output_train)
        duration_training = time.time() - tic

        # Make the prediction
        tic1 = time.time()
        prediction = model.predict(array_inputs_test)
        duration_prediction = time.time() - tic1

        # Evaluate the model prediction
        metrics = evaluation_model(array_output_test, prediction)

        # Log in mlflow (parameter)
        mlflow.log_params(parameters)

        # Log in mlflow (metrics)
        metrics["duration_training"] = duration_training
        metrics["duration_prediction"] = duration_prediction
        mlflow.log_metrics(metrics)

        # log in mlflow (model)
        mlflow.sklearn.log_model(model, f"model")

        # Save model
        #mlflow.sklearn.save_model(model, f"mlruns/1/{uri}/artifacts/model/sklearnmodel")

        # Tag the model
        mlflow.set_tags(tags)

In this code snippet, we can see how different kinds of data about model creation and prediction test statistics is logged. The information here is very similar to the information captured by Kubeflow Metadata and includes inputs, models, and metrics.

Finally, similar to Kubeflow Metadata, MLflow allows you to access this metadata programmatically. The main APIs provided by MLflow include:

df_runs = mlflow.search_runs(experiment_ids="0") 
print("Number of runs done : ", len(df_runs))

df_runs.sort_values(["metrics.rmse"], ascending = True, inplace = True) 
df_runs.head()

MLFlow will sort runs by Root Mean Square Error (rmse) and show the best ones.

For additional capabilities of the programmatic runs querying consult the MLflow documentation.

With all the capabilities of running programmatic queries, the most powerful way to evaluate runs’ metadata is through MLflow UI, which we will cover next.

Using MLflow UI

The Tracking UI in MLflow lets you visualize, search and compare runs, as well as download run artifacts or metadata for analysis in other tools. Because MLflow is not part of the Kubeflow its access is not provided by Kubeflow UI. Based on the provided Virtual service, MLflow UI is available at <Kubeflow Istio ingress gateway URL>/mlflow[].

Figure 6-7 shows the results produced by the run described.

Figure 6-7. MLflow main page

It is possible to filter results using the search box. For example, if we want to see only results for the KNN model, then the search criteria tags.model="knn” can be used. You can also use more complex filters, such as tags.model="knn” and metrics.duration_prediction < 0.002, which will return results for the KNN model for which prediction duration is less than 0.002 sec. By clicking on the individual run we can see its details, as shown in Figure 6-8.

Figure 6-8. View of the individual run

Alternatively, we can compare several runs by picking several runs and click compare, as seen in Figure 6-9.

Figure 6-9. Run comparison view

We can also view metrics comparison for multiple runs, as seen in Figure 6-10.⁸

Figure 6-10. Run metrics comparison view