SageMaker Model Monitor - visualizing monitoring results

This notebook’s CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

The prebuilt container from SageMaker computes a variety of statistics and evaluates constraints out of the box. This notebook demonstrates how you can visualize them. You can grab the ProcessingJob arn from the executions behind a MonitoringSchedule and use this notebook to visualize the results.

Let’s import some python libraries that will be helpful for visualization

[ ]:

from IPython.display import HTML, display
import json
import os
import boto3

import sagemaker
from sagemaker import session
from sagemaker.model_monitor import MonitoringExecution
from sagemaker.s3 import S3Downloader

Get Utilities for Rendering

The functions for plotting and rendering distribution statistics or constraint violations are implemented in a utils file so let’s grab that.

[ ]:

!wget https://raw.githubusercontent.com/awslabs/amazon-sagemaker-examples/main/sagemaker_model_monitor/visualization/utils.py

import utils as mu

Get Execution and Baseline details from Processing Job Arn

Enter the ProcessingJob arn for an execution of a MonitoringSchedule below to get the result files associated with that execution

[ ]:

processing_job_arn = "FILL-IN-PROCESSING-JOB-ARN"

[ ]:

execution = MonitoringExecution.from_processing_arn(
    sagemaker_session=session.Session(), processing_job_arn=processing_job_arn
)
exec_inputs = {inp["InputName"]: inp for inp in execution.describe()["ProcessingInputs"]}
exec_results = execution.output.destination

[ ]:

baseline_statistics_filepath = (
    exec_inputs["baseline"]["S3Input"]["S3Uri"] if "baseline" in exec_inputs else None
)
execution_statistics_filepath = os.path.join(exec_results, "statistics.json")
violations_filepath = os.path.join(exec_results, "constraint_violations.json")

baseline_statistics = (
    json.loads(S3Downloader.read_file(baseline_statistics_filepath))
    if baseline_statistics_filepath is not None
    else None
)
execution_statistics = json.loads(S3Downloader.read_file(execution_statistics_filepath))
violations = json.loads(S3Downloader.read_file(violations_filepath))["violations"]

Overview

The code below shows the violations and constraint checks across all features in a simple table.

[ ]:

mu.show_violation_df(
    baseline_statistics=baseline_statistics,
    latest_statistics=execution_statistics,
    violations=violations,
)

Distributions

This section visualizes the distribution and renders the distribution statistics for all features

[ ]:

features = mu.get_features(execution_statistics)
feature_baselines = mu.get_features(baseline_statistics)

[ ]:

mu.show_distributions(features)

Execution Stats vs Baseline

[ ]:

mu.show_distributions(features, feature_baselines)

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.