!pip install smdebug matplotlib

Amazon SageMaker Debugger Tutorial: How to Use the Built-in Debugging Rules

---

This notebook’s CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

---

Amazon SageMaker Debugger is a feature that offers capability to debug training jobs of your machine learning model and identify training problems in real time. While a training job looks like it’s working like a charm, the model might have some common problems, such as loss not decreasing, overfitting, and underfitting. To better understand, practitioners have to debug the training job, while it can be challenging to track and analyze all of the output tensors.

SageMaker Debugger covers the major deep learning frameworks (TensorFlow, PyTorch, and MXNet) and machine learning algorithm (XGBoost) to do the debugging jobs with minimal coding. Debugger provides an automatic detection of training problems through its built-in rules, and you can find a full list of the built-in rules for debugging at List of Debugger Built-in Rules.

In this tutorial, you will learn how to use SageMaker Debugger and its built-in rules to debug your model.

The workflow is as follows: * Step 1: Import SageMaker Python SDK and the Debugger client library smdebug * Step 2: Create a Debugger built-in rule list object * Step 3: Construct a SageMaker estimator * Step 4: Run the training job * Step 5: Check training progress on Studio Debugger insights dashboard and the built-in rules evaluation status * Step 6: Create a Debugger trial object to access the saved tensors

## Step 1: Import SageMaker Python SDK and the SMDebug client library

Important: To use the new Debugger features, you need to upgrade the SageMaker Python SDK and the SMDebug libary. In the following cell, change the third line to install_needed=True and run to upgrade the libraries.

import sys
import IPython
install_needed = False  # Set to True to upgrade
if install_needed:
    print("installing deps and restarting kernel")
    !{sys.executable} -m pip install -U sagemaker
    !{sys.executable} -m pip install -U smdebug
    IPython.Application.instance().kernel.do_shutdown(True)

Check the SageMaker Python SDK and the SMDebug library versions.

import sagemaker

sagemaker.__version__

import smdebug

smdebug.__version__

## Step 2: Create a Debugger built-in rule list object

from sagemaker.debugger import Rule, ProfilerRule, rule_configs

The following code cell shows how to configure a rule object for debugging and profiling. For more information about the Debugger built-in rules, see List of Debugger Built-in Rules.

built_in_rules = [
    Rule.sagemaker(rule_configs.overfit()),
    ProfilerRule.sagemaker(rule_configs.ProfilerReport()),
]

## Step 3: Construct a SageMaker estimator

Using the rule object created in the previous cell, construct a SageMaker estimator.

The estimator can be one of the SageMaker framework estimators, TensorFlow, PyTorch, MXNet, and XGBoost, or the SageMaker generic estimator. For more information about what framework versions are supported, see Debugger-supported Frameworks and Algorithms.

In this tutorial, the SageMaker TensorFlow estimator is constructed to run a TensorFlow training script with the Keras ResNet50 model and the cifar10 dataset.

import boto3
from sagemaker.tensorflow import TensorFlow

session = boto3.session.Session()
region = session.region_name

estimator = TensorFlow(
    role=sagemaker.get_execution_role(),
    instance_count=1,
    instance_type="ml.p3.8xlarge",
    image_uri=f"763104351884.dkr.ecr.{region}.amazonaws.com/tensorflow-training:2.3.1-gpu-py37-cu110-ubuntu18.04",
    # framework_version='2.3.1',
    # py_version="py37",
    max_run=3600,
    source_dir="./src",
    entry_point="tf-resnet50-cifar10.py",
    # Debugger Parameters
    rules=built_in_rules,
)

## Step 4: Run the training job With the wait=False option, you can proceed to the next notebook cell without waiting for the training job logs to be printed out.

estimator.fit(wait=False)

## Step 5: Check training progress on Studio Debugger insights dashboard and the built-in rules evaluation status

• Option 1 - Use SageMaker Studio Debugger insights and Experiments. This is a non-coding approach.

• Option 2 - Use the following code cells. This is a code-based approach.

Option 1 - Open Studio Debugger insights dashboard to get insights into the training job

Through the Debugger insights dashboard on Studio, you can check the training jobs status, system resource utilization, and suggestions to optimize model performance. The following screenshot shows the Debugger insights dashboard interface.

The following heatmap shows the ml.p3.8xlarge instance utilization while the training job is running or after the job has completed. To learn how to access the Debugger insights dashboard, see Debugger on Studio in the SageMaker Debugger developer guide.

Option 2 - Run the following scripts for the code-based option

The following two code cells return the current training job name, status, and the rule status in real time.

Print the training job name

job_name = estimator.latest_training_job.name
print("Training job name: {}".format(job_name))

Print the training job and rule evaluation status

The following script returns the status in real time every 15 seconds, until the secondary training status turns to one of the descriptions, Training, Stopped, Completed, or Failed. Once the training job status turns into the Training, you will be able to retrieve tensors from the default S3 bucket.

import time

client = estimator.sagemaker_session.sagemaker_client
description = client.describe_training_job(TrainingJobName=job_name)
if description["TrainingJobStatus"] != "Completed":
    while description["SecondaryStatus"] not in {"Training", "Stopped", "Completed", "Failed"}:
        description = client.describe_training_job(TrainingJobName=job_name)
        primary_status = description["TrainingJobStatus"]
        secondary_status = description["SecondaryStatus"]
        print(
            "Current job status: [PrimaryStatus: {}, SecondaryStatus: {}] | {} Rule Evaluation Status: {}".format(
                primary_status,
                secondary_status,
                estimator.latest_training_job.rule_job_summary()[0]["RuleConfigurationName"],
                estimator.latest_training_job.rule_job_summary()[0]["RuleEvaluationStatus"],
            )
        )
        time.sleep(30)

## Step 6: Create a Debugger trial object to access the saved model parameters

To access the saved tensors by Debugger, use the smdebug client library to create a Debugger trial object. The following code cell sets up a tutorial_trial object, and waits until it finds available tensors from the default S3 bucket.

from smdebug.trials import create_trial

tutorial_trial = create_trial(estimator.latest_job_debugger_artifacts_path())

The Debugger trial object accesses the SageMaker estimator’s Debugger artifact path, and fetches the output tensors saved for debugging.

Print the default S3 bucket URI where the Debugger output tensors are stored

tutorial_trial.path

Print the Debugger output tensor names

tutorial_trial.tensor_names()

Print the list of steps where the tensors are saved

The smdebug ModeKeys class provides training phase mode keys that you can use to sort training (TRAIN) and validation (EVAL) steps and their corresponding values.

from smdebug.core.modes import ModeKeys

tutorial_trial.steps(mode=ModeKeys.TRAIN)

tutorial_trial.steps(mode=ModeKeys.EVAL)

Plot the loss curve

The following script plots the loss and accuracy curves of training and validation loops.

trial = tutorial_trial


def get_data(trial, tname, mode):
    tensor = trial.tensor(tname)
    steps = tensor.steps(mode=mode)
    vals = [tensor.value(s, mode=mode) for s in steps]
    return steps, vals

import matplotlib.pyplot as plt
from mpl_toolkits.axes_grid1 import host_subplot


def plot_tensor(trial, tensor_name):
    tensor_name = tensor_name
    steps_train, vals_train = get_data(trial, tensor_name, mode=ModeKeys.TRAIN)
    steps_eval, vals_eval = get_data(trial, tensor_name, mode=ModeKeys.EVAL)

    fig = plt.figure(figsize=(10, 7))
    host = host_subplot(111)

    par = host.twiny()

    host.set_xlabel("Steps (TRAIN)")
    par.set_xlabel("Steps (EVAL)")
    host.set_ylabel(tensor_name)

    (p1,) = host.plot(steps_train, vals_train, label=tensor_name)
    (p2,) = par.plot(steps_eval, vals_eval, label="val_" + tensor_name)

    leg = plt.legend()

    host.xaxis.get_label().set_color(p1.get_color())
    leg.texts[0].set_color(p1.get_color())

    par.xaxis.get_label().set_color(p2.get_color())
    leg.texts[1].set_color(p2.get_color())

    plt.ylabel(tensor_name)

    plt.show()


plot_tensor(trial, "loss")
plot_tensor(trial, "accuracy")

Note : Rerun the above cell if you don’t see any plots!

In this tutorial, you learned how to use SageMaker Debugger with the minimal coding through SageMaker Studio and Jupyter notebook. The Debugger built-in rules detect training anomalies while concurrently reading in the output tensors, such as weights, activation outputs, gradients, accuracy, and loss, from your training jobs. In the next tutorial videos, you will learn more features of Debugger, such as how to analyze the tensors, change the built-in debugging rule parameters and thresholds, and save the tensors at your preferred S3 bucket URI.

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.