Debugger

Examples on how to use SageMaker Debugger.

Get started with SageMaker Debugger

Debugging

• Debugging Amazon SageMaker training jobs in real time with Debugger
• Amazon SageMaker Debugger XGBoost training report for Higgs Boson Detection Challenge

Profiling

• Monitor System Bottlenecks and Profile Framework Operators using Amazon Debugger
• Profile machine learning training with Amazon SageMaker Debugger
• Profiling PyTorch Single GPU Single Node Training Job with Amazon SageMaker Debugger

---

Debugging Model Parameters

You can track and debug model parameters, such as weights, gradients, biases, and scalar values of your training job. Available deep learning frameworks are Apache MXNet, TensorFlow, PyTorch, and XGBoost.

Real-time analysis of deep learning models

• Using SageMaker debugger to monitor autoencoder model training
• Using SageMaker debugger to monitor attentions in BERT model training
• Using SageMaker debugger to visualize class activation maps in CNNs

Apache MXNet

• Enable Spot Training with Amazon SageMaker Debugger
• Tensor Analysis Using Amazon SageMaker Debugger
• Visualizing Debugging Tensors of MXNet training

TensorFlow 2.x

• Amazon SageMaker - Tensorflow 2.1

TensorFlow 1.x

• Amazon SageMaker Debugger Tutorial: How to Use the Built-in Debugging Rules
• Amazon SageMaker Debugger - Reacting to Cloudwatch Events from Rules
• Amazon SageMaker - Debugging with custom rules
• Detect Stalled Training and Invoke Actions Using SageMaker Debugger Rule

PyTorch

• Using SageMaker Debugger and SageMaker Experiments for iterative model pruning
• Notebook CI Test Results
• Using SageMaker Debugger and SageMaker Experiments for iterative model pruning (AlexNet)
• Using Amazon SageMaker Debugger for PyTorch Training Jobs

XGBoost

• Debugging XGBoost training jobs in real time with Amazon SageMaker Debugger
• Explainability with Amazon SageMaker Debugger
• Amazon SageMaker Debugger XGBoost training report for Higgs Boson Detection Challenge

Bring your own container

• Build a Custom Training Container and Debug Training Jobs with Amazon SageMaker Debugger

---

Profiling System Bottlenecks and Framework Operators

Debugger provides the following profile features:

• Monitoring system bottlenecks – Monitor system resource utilization rate, such as CPU, GPU, memories, network, and data I/O metrics. This is a framework and model agnostic feature and available for any training jobs in SageMaker.

• Profiling deep learning framework operations – Profile deep learning operations of the TensorFlow and PyTorch frameworks, such as step durations, data loaders, forward and backward operations, Python profiling metrics, and framework-specific metrics.

Tensorflow

• Profiling TensorFlow Single GPU Single Node Training Job with Amazon SageMaker Debugger
• Profiling TensorFlow Multi GPU Multi Node Training Job with Amazon SageMaker Debugger (SageMaker SDK)
• Profiling TensorFlow Multi GPU Multi Node Training Job with Amazon SageMaker Debugger (SageMaker API)
• How to identify low GPU utilization due to small batch size
• Identify a CPU bottleneck caused by a callback process with Amazon SageMaker Debugger

PyTorch

• Profiling PyTorch Single GPU Single Node Training Job with Amazon SageMaker Debugger
• Profiling PyTorch Multi GPU Multi Node Training Job with Amazon SageMaker Debugger