Debugger

Examples on how to use SageMaker Debugger.

Get started with SageMaker Debugger

Debugging

Profiling

• Monitor System Bottlenecks and Profile Framework Operators using Amazon Debugger
• Profile machine learning training with Amazon SageMaker Debugger
• Profiling PyTorch Single GPU Single Node Training Job with Amazon SageMaker Debugger

---

Debugging Model Parameters

You can track and debug model parameters, such as weights, gradients, biases, and scalar values of your training job. Available deep learning frameworks are Apache MXNet, TensorFlow, PyTorch, and XGBoost.

Real-time analysis of deep learning models

• Using SageMaker debugger to monitor autoencoder model training
• Using SageMaker debugger to monitor attentions in BERT model training

Apache MXNet

TensorFlow 2.x

TensorFlow 1.x

• Amazon SageMaker Debugger Tutorial: How to Use the Built-in Debugging Rules
• Amazon SageMaker Debugger - Reacting to Cloudwatch Events from Rules
• Amazon SageMaker - Debugging with custom rules
• Detect Stalled Training and Invoke Actions Using SageMaker Debugger Rule

PyTorch

• Using SageMaker Debugger and SageMaker Experiments for iterative model pruning
• Notebook CI Test Results
• Using Amazon SageMaker Debugger for PyTorch Training Jobs

XGBoost

• Explainability with Amazon SageMaker Debugger

Bring your own container

• Build a Custom Training Container and Debug Training Jobs with Amazon SageMaker Debugger

---

Profiling System Bottlenecks and Framework Operators

Debugger provides the following profile features:

• Monitoring system bottlenecks – Monitor system resource utilization rate, such as CPU, GPU, memories, network, and data I/O metrics. This is a framework and model agnostic feature and available for any training jobs in SageMaker.

• Profiling deep learning framework operations – Profile deep learning operations of the TensorFlow and PyTorch frameworks, such as step durations, data loaders, forward and backward operations, Python profiling metrics, and framework-specific metrics.

Tensorflow

• Profiling TensorFlow Single GPU Single Node Training Job with Amazon SageMaker Debugger
• Profiling TensorFlow Multi GPU Multi Node Training Job with Amazon SageMaker Debugger (SageMaker SDK)
• How to identify low GPU utilization due to small batch size

PyTorch

• Profiling PyTorch Single GPU Single Node Training Job with Amazon SageMaker Debugger
• Profiling PyTorch Multi GPU Multi Node Training Job with Amazon SageMaker Debugger