Reinforcement Learning

Get started with RL

• Mountain Car with Amazon SageMaker RL
  • PatientMountainCar
  • PatientContinuousMountainCar
    • Pre-requisites
  • Imports
  • Setup S3 bucket
  • Define Variables
  • Configure settings
  • Create an IAM role
  • Install docker for local mode
    • Setup the environment
    • Configure the presets for RL algorithm
    • Write the Training Code
    • Train the RL model using the Python SDK Script mode
    • Store intermediate training output and model checkpoints
    • Visualization
  • Plot metrics for training job
  • Visualize the rendered gifs
    • Evaluation of RL models
  • Load checkpointed model
  • Run the evaluation step
  • Visualize the output
    • Model deployment
  • Clean up endpoint
• Cart-pole Balancing Model with Amazon SageMaker and Ray
  • Introduction
  • Pre-requisites
    • Imports
    • Setup S3 bucket
    • Define Variables
    • Configure where training happens
    • Create an IAM role
    • Install docker for local mode
  • Write the Training Code
  • Train the RL model using the Python SDK Script mode
    • Plot metrics for training job
  • Evaluation of RL models
    • Load checkpointed model
  • Model deployment
    • Clean up endpoint
• Roboschool simulations of physical robotics with Amazon SageMaker
  • Introduction
  • Env
  • Pre-requisites
    • Imports
    • Setup S3 bucket
    • Define Variables
    • Configure where training happens
    • Create an IAM role
    • Install docker for local mode
  • Build docker container
  • Write the Training Code
  • Train the RL model using the Python SDK Script mode
  • Visualization
    • Fetch videos of training rollouts
    • Plot metrics for training job
    • Monitor training progress
  • Evaluation of RL models
    • Load checkpointed model
    • Visualize the output
  • Model deployment
    • Clean up endpoint

Cart pole

A cart pole simulation is the act of balancing a broom upright by balancing it on your hand. The broom is the “pole” and your hand is replaced with a “cart” moving back and forth on a linear track. This simplified example works in 2 dimensions, so the cart can only move in a line back and forth, and the pole can only fall forwards or backwards, not to the sides. These examples use PyTorch or TensorFlow and SageMaker RL to solve a cart pole problem.

• Cart-pole Balancing Model with Amazon SageMaker and Coach library
  • Introduction
  • Pre-requisites
    • Imports
    • Setup S3 bucket
    • Define Variables
    • Configure where training happens
    • Create an IAM role
    • Install docker for local mode
  • Setup the environment
  • Configure the presets for RL algorithm
  • Write the Training Code
  • Train the RL model using the Python SDK Script mode
  • Store intermediate training output and model checkpoints
  • Visualization
    • Plot metrics for training job
    • Visualize the rendered gifs
  • Evaluation of RL models
    • Load checkpointed model
    • Run the evaluation step
    • Visualize the output
  • Model deployment
    • Clean up endpoint
• Training Batch Reinforcement Learning Policies with Amazon SageMaker RL and Coach library
  • Pre-requisites
    • Roles and permissions
    • Setup S3 buckets
    • Define Variables
    • Configure settings
    • Create an IAM role
    • Install docker for local mode
  • Collect offline data
    • Save Dataframe as CSV for Batch RL Training
  • Configure the presets for RL algorithm
  • Write the Training Code
  • Train the RL model using the Python SDK Script mode
  • Store intermediate training output and model checkpoints
  • Visualization
    • Plot metrics for training job
  • Evaluation of RL models
    • Load checkpointed model
    • Batch Transform
    • Reference
• Cart-pole Balancing Model with Amazon SageMaker on SageMaker Managed Spot Training
  • Introduction
  • Pre-requisites
    • Imports
    • Setup S3 bucket
    • Define Variables
    • Configure where training happens
    • Create an IAM role
    • Install docker for local mode
  • Setup the environment
  • Configure the presets for RL algorithm
  • Write the Training Code
  • Train the RL model using the Python SDK Script mode
  • Store intermediate training output and model checkpoints
  • Visualization
    • Plot metrics for training job
    • Visualize the rendered gifs
  • Evaluation of RL models
    • Load checkpointed model
    • Run the evaluation step
    • Visualize the output
  • Model deployment
    • Clean up endpoint

Contextual bandits

Explore a number of actions with contextual bandits algorithms in SageMaker.

• Contextual Bandits with Parametric Actions – Experimentation Mode
  • What is Experimentation Mode?
    • Pre-requisites
  • Imports
  • Setup S3 bucket
  • Configure where training happens
  • Create an IAM role
  • Simulation environment (from MovieLens data)
    • Train the Bandit model using the Python SDK Script mode
      • Define the hyperparameters and the training job name prefix
      • Download the outputs to plot performance
  • Create a SageMaker model for inference
  • 1. Batch Transform
  • Generating test dataset for inference
  • Download batch transform results
  • 2. Real-time inference
  • Clean Up endpoint
• Contextual Bandits with Amazon SageMaker RL
  • Why Contextual Bandits?
  • What does this notebook contain?
    • Local Mode
  • Configuration
  • IAM role setup
    • Additional permissions for the IAM role
  • Client application (Environment)
  • Step-by-step bandits model development
    • 1. Model Initialization
    • 2. Deploy the First Model
    • 3. Initialize the Client Application
    • 4. Reward Ingestion
    • 5. Model Re-training and Re-deployment
  • Bandits model deployment with the end-to-end loop
  • Visualization
    • Get mean rewards
  • Clean up

Roboschool

Roboschool is a physics simulator that is commonly used to train RL policies for robotic systems.

• Tune hyperparameters for your RL training job
  • Env
  • Pre-requisites
    • Imports
    • Setup S3 bucket
    • Define Variables
    • Configure resources to be used for tuning
    • Create an IAM role
  • Build docker container
  • Configure Tuning
  • Prepare to launch the tuning job.
  • Monitor progress
• Training Roboschool agents using distributed RL training across multiple nodes with Amazon SageMaker
  • Env
  • Pre-requisites
    • Imports
    • Setup S3 bucket
    • Define Variables
    • Configure where training happens
    • Create an IAM role
    • Install docker for local mode
  • Build docker container
  • Write the Training Code
  • Ray homogeneous scaling - Specify train_instance_count > 1
    • Visualization
    • Fetch videos of training rollouts
    • Plot metrics for training job
    • Monitor training progress
  • Ray heterogeneous scaling
    • Configure instance types
    • Launch primary cluster (1 GPU training instance)
    • Launch secondary cluster (2 CPU instances)
    • Visualization
    • Fetch videos of training rollouts
    • Plot metrics for training job
• Roboschool simulations training with stable baselines on AWS SageMaker RL
  • Introductions
  • Pre-requisites
    • Imports
    • Setup S3 bucket
    • Define Variables
    • Configure where training happens
    • Create an IAM role
    • Install docker for local mode
  • Build docker container
  • Write the Training Code
  • Train the RL model using the Python SDK Script mode
  • Visualization
    • Fetch videos of training rollouts
    • RL output video
    • Example of trained walking HalfCheetah

Use cases

Autoscaling

This example demonstrates how to use RL to address scaling a production service by adding and removing resources (e.g. servers or EC2 instances) in reaction to a dynamic load.

• Autoscaling a service with Amazon SageMaker
  • Problem Statement
  • Using Amazon SageMaker for RL
  • Pre-requisites
    • Roles and permissions
    • Steup S3 buckets
    • Define Variables
    • Configure settings
    • Create an IAM role
    • Install docker for local mode
  • Set up the environment
    • Visualize the simulated load
  • Configure the presets for RL algorithm
  • Write the Training Code
  • Train the RL model using the Python SDK Script mode
  • Store intermediate training output and model checkpoints
  • Visualization
    • Plot rate of learning
  • Evaluation of RL models
    • Load the checkpointed models
    • Run the evaluation step
  • Hosting
    • Inference
    • Delete the Endpoint

Energy

Training an RL algorithm in a real HVAC system can take time to converge as well as potentially lead to hazardous settings as the agent explores its state space. This example uses the EnergyPlus simulator to showcase how you can train an HVAC optimization RL model with Amazon SageMaker.

• HVAC with Amazon SageMaker RL
  • Introduction
  • Pre-requisites
    • Imports
    • Setup S3 bucket
    • Define Variables
    • Configure settings
    • Create an IAM role
    • Install docker for local mode
  • Build docker container
  • Setup the environment
  • Configure the presets for RL algorithm
  • Write the Training Code
  • Train the RL model using the Python SDK Script mode
  • Store intermediate training output and model checkpoints
  • Visualization
    • Plot metrics for training job
  • Evaluation of RL models
    • Load checkpointed model
    • Run the evaluation step
  • Model deployment
    • Clean up endpoint

Game play

Use RL to train an agent to play in a Unity3D environment.

• Unity3D Game with Amazon SageMaker RL
  • Introduction
  • Pre-requisites
    • Imports
    • Setup S3 bucket
    • Define Variables
    • Configure where training happens
    • Create an IAM role
    • Install docker for local mode
  • Use Unity Example Environment
    • Write the Training Code
    • Train the RL model using the Python SDK Script mode
    • Plot metrics for training job
  • (Optional) Bring customized Unity environment
    • Train the RL model using the Python SDK Script mode with provided Unity executables
  • Evaluation of RL models
    • Load checkpointed model
  • Model deployment
    • Clean up endpoint

Game server

A reinforcement learning-based system using SageMaker Autopilot and SageMaker RL that learns to allocate resources in response to player usage patterns.

• Game servers autopilot
  • Pre-requisites
    • Imports
    • Setup S3 bucket
    • Define Variables
    • Parameters
    • Create an IAM role
  • Set up the environment
  • Configure the presets for RL algorithm
  • Train the RL model using the Python SDK Script mode
  • Store intermediate training output and model checkpoints
  • Evaluation of RL models
    • Load checkpointed model
  • Run the evaluation step
  • Hosting
    • Model deployment
    • Inference
    • Delete the Endpoint

Knapsack problem

Use SageMaker RL to address a canonical operations research problem, aka, a knapsack problem.

• Solving Knapsack Problem with Amazon SageMaker RL
  • Problem Statement
  • Using Amazon SageMaker RL
    • Pre-requsites
      • Imports
      • Settings
      • Install docker for local mode
      • Create an IAM role
      • Setup the environment
      • Configure the presets for RL algorithm
      • Write the Training Code
    • Train the model using Python SDK/ script mode
    • Store intermediate training output and model checkpoints
    • Visualization
      • Plot metrics for training job
      • Visualize the rendered gifs
    • Evaluation of RL models
      • Load checkpointed model
      • Run the evaluation step
    • Visualize the output

Object tracker

Use RL to train a TurtleBot object tracker using Amazon SageMaker Reinforcement Learning and AWS RoboMaker.

Network compression

Network to network compression via policy gradient reinforcement learning.

• Distributed Neural Network Compression using Reinforcement Learning
  • Introduction
  • Attribution
  • Pre-requisites
    • Roles and permissions
    • Auxiliary services and settings
  • The gamification of neural network compression
    • A custom gym environment
    • Setup data and upload to S3
    • Prepare teacher weights
    • The RL agent
    • Training
  • Process Outputs

Portfolio management

Use SageMaker RL to manage a stock portfolio by continuously reallocating several stocks.

• Portfolio Management with Amazon SageMaker RL
  • Problem Statement
  • Dataset
    • Dataset License
  • Using reinforcement learning on Amazon SageMaker RL
  • Pre-requisites
    • Roles and permissions
    • Steup S3 buckets
    • Define Variables
    • Configure settings
    • Create an IAM role
    • Install docker for local mode
  • Set up the environment
  • Configure the presets for RL algorithm
  • Write the Training Code
  • Train the RL model using the Python SDK Script mode
  • Store intermediate training output and model checkpoints
  • Visualization
    • Plot rate of learning
    • Visualize the portfolio value
  • Load the checkpointed models for evaluation
    • Run the evaluation step
  • Risk Disclaimer (for live-trading)

Resource allocation

Solve resource allocation problems with SageMaker RL.

• Solving Bin Packing Problem with Amazon SageMaker RL
  • Problem Statement
  • Using Amazon SageMaker RL
  • Pre-requsites
    • Roles and permissions
    • Setup S3 bucket
    • Define Variables
    • Configure where training happens
    • Create an IAM role
    • Install docker for local mode
  • Setup the environment
  • Write the training code
  • Train the RL model using the Python SDK Script mode
    • Define Metric
    • Define Estimator
  • Visualization
    • Plot metrics for training job
      • Monitor training progress
  • Training Results
• Solving Multi-Period Newsvendor Problem with Amazon SageMaker RL
  • Problem Statement
  • Using Amazon SageMaker RL
  • Pre-requsites
    • Roles and permissions
    • Setup S3 bucket
    • Define Variables
    • Configure where training happens
    • Create an IAM role
    • Install docker for local mode
  • Setup the environment
  • Write the training code
  • Train the RL model using the Python SDK Script mode
    • Define Metric
    • Define Estimator
  • Visualization
    • Plot metrics for training job
      • Monitor training progress
  • Training Results
• Solving Vehicle Routing Problem with Amazon SageMaker RL
  • NOTE: This notebook only works with an older version of Ray and Tensorflow. We are not planning to upgrade this notebook to use the latest Ray at the moment.
  • Problem Statement
  • Using Amazon SageMaker for RL
  • Pre-requisites
    • Roles and permissions
    • Setup S3 bucket
    • Define Variables
    • Configure settings
    • Create an IAM role
    • Install docker for local mode
  • Set up the environment
  • Write the training code
  • Train the RL model using the Python SDK Script mode
    • Define Metric
    • Define Estimator
  • Visualization
    • Plot metrics for training job
      • Monitor training progress
  • Training Results

Tic-tac-toe

Play global thermonuclear war with a computer.

• Learning Tic-Tac-Toe with Reinforcement Learning
  • Outline
  • Overview
  • Setup
  • Code
    • Environment
    • Preset
    • Launcher
  • Train
  • Deploy
    • Inference
    • Endpoint
  • Play
  • Wrap Up

Traveling salesman problem

Use SageMaker RL to solve this classic problem with a twist: a restaurant delivery service on a 2D gridworld.

• Traveling Salesman Problem with Reinforcement Learning
  • Description of Problem
  • Why Reinforcement Learning?
  • Easy Version of TSP
    • States
    • Actions
    • Rewards
  • Using AWS SageMaker for RL
    • Prerequisites
      • Imports
      • Settings
      • Install docker for local mode
      • Create an IAM role
      • Setup the environment
      • Configure the presets for RL algorithm
      • Write the Training Code
    • Train the RL model using the Python SDK Script mode
    • Store intermediate training output and model checkpoints
    • Visualization
      • Comparing against a baseline policy
      • Plot metrics for training job
      • Visualize the rendered gifs
    • Evaluation of RL models
      • Load checkpointed model
      • Run the evaluation step
  • Medium version of TSP
    • States
    • Actions
    • Rewards
  • Using AWS SageMaker for RL
    • Train the model using Python SDK/Script mode
  • Visualize, Compare with Baseline and Evaluate
  • Vehicle Routing Problem with Reinforcement Learning
    • States
    • Actions
    • Rewards
  • Using AWS SageMaker RL
    • Train the model using Python SDK/Script mode
  • Visualize, Compare with Baseline and Evaluate