Reinforcement Learning

Get started with RL

• Mountain Car with Amazon SageMaker RL
  • PatientMountainCar
  • PatientContinuousMountainCar
    • Pre-requisites
  • Imports
  • Setup S3 bucket
  • Define Variables
  • Configure settings
  • Create an IAM role
  • Install docker for local mode
    • Setup the environment
    • Configure the presets for RL algorithm
    • Write the Training Code
    • Train the RL model using the Python SDK Script mode
    • Store intermediate training output and model checkpoints
    • Visualization
  • Plot metrics for training job
  • Visualize the rendered gifs
    • Evaluation of RL models
  • Load checkpointed model
  • Run the evaluation step
  • Visualize the output
    • Model deployment
  • Clean up endpoint
    • Notebook CI Test Results

Cart pole

A cart pole simulation is the act of balancing a broom upright by balancing it on your hand. The broom is the “pole” and your hand is replaced with a “cart” moving back and forth on a linear track. This simplified example works in 2 dimensions, so the cart can only move in a line back and forth, and the pole can only fall forwards or backwards, not to the sides. These examples use PyTorch or TensorFlow and SageMaker RL to solve a cart pole problem.

Contextual bandits

Explore a number of actions with contextual bandits algorithms in SageMaker.

• Contextual Bandits with Parametric Actions – Experimentation Mode
  • What is Experimentation Mode?
    • Pre-requisites
  • Imports
  • Setup S3 bucket
  • Configure where training happens
  • Create an IAM role
  • Simulation environment (from MovieLens data)
    • Train the Bandit model using the Python SDK Script mode
      • Define the hyperparameters and the training job name prefix
      • Download the outputs to plot performance
  • Create a SageMaker model for inference
  • 1. Batch Transform
  • Generating test dataset for inference
  • Download batch transform results
  • 2. Real-time inference
  • Clean Up endpoint
    • Notebook CI Test Results
• Contextual Bandits with Amazon SageMaker RL
  • Why Contextual Bandits?
  • What does this notebook contain?
    • Local Mode
  • Configuration
  • IAM role setup
    • Additional permissions for the IAM role
  • Client application (Environment)
  • Step-by-step bandits model development
    • 1. Model Initialization
    • 2. Deploy the First Model
    • 3. Initialize the Client Application
    • 4. Reward Ingestion
    • 5. Model Re-training and Re-deployment
  • Bandits model deployment with the end-to-end loop
  • Visualization
    • Get mean rewards
  • Clean up

Roboschool

Roboschool is a physics simulator that is commonly used to train RL policies for robotic systems.

Use cases

Autoscaling

This example demonstrates how to use RL to address scaling a production service by adding and removing resources (e.g. servers or EC2 instances) in reaction to a dynamic load.

Energy

Training an RL algorithm in a real HVAC system can take time to converge as well as potentially lead to hazardous settings as the agent explores its state space. This example uses the EnergyPlus simulator to showcase how you can train an HVAC optimization RL model with Amazon SageMaker.

• HVAC with Amazon SageMaker RL
  • Introduction
  • Pre-requisites
    • Imports
    • Setup S3 bucket
    • Define Variables
    • Configure settings
    • Create an IAM role
    • Install docker for local mode
  • Build docker container
  • Setup the environment
  • Configure the presets for RL algorithm
  • Write the Training Code
  • Train the RL model using the Python SDK Script mode
  • Store intermediate training output and model checkpoints
  • Visualization
    • Plot metrics for training job
  • Evaluation of RL models
    • Load checkpointed model
    • Run the evaluation step
  • Model deployment
    • Clean up endpoint
  • Notebook CI Test Results

Game play

Use RL to train an agent to play in a Unity3D environment.

• Unity3D Game with Amazon SageMaker RL
  • Introduction
  • Pre-requisites
    • Imports
    • Setup S3 bucket
    • Define Variables
    • Configure where training happens
    • Create an IAM role
    • Install docker for local mode
  • Use Unity Example Environment
    • Write the Training Code
    • Train the RL model using the Python SDK Script mode
    • Plot metrics for training job
  • (Optional) Bring customized Unity environment
    • Train the RL model using the Python SDK Script mode with provided Unity executables
  • Evaluation of RL models
    • Load checkpointed model
  • Model deployment
    • Clean up endpoint
  • Notebook CI Test Results

Game server

A reinforcement learning-based system using SageMaker Autopilot and SageMaker RL that learns to allocate resources in response to player usage patterns.

• Game servers autopilot
  • Pre-requisites
    • Imports
    • Setup S3 bucket
    • Define Variables
    • Parameters
    • Create an IAM role
  • Set up the environment
  • Configure the presets for RL algorithm
  • Train the RL model using the Python SDK Script mode
  • Store intermediate training output and model checkpoints
  • Evaluation of RL models
    • Load checkpointed model
  • Run the evaluation step
  • Hosting
    • Model deployment
    • Inference
    • Delete the Endpoint
  • Notebook CI Test Results

Knapsack problem

Use SageMaker RL to address a canonical operations research problem, aka, a knapsack problem.

Object tracker

Use RL to train a TurtleBot object tracker using Amazon SageMaker Reinforcement Learning and AWS RoboMaker.

Network compression

Network to network compression via policy gradient reinforcement learning.

Portfolio management

Use SageMaker RL to manage a stock portfolio by continuously reallocating several stocks.

Resource allocation

Solve resource allocation problems with SageMaker RL.

Tic-tac-toe

Play global thermonuclear war with a computer.

Stock Trading

Try stock trading with SageMaker RL.

• Stock Trading with Amazon SageMaker RL
  • Problem Statement
  • Dataset
    • Dataset License
  • Using Amazon SageMaker for RL
  • Pre-requisites
    • Roles and permissions
    • Steup S3 buckets
    • Define Variables
    • Configure settings
    • Create an IAM role
    • Install docker for local mode
  • Set up the environment
  • Configure the presets for RL algorithm
  • Write the Training Code
  • Train the RL model using the Python SDK Script mode
  • Store intermediate training output and model checkpoints
  • Visualization
    • Plot rate of learning
  • Load the checkpointed models for evaluation
    • Run the evaluation step
  • Risk Disclaimer (for live-trading)
  • Notebook CI Test Results

Traveling salesman problem

Use SageMaker RL to solve this classic problem with a twist: a restaurant delivery service on a 2D gridworld.

• Traveling Salesman Problem with Reinforcement Learning
  • Description of Problem
  • Why Reinforcement Learning?
  • Easy Version of TSP
    • States
    • Actions
    • Rewards
  • Using AWS SageMaker for RL
    • Prerequisites
      • Imports
      • Settings
      • Install docker for local mode
      • Create an IAM role
      • Setup the environment
      • Configure the presets for RL algorithm
      • Write the Training Code
    • Train the RL model using the Python SDK Script mode
    • Store intermediate training output and model checkpoints
    • Visualization
      • Comparing against a baseline policy
      • Plot metrics for training job
      • Visualize the rendered gifs
    • Evaluation of RL models
      • Load checkpointed model
      • Run the evaluation step
  • Medium version of TSP
    • States
    • Actions
    • Rewards
  • Using AWS SageMaker for RL
    • Train the model using Python SDK/Script mode
  • Visualize, Compare with Baseline and Evaluate
  • Vehicle Routing Problem with Reinforcement Learning
    • States
    • Actions
    • Rewards
  • Using AWS SageMaker RL
    • Train the model using Python SDK/Script mode
  • Visualize, Compare with Baseline and Evaluate
  • Notebook CI Test Results