{ "cells": [ { "cell_type": "markdown", "id": "looking-courage", "metadata": { "papermill": { "duration": 0.024036, "end_time": "2021-05-27T00:07:33.575998", "exception": false, "start_time": "2021-05-27T00:07:33.551962", "status": "completed" }, "tags": [] }, "source": [ "# Amazon SageMaker Studio Walkthrough\n" ] }, { "attachments": {}, "cell_type": "markdown", "id": "576a382a", "metadata": {}, "source": [ "---\n", "\n", "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n", "\n", "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/aws_sagemaker_studio|getting_started|xgboost_customer_churn_studio.ipynb)\n", "\n", "---" ] }, { "cell_type": "markdown", "id": "2e85a7f0", "metadata": { "papermill": { "duration": 0.024036, "end_time": "2021-05-27T00:07:33.575998", "exception": false, "start_time": "2021-05-27T00:07:33.551962", "status": "completed" }, "tags": [] }, "source": [ "_**Using Gradient Boosted Trees to Predict Mobile Customer Departure**_\n", "\n", "---\n", "\n", "This notebook walks you through some of the main features of Amazon SageMaker Studio. \n", "\n", "* [Amazon SageMaker Experiments](https://docs.aws.amazon.com/sagemaker/latest/dg/experiments.html)\n", " * Manage multiple trials\n", " * Experiment with hyperparameters and charting\n", "* [Amazon SageMaker Debugger](https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html)\n", " * Debug your model \n", "* [Model hosting](https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-hosting.html)\n", " * Set up a persistent endpoint to get predictions from your model\n", "* [SageMaker Model Monitor](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor.html)\n", " * Monitor the quality of your model\n", " * Set alerts for when model quality deviates\n", " \n", "Run this notebook from within Studio. For Studio onboarding and set up instructions, see [README](README.md).\n", "\n", "---\n", "\n", "## Contents\n", "\n", "1. [Background](#Background) - Predicting customer churn with XGBoost\n", "1. [Data](#Data) - Prep the dataset and upload it to Amazon S3\n", "1. [Train](#Train) - Train with the Amazon SageMaker XGBoost algorithm\n", " - [Amazon SageMaker Experiments](#Amazon-SageMaker-Experiments)\n", " - [Amazon SageMaker Debugger](#Amazon-SageMaker-Debugger)\n", "1. [Host](#Host)\n", "1. [SageMaker Model Monitor](#SageMaker-Model-Monitor)\n", "\n", "---\n", "\n", "## Background\n", "\n", "_This notebook has been adapted from an [AWS blog post](https://aws.amazon.com/blogs/ai/predicting-customer-churn-with-amazon-machine-learning/). \n", "\n", "Losing customers is costly for any business. Identifying unhappy customers early on gives you a chance to offer them incentives to stay. This notebook describes using machine learning (ML) for automated identification of unhappy customers, also known as customer churn prediction. It uses Amazon SageMaker features for managing experiments, training the model, and monitoring the deployed model. \n", "\n", "Let's import the Python libraries we'll need for this exercise." ] }, { "cell_type": "code", "execution_count": 2, "id": "guilty-instrumentation", "metadata": { "execution": { "iopub.execute_input": "2021-05-27T00:07:33.628811Z", "iopub.status.busy": "2021-05-27T00:07:33.628168Z", "iopub.status.idle": "2021-05-27T00:07:51.411063Z", "shell.execute_reply": "2021-05-27T00:07:51.411446Z" }, "papermill": { "duration": 17.811928, "end_time": "2021-05-27T00:07:51.411586", "exception": false, "start_time": "2021-05-27T00:07:33.599658", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[31mERROR: aiobotocore 1.2.1 has requirement botocore<1.19.53,>=1.19.52, but you'll have botocore 1.20.81 which is incompatible.\u001b[0m\n", "Collecting sagemaker-experiments\n", " Downloading sagemaker_experiments-0.1.31-py3-none-any.whl (42 kB)\n", "\u001b[K |████████████████████████████████| 42 kB 1.3 MB/s eta 0:00:011\n", "\u001b[?25hRequirement already satisfied: boto3>=1.16.27 in /opt/conda/lib/python3.7/site-packages (from sagemaker-experiments) (1.17.81)\n", "Requirement already satisfied: botocore<1.21.0,>=1.20.81 in /opt/conda/lib/python3.7/site-packages (from boto3>=1.16.27->sagemaker-experiments) (1.20.81)\n", "Requirement already satisfied: s3transfer<0.5.0,>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from boto3>=1.16.27->sagemaker-experiments) (0.4.2)\n", "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3>=1.16.27->sagemaker-experiments) (0.10.0)\n", "Requirement already satisfied: urllib3<1.27,>=1.25.4 in /opt/conda/lib/python3.7/site-packages (from botocore<1.21.0,>=1.20.81->boto3>=1.16.27->sagemaker-experiments) (1.25.8)\n", "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /opt/conda/lib/python3.7/site-packages (from botocore<1.21.0,>=1.20.81->boto3>=1.16.27->sagemaker-experiments) (2.8.1)\n", "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.21.0,>=1.20.81->boto3>=1.16.27->sagemaker-experiments) (1.14.0)\n", "Installing collected packages: sagemaker-experiments\n", "Successfully installed sagemaker-experiments-0.1.31\n" ] } ], "source": [ "import sys\n", "\n", "!{sys.executable} -m pip install -qU awscli boto3 \"sagemaker>=1.71.0,<2.0.0\"\n", "!{sys.executable} -m pip install sagemaker-experiments" ] }, { "cell_type": "code", "execution_count": 3, "id": "composite-greensboro", "metadata": { "execution": { "iopub.execute_input": "2021-05-27T00:07:51.469204Z", "iopub.status.busy": "2021-05-27T00:07:51.468727Z", "iopub.status.idle": "2021-05-27T00:07:52.602862Z", "shell.execute_reply": "2021-05-27T00:07:52.603249Z" }, "papermill": { "duration": 1.165375, "end_time": "2021-05-27T00:07:52.603385", "exception": false, "start_time": "2021-05-27T00:07:51.438010", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import io\n", "import os\n", "import sys\n", "import time\n", "import json\n", "from IPython.display import display\n", "from time import strftime, gmtime\n", "import boto3\n", "import re\n", "\n", "\n", "import sagemaker\n", "from sagemaker import get_execution_role\n", "from sagemaker.predictor import csv_serializer\n", "from sagemaker.debugger import rule_configs, Rule, DebuggerHookConfig\n", "from sagemaker.model_monitor import DataCaptureConfig, DatasetFormat, DefaultModelMonitor\n", "from sagemaker.s3 import S3Uploader, S3Downloader\n", "\n", "from smexperiments.experiment import Experiment\n", "from smexperiments.trial import Trial\n", "from smexperiments.trial_component import TrialComponent\n", "from smexperiments.tracker import Tracker" ] }, { "cell_type": "code", "execution_count": 4, "id": "fleet-antique", "metadata": { "execution": { "iopub.execute_input": "2021-05-27T00:07:52.666002Z", "iopub.status.busy": "2021-05-27T00:07:52.665428Z", "iopub.status.idle": "2021-05-27T00:07:53.183202Z", "shell.execute_reply": "2021-05-27T00:07:53.182756Z" }, "papermill": { "duration": 0.5543, "end_time": "2021-05-27T00:07:53.183308", "exception": false, "start_time": "2021-05-27T00:07:52.629008", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "sess = boto3.Session()\n", "sm = sess.client(\"sagemaker\")\n", "role = sagemaker.get_execution_role()" ] }, { "cell_type": "markdown", "id": "equivalent-intro", "metadata": { "papermill": { "duration": 0.025411, "end_time": "2021-05-27T00:07:53.234309", "exception": false, "start_time": "2021-05-27T00:07:53.208898", "status": "completed" }, "tags": [] }, "source": [ "---\n", "## Data\n", "\n", "Mobile operators' records show which customers ended up churning and which continued using the service. We can use this historical information to train an ML model that can predict customer churn. After training the model, we can pass the profile information of an arbitrary customer (the same profile information that we used to train the model) to the model, and have the model predict whether this customer will churn.\n", "\n", "The dataset that we use is publicly available and was mentioned in the book [Discovering Knowledge in Data](https://www.amazon.com/dp/0470908742/) by Daniel T. Larose. It's attributed by the author to the University of California Irvine Repository of Machine Learning Datasets. The downloaded and preprocessed dataset is in the `data` folder that accompanies this notebook. It's been split into training and validation datasets. To see how the dataset was preprocessed, see this [XGBoost customer churn notebook that starts with the original dataset](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/introduction_to_applying_machine_learning/xgboost_customer_churn/xgboost_customer_churn.ipynb). \n", "\n", "We'll train on a CSV file without the header. But for now, the following cell uses `pandas` to load some of the data from a version of the training data that has a header. \n", "\n", "Explore the data to see the dataset's features and what data will be used to train a the model." ] }, { "cell_type": "code", "execution_count": 5, "id": "talented-complaint", "metadata": { "execution": { "iopub.execute_input": "2021-05-27T00:07:53.291163Z", "iopub.status.busy": "2021-05-27T00:07:53.290431Z", "iopub.status.idle": "2021-05-27T00:07:53.346672Z", "shell.execute_reply": "2021-05-27T00:07:53.347057Z" }, "papermill": { "duration": 0.08743, "end_time": "2021-05-27T00:07:53.347197", "exception": false, "start_time": "2021-05-27T00:07:53.259767", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[Errno 2] No such file or directory: '/root/amazon-sagemaker-examples/aws_sagemaker_studio/getting_started'\n", "/opt/ml/processing/input\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ChurnAccount LengthVMail MessageDay MinsDay CallsEve MinsEve CallsNight MinsNight CallsIntl MinsIntl CallsCustServ CallsState_AKState_ALState_ARState_AZState_CAState_COState_CTState_DCState_DEState_FLState_GAState_HIState_IAState_IDState_ILState_INState_KSState_KYState_LAState_MAState_MDState_MEState_MIState_MNState_MOState_MSState_MTState_NCState_NDState_NEState_NHState_NJState_NMState_NVState_NYState_OHState_OKState_ORState_PAState_RIState_SCState_SDState_TNState_TXState_UTState_VAState_VTState_WAState_WIState_WVState_WYArea Code_408Area Code_415Area Code_510Int'l Plan_noInt'l Plan_yesVMail Plan_noVMail Plan_yes
001060274.4120198.682160.8626.0310000000000000100000000000000000000000000000000000000011010
10280187.894248.686208.812410.6500000000000000000000000000000000000000000000000000010101010
211480279.3104201.687280.8997.9220000000000000000000000000000000001000000000000000000101010
301320191.9107206.9127272.08812.6210000000000000000000000000000000000000100000000000000011010
409229155.4110188.5104254.91188.0430000000000000000000000000000000100000000000000000000011001
.....................................................................................................................................................................................................................
232801060194.8133213.473190.89211.5700000000000000000000000000000000000000000000000000100011010
232911250143.28088.194233.21358.8740000000000000000000000000010000000000000000000000000011010
233001290143.7114297.898212.68611.4840000000000000000000000000000000000000000000000000010011010
233101590198.8107195.591213.312016.5750000000000000000000000000000000000001000000000000000011010
233209933179.193238.3102165.79610.6120000000000000000000000000000000000000000100000000000101001
\n", "

2333 rows × 70 columns

\n", "
" ], "text/plain": [ " Churn Account Length VMail Message Day Mins Day Calls Eve Mins \\\n", "0 0 106 0 274.4 120 198.6 \n", "1 0 28 0 187.8 94 248.6 \n", "2 1 148 0 279.3 104 201.6 \n", "3 0 132 0 191.9 107 206.9 \n", "4 0 92 29 155.4 110 188.5 \n", "... ... ... ... ... ... ... \n", "2328 0 106 0 194.8 133 213.4 \n", "2329 1 125 0 143.2 80 88.1 \n", "2330 0 129 0 143.7 114 297.8 \n", "2331 0 159 0 198.8 107 195.5 \n", "2332 0 99 33 179.1 93 238.3 \n", "\n", " Eve Calls Night Mins Night Calls Intl Mins Intl Calls \\\n", "0 82 160.8 62 6.0 3 \n", "1 86 208.8 124 10.6 5 \n", "2 87 280.8 99 7.9 2 \n", "3 127 272.0 88 12.6 2 \n", "4 104 254.9 118 8.0 4 \n", "... ... ... ... ... ... \n", "2328 73 190.8 92 11.5 7 \n", "2329 94 233.2 135 8.8 7 \n", "2330 98 212.6 86 11.4 8 \n", "2331 91 213.3 120 16.5 7 \n", "2332 102 165.7 96 10.6 1 \n", "\n", " CustServ Calls State_AK State_AL State_AR State_AZ State_CA \\\n", "0 1 0 0 0 0 0 \n", "1 0 0 0 0 0 0 \n", "2 2 0 0 0 0 0 \n", "3 1 0 0 0 0 0 \n", "4 3 0 0 0 0 0 \n", "... ... ... ... ... ... ... \n", "2328 0 0 0 0 0 0 \n", "2329 4 0 0 0 0 0 \n", "2330 4 0 0 0 0 0 \n", "2331 5 0 0 0 0 0 \n", "2332 2 0 0 0 0 0 \n", "\n", " State_CO State_CT State_DC State_DE State_FL State_GA State_HI \\\n", "0 0 0 0 0 0 0 0 \n", "1 0 0 0 0 0 0 0 \n", "2 0 0 0 0 0 0 0 \n", "3 0 0 0 0 0 0 0 \n", "4 0 0 0 0 0 0 0 \n", "... ... ... ... ... ... ... ... \n", "2328 0 0 0 0 0 0 0 \n", "2329 0 0 0 0 0 0 0 \n", "2330 0 0 0 0 0 0 0 \n", "2331 0 0 0 0 0 0 0 \n", "2332 0 0 0 0 0 0 0 \n", "\n", " State_IA State_ID State_IL State_IN State_KS State_KY State_LA \\\n", "0 0 1 0 0 0 0 0 \n", "1 0 0 0 0 0 0 0 \n", "2 0 0 0 0 0 0 0 \n", "3 0 0 0 0 0 0 0 \n", "4 0 0 0 0 0 0 0 \n", "... ... ... ... ... ... ... ... \n", "2328 0 0 0 0 0 0 0 \n", "2329 0 0 0 0 0 0 0 \n", "2330 0 0 0 0 0 0 0 \n", "2331 0 0 0 0 0 0 0 \n", "2332 0 0 0 0 0 0 0 \n", "\n", " State_MA State_MD State_ME State_MI State_MN State_MO State_MS \\\n", "0 0 0 0 0 0 0 0 \n", "1 0 0 0 0 0 0 0 \n", "2 0 0 0 0 0 0 0 \n", "3 0 0 0 0 0 0 0 \n", "4 0 0 0 0 0 0 0 \n", "... ... ... ... ... ... ... ... \n", "2328 0 0 0 0 0 0 0 \n", "2329 0 0 0 0 0 0 0 \n", "2330 0 0 0 0 0 0 0 \n", "2331 0 0 0 0 0 0 0 \n", "2332 0 0 0 0 0 0 0 \n", "\n", " State_MT State_NC State_ND State_NE State_NH State_NJ State_NM \\\n", "0 0 0 0 0 0 0 0 \n", "1 0 0 0 0 0 0 0 \n", "2 0 0 0 0 0 0 0 \n", "3 0 0 0 0 0 0 0 \n", "4 0 0 0 0 0 1 0 \n", "... ... ... ... ... ... ... ... \n", "2328 0 0 0 0 0 0 0 \n", "2329 1 0 0 0 0 0 0 \n", "2330 0 0 0 0 0 0 0 \n", "2331 0 0 0 0 0 0 0 \n", "2332 0 0 0 0 0 0 0 \n", "\n", " State_NV State_NY State_OH State_OK State_OR State_PA State_RI \\\n", "0 0 0 0 0 0 0 0 \n", "1 0 0 0 0 0 0 0 \n", "2 1 0 0 0 0 0 0 \n", "3 0 0 0 0 1 0 0 \n", "4 0 0 0 0 0 0 0 \n", "... ... ... ... ... ... ... ... \n", "2328 0 0 0 0 0 0 0 \n", "2329 0 0 0 0 0 0 0 \n", "2330 0 0 0 0 0 0 0 \n", "2331 0 0 0 1 0 0 0 \n", "2332 0 0 0 0 0 0 0 \n", "\n", " State_SC State_SD State_TN State_TX State_UT State_VA State_VT \\\n", "0 0 0 0 0 0 0 0 \n", "1 0 0 0 0 0 0 0 \n", "2 0 0 0 0 0 0 0 \n", "3 0 0 0 0 0 0 0 \n", "4 0 0 0 0 0 0 0 \n", "... ... ... ... ... ... ... ... \n", "2328 0 0 0 0 0 0 0 \n", "2329 0 0 0 0 0 0 0 \n", "2330 0 0 0 0 0 0 0 \n", "2331 0 0 0 0 0 0 0 \n", "2332 1 0 0 0 0 0 0 \n", "\n", " State_WA State_WI State_WV State_WY Area Code_408 Area Code_415 \\\n", "0 0 0 0 0 0 0 \n", "1 0 0 0 1 0 1 \n", "2 0 0 0 0 0 1 \n", "3 0 0 0 0 0 0 \n", "4 0 0 0 0 0 0 \n", "... ... ... ... ... ... ... \n", "2328 0 0 1 0 0 0 \n", "2329 0 0 0 0 0 0 \n", "2330 0 0 0 1 0 0 \n", "2331 0 0 0 0 0 0 \n", "2332 0 0 0 0 0 1 \n", "\n", " Area Code_510 Int'l Plan_no Int'l Plan_yes VMail Plan_no \\\n", "0 1 1 0 1 \n", "1 0 1 0 1 \n", "2 0 1 0 1 \n", "3 1 1 0 1 \n", "4 1 1 0 0 \n", "... ... ... ... ... \n", "2328 1 1 0 1 \n", "2329 1 1 0 1 \n", "2330 1 1 0 1 \n", "2331 1 1 0 1 \n", "2332 0 1 0 0 \n", "\n", " VMail Plan_yes \n", "0 0 \n", "1 0 \n", "2 0 \n", "3 0 \n", "4 1 \n", "... ... \n", "2328 0 \n", "2329 0 \n", "2330 0 \n", "2331 0 \n", "2332 1 \n", "\n", "[2333 rows x 70 columns]" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Set the path we can find the data files that go with this notebook\n", "%cd /root/amazon-sagemaker-examples/aws_sagemaker_studio/getting_started\n", "local_data_path = \"./data/training-dataset-with-header.csv\"\n", "data = pd.read_csv(local_data_path)\n", "pd.set_option(\"display.max_columns\", 500) # Make sure we can see all of the columns\n", "pd.set_option(\"display.max_rows\", 10) # Keep the output on one page\n", "data" ] }, { "cell_type": "markdown", "id": "spanish-updating", "metadata": { "papermill": { "duration": 0.02681, "end_time": "2021-05-27T00:07:53.404436", "exception": false, "start_time": "2021-05-27T00:07:53.377626", "status": "completed" }, "tags": [] }, "source": [ "Now we'll upload the files to S3 for training but first we will create an S3 bucket for the data if one does not already exist." ] }, { "cell_type": "code", "execution_count": 6, "id": "twelve-platinum", "metadata": { "execution": { "iopub.execute_input": "2021-05-27T00:07:53.464940Z", "iopub.status.busy": "2021-05-27T00:07:53.464212Z", "iopub.status.idle": "2021-05-27T00:07:54.431426Z", "shell.execute_reply": "2021-05-27T00:07:54.430961Z" }, "papermill": { "duration": 1.000089, "end_time": "2021-05-27T00:07:54.431537", "exception": false, "start_time": "2021-05-27T00:07:53.431448", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Looks like you already have a bucket of this name. That's good. Uploading the data files...\n", "s3://sagemaker-studio-us-west-2-521695447989/xgboost-churn/train/train.csv\n", "s3://sagemaker-studio-us-west-2-521695447989/xgboost-churn/validation/validation.csv\n" ] } ], "source": [ "account_id = sess.client(\"sts\", region_name=sess.region_name).get_caller_identity()[\"Account\"]\n", "bucket = \"sagemaker-studio-{}-{}\".format(sess.region_name, account_id)\n", "prefix = \"xgboost-churn\"\n", "\n", "try:\n", " if sess.region_name == \"us-east-1\":\n", " sess.client(\"s3\").create_bucket(Bucket=bucket)\n", " else:\n", " sess.client(\"s3\").create_bucket(\n", " Bucket=bucket, CreateBucketConfiguration={\"LocationConstraint\": sess.region_name}\n", " )\n", "except Exception as e:\n", " print(\n", " \"Looks like you already have a bucket of this name. That's good. Uploading the data files...\"\n", " )\n", "\n", "# Return the URLs of the uploaded file, so they can be reviewed or used elsewhere\n", "s3url = S3Uploader.upload(\"data/train.csv\", \"s3://{}/{}/{}\".format(bucket, prefix, \"train\"))\n", "print(s3url)\n", "s3url = S3Uploader.upload(\n", " \"data/validation.csv\", \"s3://{}/{}/{}\".format(bucket, prefix, \"validation\")\n", ")\n", "print(s3url)" ] }, { "cell_type": "markdown", "id": "veterinary-spirit", "metadata": { "papermill": { "duration": 0.027816, "end_time": "2021-05-27T00:07:54.487118", "exception": false, "start_time": "2021-05-27T00:07:54.459302", "status": "completed" }, "tags": [] }, "source": [ "---\n", "## Train\n", "\n", "Let's move on to training. We'll training a class of models known as gradient boosted decision trees on the data that we just uploaded using the XGBoost library. \n", "\n", "Because we're using XGBoost, we'll first need to specify the locations of the XGBoost algorithm containers." ] }, { "cell_type": "code", "execution_count": 7, "id": "cosmetic-handle", "metadata": { "execution": { "iopub.execute_input": "2021-05-27T00:07:54.554168Z", "iopub.status.busy": "2021-05-27T00:07:54.553499Z", "iopub.status.idle": "2021-05-27T00:07:54.564211Z", "shell.execute_reply": "2021-05-27T00:07:54.563822Z" }, "papermill": { "duration": 0.049606, "end_time": "2021-05-27T00:07:54.564316", "exception": false, "start_time": "2021-05-27T00:07:54.514710", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:sagemaker.amazon.amazon_estimator:'get_image_uri' method will be deprecated in favor of 'ImageURIProvider' class in SageMaker Python SDK v2.\n", "WARNING:sagemaker.amazon.amazon_estimator:There is a more up to date SageMaker XGBoost image. To use the newer image, please set 'repo_version'='1.0-1'. For example:\n", "\tget_image_uri(region, 'xgboost', '1.0-1').\n" ] } ], "source": [ "from sagemaker.amazon.amazon_estimator import get_image_uri\n", "\n", "docker_image_name = get_image_uri(boto3.Session().region_name, \"xgboost\", repo_version=\"0.90-2\")" ] }, { "cell_type": "markdown", "id": "assisted-invalid", "metadata": { "papermill": { "duration": 0.028221, "end_time": "2021-05-27T00:07:54.621073", "exception": false, "start_time": "2021-05-27T00:07:54.592852", "status": "completed" }, "tags": [] }, "source": [ "Then, because we're training with the CSV file format, we'll create `s3_input`s that our training function can use as a pointer to the files in S3." ] }, { "cell_type": "code", "execution_count": 8, "id": "interested-crawford", "metadata": { "execution": { "iopub.execute_input": "2021-05-27T00:07:54.681799Z", "iopub.status.busy": "2021-05-27T00:07:54.681154Z", "iopub.status.idle": "2021-05-27T00:07:54.684843Z", "shell.execute_reply": "2021-05-27T00:07:54.684412Z" }, "papermill": { "duration": 0.0355, "end_time": "2021-05-27T00:07:54.684943", "exception": false, "start_time": "2021-05-27T00:07:54.649443", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:sagemaker:'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.\n", "WARNING:sagemaker:'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.\n" ] } ], "source": [ "s3_input_train = sagemaker.s3_input(\n", " s3_data=\"s3://{}/{}/train\".format(bucket, prefix), content_type=\"csv\"\n", ")\n", "s3_input_validation = sagemaker.s3_input(\n", " s3_data=\"s3://{}/{}/validation/\".format(bucket, prefix), content_type=\"csv\"\n", ")" ] }, { "cell_type": "markdown", "id": "ready-platinum", "metadata": { "papermill": { "duration": 0.028768, "end_time": "2021-05-27T00:07:54.742176", "exception": false, "start_time": "2021-05-27T00:07:54.713408", "status": "completed" }, "tags": [] }, "source": [ "### Amazon SageMaker Experiments\n", "\n", "Amazon SageMaker Experiments allows us to keep track of model training; organize related models together; and log model configuration, parameters, and metrics so we can reproduce and iterate on previously trained models and compare models. We'll create a single experiment to keep track of the different approaches to training the model that we'll try.\n", "\n", "Each approach or block of training code we run will be an experiment trial. Later, we'll compare different trials in Studio.\n", "\n", "Let's create the experiment.\n", "\n" ] }, { "cell_type": "code", "execution_count": 9, "id": "early-aurora", "metadata": { "execution": { "iopub.execute_input": "2021-05-27T00:07:54.811129Z", "iopub.status.busy": "2021-05-27T00:07:54.810422Z", "iopub.status.idle": "2021-05-27T00:07:54.946604Z", "shell.execute_reply": "2021-05-27T00:07:54.946983Z" }, "papermill": { "duration": 0.176423, "end_time": "2021-05-27T00:07:54.947121", "exception": false, "start_time": "2021-05-27T00:07:54.770698", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "sess = sagemaker.session.Session()\n", "\n", "create_date = strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n", "customer_churn_experiment = Experiment.create(\n", " experiment_name=\"customer-churn-prediction-xgboost-{}\".format(create_date),\n", " description=\"Using xgboost to predict customer churn\",\n", " sagemaker_boto_client=boto3.client(\"sagemaker\"),\n", ")" ] }, { "cell_type": "markdown", "id": "crazy-verification", "metadata": { "papermill": { "duration": 0.028606, "end_time": "2021-05-27T00:07:55.004437", "exception": false, "start_time": "2021-05-27T00:07:54.975831", "status": "completed" }, "tags": [] }, "source": [ "#### Hyperparameters\n", "Now, we can specify our XGBoost hyperparameters, including the following key hyperparameters:\n", "- `max_depth` Controls how deep each tree within the algorithm can be built. Deeper trees can lead to better fit, but are more computationally expensive and can lead to overfitting. Typically, you need to explore trade-offs in model performance between using a large number of shallow trees and a smaller number of deeper trees.\n", "- `subsample` Controls training data sampling. This hyperparameter can help reduce overfitting, but setting it too low can also starve the model of data.\n", "- `num_round` Controls the number of boosting rounds. This is essentially the number of subsequent models that are trained using the residuals of previous iterations. More rounds can produce a better fit on the training data, but can be computationally expensive or lead to overfitting.\n", "- `eta` Controls how aggressive each round of boosting is. Larger values lead to more conservative boosting.\n", "- `gamma` Controls how aggressively trees are grown. Larger values lead to more conservative models.\n", "- `min_child_weight` Also controls how aggresively trees are grown. Large values lead to a more conservative algorithm.\n", "\n", "For more details, see [XGBoost's hyperparameters GitHub page](https://github.com/dmlc/xgboost/blob/master/doc/parameter.md)." ] }, { "cell_type": "code", "execution_count": 10, "id": "electronic-county", "metadata": { "execution": { "iopub.execute_input": "2021-05-27T00:07:55.065035Z", "iopub.status.busy": "2021-05-27T00:07:55.064567Z", "iopub.status.idle": "2021-05-27T00:07:55.066351Z", "shell.execute_reply": "2021-05-27T00:07:55.066703Z" }, "papermill": { "duration": 0.0342, "end_time": "2021-05-27T00:07:55.066829", "exception": false, "start_time": "2021-05-27T00:07:55.032629", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "hyperparams = {\n", " \"max_depth\": 5,\n", " \"subsample\": 0.8,\n", " \"num_round\": 600,\n", " \"eta\": 0.2,\n", " \"gamma\": 4,\n", " \"min_child_weight\": 6,\n", " \"silent\": 0,\n", " \"objective\": \"binary:logistic\",\n", "}" ] }, { "cell_type": "markdown", "id": "requested-triple", "metadata": { "papermill": { "duration": 0.028368, "end_time": "2021-05-27T00:07:55.123438", "exception": false, "start_time": "2021-05-27T00:07:55.095070", "status": "completed" }, "tags": [] }, "source": [ "#### Trial 1 - XGBoost in algorithm mode\n", "\n", "For our first trial, we'll use the built-in xgboost container to train a model without providing additional code. This way, we can use XGBoost to train and deploy a model as we would with other Amazon SageMaker built-in algorithms.\n", "\n", "We'll create a new `trial` object for this trial and associate the trial with the experiment that we created earlier. To train the model, we'll create an estimator and specify a few parameters, such as the type of training instances we'd like to use and how many and where to store the trained model artifacts. \n", "\n", "We'll also associate the training job with the experiment trial that we just created (when we call `estimator.fit`)." ] }, { "cell_type": "code", "execution_count": 11, "id": "elect-nothing", "metadata": { "collapsed": true, "execution": { "iopub.execute_input": "2021-05-27T00:07:55.190233Z", "iopub.status.busy": "2021-05-27T00:07:55.189785Z", "iopub.status.idle": "2021-05-27T00:13:38.067349Z", "shell.execute_reply": "2021-05-27T00:13:38.066945Z" }, "jupyter": { "outputs_hidden": true }, "papermill": { "duration": 342.915799, "end_time": "2021-05-27T00:13:38.067461", "exception": false, "start_time": "2021-05-27T00:07:55.151662", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:sagemaker.estimator:Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.\n", "INFO:sagemaker:Creating training-job with name: demo-xgboost-customer-churn-2021-05-27-00-07-55-255\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2021-05-27 00:07:55 Starting - Starting the training job...\n", "2021-05-27 00:07:58 Starting - Launching requested ML instances......\n", "2021-05-27 00:09:10 Starting - Preparing the instances for training.........\n", "2021-05-27 00:10:30 Downloading - Downloading input data......\n", "2021-05-27 00:11:49 Training - Downloading the training image.........\n", "2021-05-27 00:13:25 Uploading - Uploading generated training model\n", "2021-05-27 00:13:25 Completed - Training job completed\n", "\u001b[34mINFO:sagemaker-containers:Imported framework sagemaker_xgboost_container.training\u001b[0m\n", "\u001b[34mINFO:sagemaker-containers:Failed to parse hyperparameter objective value binary:logistic to Json.\u001b[0m\n", "\u001b[34mReturning the value itself\u001b[0m\n", "\u001b[34mINFO:sagemaker-containers:No GPUs detected (normal if no gpus installed)\u001b[0m\n", "\u001b[34mINFO:sagemaker_xgboost_container.training:Running XGBoost Sagemaker in algorithm mode\u001b[0m\n", "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n", "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n", "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n", "\u001b[34m[00:13:11] 2333x69 matrix with 160977 entries loaded from /opt/ml/input/data/train?format=csv&label_column=0&delimiter=,\u001b[0m\n", "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n", "\u001b[34m[00:13:11] 666x69 matrix with 45954 entries loaded from /opt/ml/input/data/validation?format=csv&label_column=0&delimiter=,\u001b[0m\n", "\u001b[34mINFO:root:Single node training.\u001b[0m\n", "\u001b[34mINFO:root:Train matrix has 2333 rows\u001b[0m\n", "\u001b[34mINFO:root:Validation matrix has 666 rows\u001b[0m\n", "\u001b[34m[0]#011train-error:0.077154#011validation-error:0.099099\u001b[0m\n", "\u001b[34m[1]#011train-error:0.050579#011validation-error:0.081081\u001b[0m\n", "\u001b[34m[2]#011train-error:0.048864#011validation-error:0.075075\u001b[0m\n", "\u001b[34m[3]#011train-error:0.046721#011validation-error:0.072072\u001b[0m\n", "\u001b[34m[4]#011train-error:0.048007#011validation-error:0.073574\u001b[0m\n", "\u001b[34m[5]#011train-error:0.046721#011validation-error:0.070571\u001b[0m\n", "\u001b[34m[6]#011train-error:0.045435#011validation-error:0.073574\u001b[0m\n", "\u001b[34m[7]#011train-error:0.043721#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[8]#011train-error:0.045006#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[9]#011train-error:0.042435#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[10]#011train-error:0.040291#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[11]#011train-error:0.039006#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[12]#011train-error:0.038577#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[13]#011train-error:0.03772#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[14]#011train-error:0.03772#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[15]#011train-error:0.039434#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[16]#011train-error:0.038577#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[17]#011train-error:0.03772#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[18]#011train-error:0.039434#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[19]#011train-error:0.039863#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[20]#011train-error:0.039434#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[21]#011train-error:0.038577#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[22]#011train-error:0.038148#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[23]#011train-error:0.036862#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[24]#011train-error:0.036005#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[25]#011train-error:0.034291#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[26]#011train-error:0.033862#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[27]#011train-error:0.033862#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[28]#011train-error:0.033862#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[29]#011train-error:0.033862#011validation-error:0.072072\u001b[0m\n", "\u001b[34m[30]#011train-error:0.033862#011validation-error:0.070571\u001b[0m\n", "\u001b[34m[31]#011train-error:0.034291#011validation-error:0.072072\u001b[0m\n", "\u001b[34m[32]#011train-error:0.034719#011validation-error:0.070571\u001b[0m\n", "\u001b[34m[33]#011train-error:0.034719#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[34]#011train-error:0.033005#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[35]#011train-error:0.033862#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[36]#011train-error:0.033862#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[37]#011train-error:0.033005#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[38]#011train-error:0.033433#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[39]#011train-error:0.033005#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[40]#011train-error:0.031719#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[41]#011train-error:0.031719#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[42]#011train-error:0.030862#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[43]#011train-error:0.03129#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[44]#011train-error:0.030862#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[45]#011train-error:0.03129#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[46]#011train-error:0.030862#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[47]#011train-error:0.030862#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[48]#011train-error:0.030433#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[49]#011train-error:0.030004#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[50]#011train-error:0.030004#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[51]#011train-error:0.030004#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[52]#011train-error:0.030004#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[53]#011train-error:0.030004#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[54]#011train-error:0.030004#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[55]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[56]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[57]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[58]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[59]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[60]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[61]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[62]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[63]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[64]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[65]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[66]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[67]#011train-error:0.030433#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[68]#011train-error:0.029147#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[69]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[70]#011train-error:0.029147#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[71]#011train-error:0.028718#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[72]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[73]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[74]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[75]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[76]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[77]#011train-error:0.029147#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[78]#011train-error:0.029576#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[79]#011train-error:0.030004#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[80]#011train-error:0.029576#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[81]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[82]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[83]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[84]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[85]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[86]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[87]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[88]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[89]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[90]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[91]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[92]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[93]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[94]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[95]#011train-error:0.027861#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[96]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[97]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[98]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[99]#011train-error:0.02829#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[100]#011train-error:0.027432#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[101]#011train-error:0.027432#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[102]#011train-error:0.027432#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[103]#011train-error:0.027432#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[104]#011train-error:0.027432#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[105]#011train-error:0.027861#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[106]#011train-error:0.027861#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[107]#011train-error:0.027861#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[108]#011train-error:0.027861#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[109]#011train-error:0.027861#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[110]#011train-error:0.02829#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[111]#011train-error:0.02829#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[112]#011train-error:0.02829#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[113]#011train-error:0.02829#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[114]#011train-error:0.02829#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[115]#011train-error:0.02829#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[116]#011train-error:0.02829#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[117]#011train-error:0.027432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[118]#011train-error:0.027432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[119]#011train-error:0.027004#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[120]#011train-error:0.027004#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[121]#011train-error:0.027004#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[122]#011train-error:0.027432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[123]#011train-error:0.027432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[124]#011train-error:0.027432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[125]#011train-error:0.027004#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[126]#011train-error:0.027004#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[127]#011train-error:0.027004#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[128]#011train-error:0.027432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[129]#011train-error:0.027432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[130]#011train-error:0.027432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[131]#011train-error:0.027432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[132]#011train-error:0.027432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[133]#011train-error:0.027432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[134]#011train-error:0.027432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[135]#011train-error:0.027432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[136]#011train-error:0.027432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[137]#011train-error:0.027432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[138]#011train-error:0.027432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[139]#011train-error:0.027432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[140]#011train-error:0.027432#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[141]#011train-error:0.027432#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[142]#011train-error:0.027432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[143]#011train-error:0.027432#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[144]#011train-error:0.027432#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[145]#011train-error:0.027432#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[146]#011train-error:0.027432#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[147]#011train-error:0.027004#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[148]#011train-error:0.027004#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[149]#011train-error:0.027004#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[150]#011train-error:0.027004#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[151]#011train-error:0.027004#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[152]#011train-error:0.027432#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[153]#011train-error:0.027004#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[154]#011train-error:0.027432#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[155]#011train-error:0.027004#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[156]#011train-error:0.027004#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[157]#011train-error:0.027004#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[158]#011train-error:0.027004#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[159]#011train-error:0.027004#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[160]#011train-error:0.027004#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[161]#011train-error:0.027004#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[162]#011train-error:0.027004#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[163]#011train-error:0.027004#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[164]#011train-error:0.027004#011validation-error:0.069069\u001b[0m\n", "\u001b[34m[165]#011train-error:0.027432#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[166]#011train-error:0.027432#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[167]#011train-error:0.027432#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[168]#011train-error:0.027432#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[169]#011train-error:0.027432#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[170]#011train-error:0.027432#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[171]#011train-error:0.027861#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[172]#011train-error:0.027861#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[173]#011train-error:0.027432#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[174]#011train-error:0.027432#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[175]#011train-error:0.027432#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[176]#011train-error:0.027432#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[177]#011train-error:0.027432#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[178]#011train-error:0.027432#011validation-error:0.067568\u001b[0m\n", "\u001b[34m[179]#011train-error:0.026575#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[180]#011train-error:0.026575#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[181]#011train-error:0.027004#011validation-error:0.066066\u001b[0m\n", "\u001b[34m[182]#011train-error:0.026575#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[183]#011train-error:0.027004#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[184]#011train-error:0.026147#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[185]#011train-error:0.026147#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[186]#011train-error:0.026147#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[187]#011train-error:0.026147#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[188]#011train-error:0.026147#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[189]#011train-error:0.026147#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[190]#011train-error:0.026147#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[191]#011train-error:0.026147#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[192]#011train-error:0.026147#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[193]#011train-error:0.025289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[194]#011train-error:0.025289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[195]#011train-error:0.025289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[196]#011train-error:0.025289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[197]#011train-error:0.025289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[198]#011train-error:0.025289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[199]#011train-error:0.025289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[200]#011train-error:0.025289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[201]#011train-error:0.025289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[202]#011train-error:0.025289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[203]#011train-error:0.025289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[204]#011train-error:0.025289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[205]#011train-error:0.025289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[206]#011train-error:0.024861#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[207]#011train-error:0.024861#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[208]#011train-error:0.024861#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[209]#011train-error:0.024861#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[210]#011train-error:0.024861#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[211]#011train-error:0.024861#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[212]#011train-error:0.025289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[213]#011train-error:0.025289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[214]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[215]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[216]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[217]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[218]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[219]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[220]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[221]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[222]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[223]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[224]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[225]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[226]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[227]#011train-error:0.025718#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[228]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[229]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[230]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[231]#011train-error:0.025289#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[232]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[233]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[234]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[235]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[236]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[237]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[238]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[239]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[240]#011train-error:0.025289#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[241]#011train-error:0.025289#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[242]#011train-error:0.025289#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[243]#011train-error:0.025289#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[244]#011train-error:0.025289#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[245]#011train-error:0.025289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[246]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[247]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[248]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[249]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[250]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[251]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[252]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[253]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[254]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[255]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[256]#011train-error:0.024432#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[257]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[258]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[259]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[260]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[261]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[262]#011train-error:0.024432#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[263]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[264]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[265]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[266]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[267]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[268]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[269]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[270]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[271]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[272]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[273]#011train-error:0.025289#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[274]#011train-error:0.024861#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[275]#011train-error:0.024432#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[276]#011train-error:0.023575#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[277]#011train-error:0.023575#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[278]#011train-error:0.023575#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[279]#011train-error:0.023575#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[280]#011train-error:0.023575#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[281]#011train-error:0.023575#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[282]#011train-error:0.023575#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[283]#011train-error:0.023575#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[284]#011train-error:0.023575#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[285]#011train-error:0.023575#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[286]#011train-error:0.023575#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[287]#011train-error:0.023575#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[288]#011train-error:0.023575#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[289]#011train-error:0.023575#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[290]#011train-error:0.023575#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[291]#011train-error:0.023575#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[292]#011train-error:0.023575#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[293]#011train-error:0.023575#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[294]#011train-error:0.023575#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[295]#011train-error:0.023575#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[296]#011train-error:0.023575#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[297]#011train-error:0.023575#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[298]#011train-error:0.023575#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[299]#011train-error:0.023575#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[300]#011train-error:0.023575#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[301]#011train-error:0.023575#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[302]#011train-error:0.023575#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[303]#011train-error:0.023575#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[304]#011train-error:0.023146#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[305]#011train-error:0.023146#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[306]#011train-error:0.023146#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[307]#011train-error:0.023575#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[308]#011train-error:0.023575#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[309]#011train-error:0.023575#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[310]#011train-error:0.023575#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[311]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[312]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[313]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[314]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[315]#011train-error:0.022718#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[316]#011train-error:0.022718#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[317]#011train-error:0.022718#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[318]#011train-error:0.022718#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[319]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[320]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[321]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[322]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[323]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[324]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[325]#011train-error:0.022718#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[326]#011train-error:0.022718#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[327]#011train-error:0.022718#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[328]#011train-error:0.022718#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[329]#011train-error:0.022718#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[330]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[331]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[332]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[333]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[334]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[335]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[336]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[337]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[338]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[339]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[340]#011train-error:0.022718#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[341]#011train-error:0.022718#011validation-error:0.064565\u001b[0m\n", "\u001b[34m[342]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[343]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[344]#011train-error:0.022718#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[345]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[346]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[347]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[348]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[349]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[350]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[351]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[352]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[353]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[354]#011train-error:0.022718#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[355]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[356]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[357]#011train-error:0.022718#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[358]#011train-error:0.022289#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[359]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[360]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[361]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[362]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[363]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[364]#011train-error:0.022289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[365]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[366]#011train-error:0.022718#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[367]#011train-error:0.022289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[368]#011train-error:0.022289#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[369]#011train-error:0.022289#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[370]#011train-error:0.022289#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[371]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[372]#011train-error:0.022289#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[373]#011train-error:0.022289#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[374]#011train-error:0.022289#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[375]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[376]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[377]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[378]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[379]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[380]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[381]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[382]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[383]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[384]#011train-error:0.022289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[385]#011train-error:0.022289#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[386]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[387]#011train-error:0.023146#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[388]#011train-error:0.022289#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[389]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[390]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[391]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[392]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[393]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[394]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[395]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[396]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[397]#011train-error:0.022718#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[398]#011train-error:0.022289#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[399]#011train-error:0.022289#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[400]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[401]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[402]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[403]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[404]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[405]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[406]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[407]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[408]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[409]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[410]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[411]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[412]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[413]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[414]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[415]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[416]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[417]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[418]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[419]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[420]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[421]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[422]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[423]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[424]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[425]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[426]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[427]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[428]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[429]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[430]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[431]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[432]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[433]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[434]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[435]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[436]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[437]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[438]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[439]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[440]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[441]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[442]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[443]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[444]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[445]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[446]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[447]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[448]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[449]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[450]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[451]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[452]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[453]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[454]#011train-error:0.021432#011validation-error:0.063063\u001b[0m\n", "\u001b[34m[455]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[456]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[457]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[458]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[459]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[460]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[461]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[462]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[463]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[464]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[465]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[466]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[467]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[468]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[469]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[470]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[471]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[472]#011train-error:0.021432#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[473]#011train-error:0.021432#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[474]#011train-error:0.021432#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[475]#011train-error:0.021432#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[476]#011train-error:0.021432#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[477]#011train-error:0.021432#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[478]#011train-error:0.021432#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[479]#011train-error:0.021432#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[480]#011train-error:0.021432#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[481]#011train-error:0.021432#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[482]#011train-error:0.02186#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[483]#011train-error:0.021432#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[484]#011train-error:0.021003#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[485]#011train-error:0.021003#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[486]#011train-error:0.021432#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[487]#011train-error:0.021432#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[488]#011train-error:0.021003#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[489]#011train-error:0.021432#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[490]#011train-error:0.021003#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[491]#011train-error:0.021003#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[492]#011train-error:0.021003#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[493]#011train-error:0.021003#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[494]#011train-error:0.021003#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[495]#011train-error:0.021003#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[496]#011train-error:0.021003#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[497]#011train-error:0.021003#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[498]#011train-error:0.021003#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[499]#011train-error:0.021003#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[500]#011train-error:0.021003#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[501]#011train-error:0.021003#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[502]#011train-error:0.021003#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[503]#011train-error:0.021003#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[504]#011train-error:0.021003#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[505]#011train-error:0.021003#011validation-error:0.058559\u001b[0m\n", "\u001b[34m[506]#011train-error:0.020574#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[507]#011train-error:0.020574#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[508]#011train-error:0.020574#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[509]#011train-error:0.020574#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[510]#011train-error:0.020574#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[511]#011train-error:0.020574#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[512]#011train-error:0.020574#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[513]#011train-error:0.020574#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[514]#011train-error:0.020574#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[515]#011train-error:0.020574#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[516]#011train-error:0.021003#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[517]#011train-error:0.021003#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[518]#011train-error:0.021003#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[519]#011train-error:0.021003#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[520]#011train-error:0.021003#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[521]#011train-error:0.020574#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[522]#011train-error:0.021003#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[523]#011train-error:0.021003#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[524]#011train-error:0.021003#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[525]#011train-error:0.021003#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[526]#011train-error:0.021003#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[527]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[528]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[529]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[530]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[531]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[532]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[533]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[534]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[535]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[536]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[537]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[538]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[539]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[540]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[541]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[542]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[543]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[544]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[545]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[546]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[547]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[548]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[549]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[550]#011train-error:0.021432#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[551]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[552]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[553]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[554]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[555]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[556]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[557]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[558]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[559]#011train-error:0.021003#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[560]#011train-error:0.021003#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[561]#011train-error:0.021003#011validation-error:0.06006\u001b[0m\n", "\u001b[34m[562]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[563]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[564]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[565]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[566]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[567]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[568]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[569]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[570]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[571]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[572]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[573]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[574]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[575]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[576]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[577]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[578]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[579]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[580]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[581]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[582]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[583]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[584]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[585]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[586]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[587]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[588]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[589]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[590]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[591]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[592]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[593]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[594]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[595]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[596]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[597]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[598]#011train-error:0.020574#011validation-error:0.061562\u001b[0m\n", "\u001b[34m[599]#011train-error:0.021003#011validation-error:0.061562\u001b[0m\n", "Training seconds: 175\n", "Billable seconds: 175\n" ] } ], "source": [ "trial = Trial.create(\n", " trial_name=\"algorithm-mode-trial-{}\".format(strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())),\n", " experiment_name=customer_churn_experiment.experiment_name,\n", " sagemaker_boto_client=boto3.client(\"sagemaker\"),\n", ")\n", "\n", "xgb = sagemaker.estimator.Estimator(\n", " image_name=docker_image_name,\n", " role=role,\n", " hyperparameters=hyperparams,\n", " train_instance_count=1,\n", " train_instance_type=\"ml.m4.xlarge\",\n", " output_path=\"s3://{}/{}/output\".format(bucket, prefix),\n", " base_job_name=\"demo-xgboost-customer-churn\",\n", " sagemaker_session=sess,\n", ")\n", "\n", "xgb.fit(\n", " {\"train\": s3_input_train, \"validation\": s3_input_validation},\n", " experiment_config={\n", " \"ExperimentName\": customer_churn_experiment.experiment_name,\n", " \"TrialName\": trial.trial_name,\n", " \"TrialComponentDisplayName\": \"Training\",\n", " },\n", ")" ] }, { "cell_type": "markdown", "id": "sized-navigation", "metadata": { "papermill": { "duration": 0.036234, "end_time": "2021-05-27T00:13:38.139993", "exception": false, "start_time": "2021-05-27T00:13:38.103759", "status": "completed" }, "tags": [] }, "source": [ "#### Review the results\n", "\n", "After the training job succeeds, you can view metrics, logs, and graphs for the trial on the Experiments tab in Studio. \n", "\n", "To view them, choose the Experiments button.\n", "\n", "![experiments_button](./images/studio_experiments_icon.png)\n", "\n", "To see the components of a specific experiment, in the Experiments list, double-click it. If you want to see the components of multiple experiments, select them with Ctrl-click, then right-click on an experiment to see the context menu. To see all of the compenents together, choose \"Open in trial component list\". This enables charting across experiments.\n", "\n", "![studio_experiments_list](./images/view_multiple_trial_components.gif)\n", "\n", "The components are sorted so that the best model is at the top. \n", "\n", "![studio_trial_list](./images/studio_trial_list.png)\n", "\n", "#### Download the model\n", "\n", "You can also find and download the model that was trained. To find the model, choose the Experiments icon in the left tray, and keep drilling down through the experiment, the most recent trial, and its most recent component until you see the Describe Trial Components page. To see links to the training and validation datasets, choose the Artifacts tab. The links are listed in the \"Input Artifacts\" section. The generated model artifact is in the \"Output Artifacts\" section.\n", "\n", "![find the model artifact](./images/find_the_model_artifact.gif)" ] }, { "cell_type": "markdown", "id": "occupied-parks", "metadata": { "papermill": { "duration": 0.036327, "end_time": "2021-05-27T00:13:38.212645", "exception": false, "start_time": "2021-05-27T00:13:38.176318", "status": "completed" }, "tags": [] }, "source": [ "#### Trying other hyperparameter values\n", "\n", "To improve a model, you typically try other hyperparameter values to see if they affect the final validation error. Let's change the `min_child_weight` value and start other training jobs with those values to see how they affect the validation error. For each `min_child_weight` value, we'll create a separate trial so that we can compare the results in Studio." ] }, { "cell_type": "code", "execution_count": 12, "id": "alpine-destruction", "metadata": { "execution": { "iopub.execute_input": "2021-05-27T00:13:38.295046Z", "iopub.status.busy": "2021-05-27T00:13:38.294525Z", "iopub.status.idle": "2021-05-27T00:13:42.112216Z", "shell.execute_reply": "2021-05-27T00:13:42.112618Z" }, "papermill": { "duration": 3.86392, "end_time": "2021-05-27T00:13:42.112761", "exception": false, "start_time": "2021-05-27T00:13:38.248841", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:sagemaker.estimator:Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.\n", "INFO:sagemaker:Creating training-job with name: demo-xgboost-customer-churn-2021-05-27-00-13-38-354\n", "WARNING:sagemaker.estimator:Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.\n", "INFO:sagemaker:Creating training-job with name: demo-xgboost-customer-churn-2021-05-27-00-13-38-613\n", "WARNING:sagemaker.estimator:Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.\n", "INFO:sagemaker:Creating training-job with name: demo-xgboost-customer-churn-2021-05-27-00-13-39-527\n", "WARNING:sagemaker.estimator:Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.\n", "INFO:sagemaker:Creating training-job with name: demo-xgboost-customer-churn-2021-05-27-00-13-40-588\n", "WARNING:sagemaker.estimator:Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.\n", "INFO:sagemaker:Creating training-job with name: demo-xgboost-customer-churn-2021-05-27-00-13-41-332\n" ] } ], "source": [ "min_child_weights = [1, 2, 4, 8, 10]\n", "\n", "for weight in min_child_weights:\n", " hyperparams[\"min_child_weight\"] = weight\n", " trial = Trial.create(\n", " trial_name=\"algorithm-mode-trial-{}-weight-{}\".format(\n", " strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime()), weight\n", " ),\n", " experiment_name=customer_churn_experiment.experiment_name,\n", " sagemaker_boto_client=boto3.client(\"sagemaker\"),\n", " )\n", "\n", " t_xgb = sagemaker.estimator.Estimator(\n", " image_name=docker_image_name,\n", " role=role,\n", " hyperparameters=hyperparams,\n", " train_instance_count=1,\n", " train_instance_type=\"ml.m4.xlarge\",\n", " output_path=\"s3://{}/{}/output\".format(bucket, prefix),\n", " base_job_name=\"demo-xgboost-customer-churn\",\n", " sagemaker_session=sess,\n", " )\n", "\n", " t_xgb.fit(\n", " {\"train\": s3_input_train, \"validation\": s3_input_validation},\n", " wait=False,\n", " experiment_config={\n", " \"ExperimentName\": customer_churn_experiment.experiment_name,\n", " \"TrialName\": trial.trial_name,\n", " \"TrialComponentDisplayName\": \"Training\",\n", " },\n", " )" ] }, { "cell_type": "markdown", "id": "temporal-question", "metadata": { "papermill": { "duration": 0.037821, "end_time": "2021-05-27T00:13:42.188779", "exception": false, "start_time": "2021-05-27T00:13:42.150958", "status": "completed" }, "tags": [] }, "source": [ "#### Create charts\n", "\n", "To create charts, you multi-select the components. Because this is a sample training and the data is quite sparse, there's not much to chart in this time series. However, you can create a scatter plot for the parameter sweep. The following image is an example.\n", "\n", "![scatter plot example](./images/scatter_plot_example.png)\n", "\n", "##### How to create a scatter plot\n", "\n", "Multi-select the components, then choose \"Add chart\". In the Chart properties panel, for Data type, choose \"Summary Statistics\". For Chart type, choose scatter plot. Then choose the `min_child_wight` hyperparameter as the X-axis (because this is the hyperparameter that you're iterating on in this notebook). For Y-axis metrics, choose either `validation:error_last` or `validation:error_avg`), and then color them by `trialComponentName`.\n", "\n", "![create a scatter plot](./images/create_a_scatter_plot.gif)\n", "\n", "You can also adjust the chart as you go by choosing other components and zooming in and out. Each item on the graph will display contextual info.\n", "\n", "![adjust a scatter plot](./images/adjust_a_scatter_plot.gif)" ] }, { "cell_type": "markdown", "id": "impaired-tender", "metadata": { "papermill": { "duration": 0.03775, "end_time": "2021-05-27T00:13:42.264169", "exception": false, "start_time": "2021-05-27T00:13:42.226419", "status": "completed" }, "tags": [] }, "source": [ "### Amazon SageMaker Debugger\n", "\n", "With Amazon SageMaker Debugger, you can debug models during training. During training, Debugger periodicially saves tensors, which specify the state of the model at that point in time. Debugger saves the tensors to Amazon S3 for analysis and visualization. This allows you to diagnose training issues with Studio.\n", "\n", "#### Specify Debugger rules\n", "\n", "To enable automated detection of common issues during training, you can attach a list of rules to evaluate the training job against.\n", "\n", "Some rule configurationss that apply to XGBoost include `AllZero`, `ClassImbalance`, `Confusion`, `LossNotDecreasing`, `Overfit`, `Overtraining`, `SimilarAcrossRuns`, `TensorVariance`, `UnchangedTensor`, `TreeDepth`. \n", "\n", "Let's use the `LossNotDecreasing` rule--which is triggered if the loss doesn't decrease monotonically at any point during training--the `Overtraining` rule, and the `Overfit` rule." ] }, { "cell_type": "code", "execution_count": 13, "id": "interim-saudi", "metadata": { "execution": { "iopub.execute_input": "2021-05-27T00:13:42.345228Z", "iopub.status.busy": "2021-05-27T00:13:42.344768Z", "iopub.status.idle": "2021-05-27T00:13:42.347134Z", "shell.execute_reply": "2021-05-27T00:13:42.347463Z" }, "papermill": { "duration": 0.045886, "end_time": "2021-05-27T00:13:42.347583", "exception": false, "start_time": "2021-05-27T00:13:42.301697", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "debug_rules = [\n", " Rule.sagemaker(rule_configs.loss_not_decreasing()),\n", " Rule.sagemaker(rule_configs.overtraining()),\n", " Rule.sagemaker(rule_configs.overfit()),\n", "]" ] }, { "cell_type": "markdown", "id": "mediterranean-lottery", "metadata": { "papermill": { "duration": 0.037848, "end_time": "2021-05-27T00:13:42.423203", "exception": false, "start_time": "2021-05-27T00:13:42.385355", "status": "completed" }, "tags": [] }, "source": [ "#### Trial 2 - XGBoost in framework mode\n", "\n", "For the next trial, we'll train a similar model, but we'll use XGBoost in framework mode. If you've worked with the open source XGBoost framework, this way of using XGBoost should be familiar to you. Using XGBoost as a framework provides more flexibility than using it as a built-in algorithm because it enables more advanced scenarios that allow you to incorporate preprocessing and post-processing scripts into your training script. Specifically, we'll be able to specify a list of rules that we want Debugger to evaluate our training process against." ] }, { "cell_type": "markdown", "id": "seeing-association", "metadata": { "papermill": { "duration": 0.037527, "end_time": "2021-05-27T00:13:42.498333", "exception": false, "start_time": "2021-05-27T00:13:42.460806", "status": "completed" }, "tags": [] }, "source": [ "#### Fit estimator\n", "\n", "To use XGBoost as a framework, you need to specify an entry-point script that can incorporate additional processing into your training jobs.\n", "\n", "We've made a couple of simple changes to enable the Debugger `smdebug`. We created a SessionHook which we pass as a callback function when creating a Booster. We passed a SaveConfig object that tells the hook to save the evaluation metrics, feature importances, and SHAP values at regular intervals. Debugger is highly configurable, so you can choose exactly what to save. We describe the changes in more detail after we train this example. For even more detail, see the [Developer Guide for XGBoost](https://github.com/awslabs/sagemaker-debugger/tree/master/docs/xgboost)." ] }, { "cell_type": "code", "execution_count": 14, "id": "reduced-appearance", "metadata": { "execution": { "iopub.execute_input": "2021-05-27T00:13:42.578134Z", "iopub.status.busy": "2021-05-27T00:13:42.577626Z", "iopub.status.idle": "2021-05-27T00:13:43.279105Z", "shell.execute_reply": "2021-05-27T00:13:43.279485Z" }, "papermill": { "duration": 0.743325, "end_time": "2021-05-27T00:13:43.279625", "exception": false, "start_time": "2021-05-27T00:13:42.536300", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36margparse\u001b[39;49;00m\n", "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mjson\u001b[39;49;00m\n", "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mos\u001b[39;49;00m\n", "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mpickle\u001b[39;49;00m\n", "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mrandom\u001b[39;49;00m\n", "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mtempfile\u001b[39;49;00m\n", "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36murllib\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mrequest\u001b[39;49;00m\n", "\n", "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mxgboost\u001b[39;49;00m\n", "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36msmdebug\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m SaveConfig\n", "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36msmdebug\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mxgboost\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m Hook\n", "\n", "\n", "\u001b[34mdef\u001b[39;49;00m \u001b[32mparse_args\u001b[39;49;00m():\n", "\n", " parser = argparse.ArgumentParser()\n", "\n", " parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--max_depth\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, default=\u001b[34m5\u001b[39;49;00m)\n", " parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--eta\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mfloat\u001b[39;49;00m, default=\u001b[34m0.2\u001b[39;49;00m)\n", " parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--gamma\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, default=\u001b[34m4\u001b[39;49;00m)\n", " parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--min_child_weight\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, default=\u001b[34m6\u001b[39;49;00m)\n", " parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--subsample\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mfloat\u001b[39;49;00m, default=\u001b[34m0.8\u001b[39;49;00m)\n", " parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--silent\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, default=\u001b[34m0\u001b[39;49;00m)\n", " parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--objective\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=\u001b[33m\"\u001b[39;49;00m\u001b[33mbinary:logistic\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n", " parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--num_round\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, default=\u001b[34m50\u001b[39;49;00m)\n", " parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--smdebug_path\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=\u001b[34mNone\u001b[39;49;00m)\n", " parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--smdebug_frequency\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, default=\u001b[34m1\u001b[39;49;00m)\n", " parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--smdebug_collections\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=\u001b[33m\"\u001b[39;49;00m\u001b[33mmetrics\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n", " parser.add_argument(\n", " \u001b[33m\"\u001b[39;49;00m\u001b[33m--output_uri\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n", " \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m,\n", " default=\u001b[33m\"\u001b[39;49;00m\u001b[33m/opt/ml/output/tensors\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n", " help=\u001b[33m\"\u001b[39;49;00m\u001b[33mS3 URI of the bucket where tensor data will be stored.\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n", " )\n", "\n", " parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--train\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=os.environ.get(\u001b[33m\"\u001b[39;49;00m\u001b[33mSM_CHANNEL_TRAIN\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m))\n", " parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--validation\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=os.environ.get(\u001b[33m\"\u001b[39;49;00m\u001b[33mSM_CHANNEL_VALIDATION\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m))\n", " parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--model-dir\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=os.environ[\u001b[33m\"\u001b[39;49;00m\u001b[33mSM_MODEL_DIR\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m])\n", "\n", " args = parser.parse_args()\n", "\n", " \u001b[34mreturn\u001b[39;49;00m args\n", "\n", "\n", "\u001b[34mdef\u001b[39;49;00m \u001b[32mcreate_smdebug_hook\u001b[39;49;00m(\n", " out_dir,\n", " train_data=\u001b[34mNone\u001b[39;49;00m,\n", " validation_data=\u001b[34mNone\u001b[39;49;00m,\n", " frequency=\u001b[34m1\u001b[39;49;00m,\n", " collections=\u001b[34mNone\u001b[39;49;00m,\n", "):\n", "\n", " save_config = SaveConfig(save_interval=frequency)\n", " hook = Hook(\n", " out_dir=out_dir,\n", " train_data=train_data,\n", " validation_data=validation_data,\n", " save_config=save_config,\n", " include_collections=collections,\n", " )\n", "\n", " \u001b[34mreturn\u001b[39;49;00m hook\n", "\n", "\n", "\u001b[34mdef\u001b[39;49;00m \u001b[32mmain\u001b[39;49;00m():\n", "\n", " args = parse_args()\n", "\n", " train, validation = args.train, args.validation\n", " parse_csv = \u001b[33m\"\u001b[39;49;00m\u001b[33m?format=csv&label_column=0\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\n", " dtrain = xgboost.DMatrix(train + parse_csv)\n", " dval = xgboost.DMatrix(validation + parse_csv)\n", "\n", " watchlist = [(dtrain, \u001b[33m\"\u001b[39;49;00m\u001b[33mtrain\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m), (dval, \u001b[33m\"\u001b[39;49;00m\u001b[33mvalidation\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)]\n", "\n", " params = {\n", " \u001b[33m\"\u001b[39;49;00m\u001b[33mmax_depth\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: args.max_depth,\n", " \u001b[33m\"\u001b[39;49;00m\u001b[33meta\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: args.eta,\n", " \u001b[33m\"\u001b[39;49;00m\u001b[33mgamma\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: args.gamma,\n", " \u001b[33m\"\u001b[39;49;00m\u001b[33mmin_child_weight\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: args.min_child_weight,\n", " \u001b[33m\"\u001b[39;49;00m\u001b[33msubsample\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: args.subsample,\n", " \u001b[33m\"\u001b[39;49;00m\u001b[33msilent\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: args.silent,\n", " \u001b[33m\"\u001b[39;49;00m\u001b[33mobjective\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: args.objective,\n", " }\n", "\n", " \u001b[37m# The output_uri is a the URI for the s3 bucket where the metrics will be\u001b[39;49;00m\n", " \u001b[37m# saved.\u001b[39;49;00m\n", " output_uri = args.smdebug_path \u001b[34mif\u001b[39;49;00m args.smdebug_path \u001b[35mis\u001b[39;49;00m \u001b[35mnot\u001b[39;49;00m \u001b[34mNone\u001b[39;49;00m \u001b[34melse\u001b[39;49;00m args.output_uri\n", "\n", " collections = (\n", " args.smdebug_collections.split(\u001b[33m\"\u001b[39;49;00m\u001b[33m,\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m) \u001b[34mif\u001b[39;49;00m args.smdebug_collections \u001b[35mis\u001b[39;49;00m \u001b[35mnot\u001b[39;49;00m \u001b[34mNone\u001b[39;49;00m \u001b[34melse\u001b[39;49;00m \u001b[34mNone\u001b[39;49;00m\n", " )\n", "\n", " hook = create_smdebug_hook(\n", " out_dir=output_uri,\n", " frequency=args.smdebug_frequency,\n", " collections=collections,\n", " train_data=dtrain,\n", " validation_data=dval,\n", " )\n", "\n", " bst = xgboost.train(\n", " params=params,\n", " dtrain=dtrain,\n", " evals=watchlist,\n", " num_boost_round=args.num_round,\n", " callbacks=[hook],\n", " )\n", "\n", " \u001b[34mif\u001b[39;49;00m \u001b[35mnot\u001b[39;49;00m os.path.exists(args.model_dir):\n", " os.makedirs(args.model_dir)\n", "\n", " model_location = os.path.join(args.model_dir, \u001b[33m\"\u001b[39;49;00m\u001b[33mxgboost-model\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n", " pickle.dump(bst, \u001b[36mopen\u001b[39;49;00m(model_location, \u001b[33m\"\u001b[39;49;00m\u001b[33mwb\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m))\n", "\n", "\n", "\u001b[34mif\u001b[39;49;00m \u001b[31m__name__\u001b[39;49;00m == \u001b[33m\"\u001b[39;49;00m\u001b[33m__main__\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\n", "\n", " main()\n", "\n", "\n", "\u001b[34mdef\u001b[39;49;00m \u001b[32mmodel_fn\u001b[39;49;00m(model_dir):\n", " \u001b[33m\"\"\"Load a model. For XGBoost Framework, a default function to load a model is not provided.\u001b[39;49;00m\n", "\u001b[33m Users should provide customized model_fn() in script.\u001b[39;49;00m\n", "\u001b[33m Args:\u001b[39;49;00m\n", "\u001b[33m model_dir: a directory where model is saved.\u001b[39;49;00m\n", "\u001b[33m Returns:\u001b[39;49;00m\n", "\u001b[33m A XGBoost model.\u001b[39;49;00m\n", "\u001b[33m XGBoost model format type.\u001b[39;49;00m\n", "\u001b[33m \"\"\"\u001b[39;49;00m\n", " model_files = (\n", " file \u001b[34mfor\u001b[39;49;00m file \u001b[35min\u001b[39;49;00m os.listdir(model_dir) \u001b[34mif\u001b[39;49;00m os.path.isfile(os.path.join(model_dir, file))\n", " )\n", " model_file = \u001b[36mnext\u001b[39;49;00m(model_files)\n", " \u001b[34mtry\u001b[39;49;00m:\n", " booster = pickle.load(\u001b[36mopen\u001b[39;49;00m(os.path.join(model_dir, model_file), \u001b[33m\"\u001b[39;49;00m\u001b[33mrb\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m))\n", " \u001b[36mformat\u001b[39;49;00m = \u001b[33m\"\u001b[39;49;00m\u001b[33mpkl_format\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\n", " \u001b[34mexcept\u001b[39;49;00m \u001b[36mException\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m exp_pkl:\n", " \u001b[34mtry\u001b[39;49;00m:\n", " booster = xgboost.Booster()\n", " booster.load_model(os.path.join(model_dir, model_file))\n", " \u001b[36mformat\u001b[39;49;00m = \u001b[33m\"\u001b[39;49;00m\u001b[33mxgb_format\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\n", " \u001b[34mexcept\u001b[39;49;00m \u001b[36mException\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m exp_xgb:\n", " \u001b[34mraise\u001b[39;49;00m ModelLoadInferenceError(\n", " \u001b[33m\"\u001b[39;49;00m\u001b[33mUnable to load model: \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m.format(\u001b[36mstr\u001b[39;49;00m(exp_pkl), \u001b[36mstr\u001b[39;49;00m(exp_xgb))\n", " )\n", " booster.set_param(\u001b[33m\"\u001b[39;49;00m\u001b[33mnthread\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[34m1\u001b[39;49;00m)\n", " \u001b[34mreturn\u001b[39;49;00m booster, \u001b[36mformat\u001b[39;49;00m\n" ] } ], "source": [ "!pygmentize xgboost_customer_churn.py" ] }, { "cell_type": "markdown", "id": "discrete-romance", "metadata": { "papermill": { "duration": 0.03849, "end_time": "2021-05-27T00:13:43.357822", "exception": false, "start_time": "2021-05-27T00:13:43.319332", "status": "completed" }, "tags": [] }, "source": [ "Let's create our framwork estimator and call `fit` to start the training job. As before, we'll create a separate trial for this run so that we can use Studio to compare it with other trials. Because we are running in framework mode, we also need to pass additional parameters, like the entry point script and the framework version, to the estimator. \n", "\n", "As training progresses, you can see logs as Debugger evaluates the rule against the training job." ] }, { "cell_type": "code", "execution_count": null, "id": "informed-collect", "metadata": { "execution": { "iopub.execute_input": "2021-05-27T00:13:43.446531Z", "iopub.status.busy": "2021-05-27T00:13:43.445782Z", "iopub.status.idle": "2021-05-27T00:13:50.817898Z", "shell.execute_reply": "2021-05-27T00:13:50.817144Z" }, "papermill": { "duration": 7.421699, "end_time": "2021-05-27T00:13:50.818092", "exception": true, "start_time": "2021-05-27T00:13:43.396393", "status": "failed" }, "tags": [] }, "outputs": [], "source": [ "entry_point_script = \"xgboost_customer_churn.py\"\n", "\n", "trial = Trial.create(\n", " trial_name=\"framework-mode-trial-{}\".format(strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())),\n", " experiment_name=customer_churn_experiment.experiment_name,\n", " sagemaker_boto_client=boto3.client(\"sagemaker\"),\n", ")\n", "\n", "framework_xgb = sagemaker.xgboost.XGBoost(\n", " image_name=docker_image_name,\n", " entry_point=entry_point_script,\n", " role=role,\n", " framework_version=\"0.90-2\",\n", " py_version=\"py3\",\n", " hyperparameters=hyperparams,\n", " train_instance_count=1,\n", " train_instance_type=\"ml.m4.xlarge\",\n", " output_path=\"s3://{}/{}/output\".format(bucket, prefix),\n", " base_job_name=\"demo-xgboost-customer-churn\",\n", " sagemaker_session=sess,\n", " rules=debug_rules,\n", ")\n", "\n", "framework_xgb.fit(\n", " {\"train\": s3_input_train, \"validation\": s3_input_validation},\n", " experiment_config={\n", " \"ExperimentName\": customer_churn_experiment.experiment_name,\n", " \"TrialName\": trial.trial_name,\n", " \"TrialComponentDisplayName\": \"Training\",\n", " },\n", ")" ] }, { "cell_type": "markdown", "id": "tight-safety", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "After the training has been running for a while you can view debug info in the Debugger panel. To get to this panel you must click through the experiment, trial, and then component.\n", "![view_debugger panel](./images/view_studio_debugger_output.gif)" ] }, { "cell_type": "markdown", "id": "neither-liechtenstein", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "---\n", "## Host the model\n", "\n", "Now that we've trained the model, let's deploy it to a hosted endpoint. To monitor the model after it's hosted and serving requests, we will also add configurations to capture data that is being sent to the endpoint." ] }, { "cell_type": "code", "execution_count": null, "id": "faced-clark", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "data_capture_prefix = \"{}/datacapture\".format(prefix)\n", "\n", "endpoint_name = \"demo-xgboost-customer-churn-\" + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n", "print(\"EndpointName = {}\".format(endpoint_name))" ] }, { "cell_type": "code", "execution_count": null, "id": "partial-sheffield", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "xgb_predictor = xgb.deploy(\n", " initial_instance_count=1,\n", " instance_type=\"ml.m4.xlarge\",\n", " endpoint_name=endpoint_name,\n", " data_capture_config=DataCaptureConfig(\n", " enable_capture=True,\n", " sampling_percentage=100,\n", " destination_s3_uri=\"s3://{}/{}\".format(bucket, data_capture_prefix),\n", " ),\n", ")" ] }, { "cell_type": "markdown", "id": "environmental-robert", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "### Invoke the deployed model\n", "\n", "Now that we have a hosted endpoint running, we can make real-time predictions from our model by making an HTTP POST request. But first, we'll need to set up serializers and deserializers for passing our `test_data` NumPy arrays to the model behind the endpoint." ] }, { "cell_type": "code", "execution_count": null, "id": "pacific-excuse", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "xgb_predictor.content_type = \"text/csv\"\n", "xgb_predictor.serializer = csv_serializer\n", "xgb_predictor.deserializer = None" ] }, { "cell_type": "markdown", "id": "animal-india", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "Now, we'll loop over our test dataset and collect predictions by invoking the XGBoost endpoint:" ] }, { "cell_type": "code", "execution_count": null, "id": "technological-protocol", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "print(\n", " \"Sending test traffic to the endpoint {}. \\nPlease wait for a minute...\".format(endpoint_name)\n", ")\n", "\n", "with open(\"data/test_sample.csv\", \"r\") as f:\n", " for row in f:\n", " payload = row.rstrip(\"\\n\")\n", " response = xgb_predictor.predict(data=payload)\n", " time.sleep(0.5)" ] }, { "cell_type": "markdown", "id": "clear-insulation", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "### Verify data capture in Amazon S3\n", "\n", "Because we made some real-time predictions by sending data to our endpoint, we should have also captured that data for monitoring purposes. \n", "\n", "Let's list the data capture files stored in Amazon S3. Expect to see different files from different time periods organized by the hour in which the invocation occurred. The format of the S3 path is:\n", "\n", "`s3://{destination-bucket-prefix}/{endpoint-name}/{variant-name}/yyyy/mm/dd/hh/filename.jsonl`" ] }, { "cell_type": "code", "execution_count": null, "id": "great-reviewer", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "from time import sleep\n", "\n", "current_endpoint_capture_prefix = \"{}/{}\".format(data_capture_prefix, endpoint_name)\n", "for _ in range(12): # wait up to a minute to see captures in S3\n", " capture_files = S3Downloader.list(\"s3://{}/{}\".format(bucket, current_endpoint_capture_prefix))\n", " if capture_files:\n", " break\n", " sleep(5)\n", "\n", "print(\"Found Data Capture Files:\")\n", "print(capture_files)" ] }, { "cell_type": "markdown", "id": "brief-sponsorship", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "All the data captured is stored in a SageMaker specific json-line formatted file. Next, Let's take a quick peek at the contents of a single line in a pretty formatted json so that we can observe the format a little better." ] }, { "cell_type": "code", "execution_count": null, "id": "correct-there", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "capture_file = S3Downloader.read_file(capture_files[-1])\n", "\n", "print(\"=====Single Data Capture====\")\n", "print(json.dumps(json.loads(capture_file.split(\"\\n\")[0]), indent=2)[:2000])" ] }, { "cell_type": "markdown", "id": "answering-alexander", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "As you can see, each inference request is captured in one line in the jsonl file. The line contains both the input and output merged together. In our example, we provided the ContentType as `text/csv` which is reflected in the `observedContentType` value. Also, we expose the enconding that we used to encode the input and output payloads in the capture format with the `encoding` value.\n", "\n", "To recap, we have observed how you can enable capturing the input and/or output payloads to an Endpoint with a new parameter. We have also observed how the captured format looks like in S3. Let's continue to explore how SageMaker helps with monitoring the data collected in S3." ] }, { "cell_type": "markdown", "id": "olive-defensive", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "---\n", "## Amazon SageMaker Model Monitor\n", "\n", "In addition to collecting data, Amazon SageMaker lets you to monitor and evaluate the data observed by the endpoints with Amazon SageMaker Model Monitor. For this, we need to:\n", "1. Create a baseline to compare against real-time traffic. \n", "1. When the baseline is ready, set up a schedule to continously evaluate and compare against the baseline.\n", "1. Send synthetic traffic to trigger alarms.\n", "\n", "**Important**: It takes one hour or more to complete this section because the shortest monitoring polling time is one hour. To see how it looks after running for a few hours and some of the synthetic traffic triggered errors, see the following graphic.\n", "\n", "![model monitor example](./images/view_model_monitor_output.gif)" ] }, { "cell_type": "markdown", "id": "transsexual-pricing", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "### Baselining and continous monitoring\n", "\n", "#### 1. Suggest baseline constraints with the training dataset\n", "\n", "The training dataset that you used to train the model is usually a good baseline dataset. Note that the training dataset data schema and the inference dataset schema should match exactly (for example, the number and type of the features).\n", "\n", "From our training dataset, let's ask Amazon SageMaker to suggest a set of baseline `constraints` and generate descriptive `statistics` to explore the data. For this example, let's upload the training dataset that we used to train the model. We'll use the dataset file with column headers so that we have descriptive feature names." ] }, { "cell_type": "code", "execution_count": null, "id": "radical-liability", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "baseline_prefix = prefix + \"/baselining\"\n", "baseline_data_prefix = baseline_prefix + \"/data\"\n", "baseline_results_prefix = baseline_prefix + \"/results\"\n", "\n", "baseline_data_uri = \"s3://{}/{}\".format(bucket, baseline_data_prefix)\n", "baseline_results_uri = \"s3://{}/{}\".format(bucket, baseline_results_prefix)\n", "print(\"Baseline data uri: {}\".format(baseline_data_uri))\n", "print(\"Baseline results uri: {}\".format(baseline_results_uri))\n", "baseline_data_path = S3Uploader.upload(\"data/training-dataset-with-header.csv\", baseline_data_uri)" ] }, { "cell_type": "markdown", "id": "stupid-workshop", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "##### Create a baselining job with the training dataset\n", "\n", "Now that we have the training data ready in Amazon S3, let's start a job to `suggest` constraints. To generate the constraints, the convenient helper starts a `ProcessingJob` using a ProcessingJob container provided by Amazon SageMaker." ] }, { "cell_type": "code", "execution_count": null, "id": "marine-fleet", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "my_default_monitor = DefaultModelMonitor(\n", " role=role,\n", " instance_count=1,\n", " instance_type=\"ml.m5.xlarge\",\n", " volume_size_in_gb=20,\n", " max_runtime_in_seconds=3600,\n", ")\n", "\n", "baseline_job = my_default_monitor.suggest_baseline(\n", " baseline_dataset=baseline_data_path,\n", " dataset_format=DatasetFormat.csv(header=True),\n", " output_s3_uri=baseline_results_uri,\n", " wait=True,\n", ")" ] }, { "cell_type": "markdown", "id": "closed-scientist", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "Once the job succeeds, we can explore the `baseline_results_uri` location in s3 to see what files where stored there." ] }, { "cell_type": "code", "execution_count": null, "id": "monetary-watts", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "print(\"Found Files:\")\n", "S3Downloader.list(\"s3://{}/{}\".format(bucket, baseline_results_prefix))" ] }, { "cell_type": "markdown", "id": "final-fundamentals", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "We have a`constraints.json` file that has information about suggested constraints. We also have a `statistics.json` which contains statistical information about the data in the baseline." ] }, { "cell_type": "code", "execution_count": null, "id": "manufactured-taxation", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "baseline_job = my_default_monitor.latest_baselining_job\n", "schema_df = pd.io.json.json_normalize(baseline_job.baseline_statistics().body_dict[\"features\"])\n", "schema_df.head(10)" ] }, { "cell_type": "code", "execution_count": null, "id": "looking-bubble", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "constraints_df = pd.io.json.json_normalize(\n", " baseline_job.suggested_constraints().body_dict[\"features\"]\n", ")\n", "constraints_df.head(10)" ] }, { "cell_type": "markdown", "id": "confirmed-doctor", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "#### 2. Analyzing subsequent captures for data quality issues\n", "\n", "Now that we have generated a baseline dataset and processed the baseline dataset to get baseline statistics and constraints, let's monitor and analyze the data that is being sent to the endpoint with monitoring schedules." ] }, { "cell_type": "markdown", "id": "defensive-attendance", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "##### Create a schedule\n", "Create a monitoring schedule for the previously created endpoint. The schedule specifies the cadence at which we run a new processing job to compare recent data captures to the baseline." ] }, { "cell_type": "code", "execution_count": null, "id": "juvenile-shareware", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "# First, copy over some test scripts to the S3 bucket so that they can be used for pre and post processing\n", "code_prefix = \"{}/code\".format(prefix)\n", "pre_processor_script = S3Uploader.upload(\n", " \"preprocessor.py\", \"s3://{}/{}\".format(bucket, code_prefix)\n", ")\n", "s3_code_postprocessor_uri = S3Uploader.upload(\n", " \"postprocessor.py\", \"s3://{}/{}\".format(bucket, code_prefix)\n", ")" ] }, { "cell_type": "markdown", "id": "excess-punishment", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "We are ready to create a model monitoring schedule for the Endpoint created before and also the baseline resources (constraints and statistics) which were generated above." ] }, { "cell_type": "code", "execution_count": null, "id": "arbitrary-astronomy", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "from sagemaker.model_monitor import CronExpressionGenerator\n", "from time import gmtime, strftime\n", "\n", "reports_prefix = \"{}/reports\".format(prefix)\n", "s3_report_path = \"s3://{}/{}\".format(bucket, reports_prefix)\n", "\n", "mon_schedule_name = \"demo-xgboost-customer-churn-model-schedule-\" + strftime(\n", " \"%Y-%m-%d-%H-%M-%S\", gmtime()\n", ")\n", "my_default_monitor.create_monitoring_schedule(\n", " monitor_schedule_name=mon_schedule_name,\n", " endpoint_input=xgb_predictor.endpoint,\n", " # record_preprocessor_script=pre_processor_script,\n", " post_analytics_processor_script=s3_code_postprocessor_uri,\n", " output_s3_uri=s3_report_path,\n", " statistics=my_default_monitor.baseline_statistics(),\n", " constraints=my_default_monitor.suggested_constraints(),\n", " schedule_cron_expression=CronExpressionGenerator.hourly(),\n", " enable_cloudwatch_metrics=True,\n", ")" ] }, { "cell_type": "markdown", "id": "ordered-tribute", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "#### 3. Start generating some artificial traffic\n", "The following block starts a thread to send some traffic to the created endpoint. This allows us to continue to send traffic to the endpoint so that we'll continually capture data for analysis. If there is no traffic, the monitoring jobs starts to fail later.\n", "\n", "To terminate this thread, you need to stop the kernel." ] }, { "cell_type": "code", "execution_count": null, "id": "numerical-estimate", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "from threading import Thread\n", "\n", "runtime_client = boto3.client(\"runtime.sagemaker\")\n", "\n", "\n", "# (just repeating code from above for convenience/ able to run this section independently)\n", "def invoke_endpoint(ep_name, file_name, runtime_client):\n", " with open(file_name, \"r\") as f:\n", " for row in f:\n", " payload = row.rstrip(\"\\n\")\n", " response = runtime_client.invoke_endpoint(\n", " EndpointName=ep_name, ContentType=\"text/csv\", Body=payload\n", " )\n", " response[\"Body\"].read()\n", " sleep(1)\n", "\n", "\n", "def invoke_endpoint_forever():\n", " while True:\n", " invoke_endpoint(endpoint_name, \"data/test-dataset-input-cols.csv\", runtime_client)\n", "\n", "\n", "thread = Thread(target=invoke_endpoint_forever)\n", "thread.start()\n", "\n", "# Note that you need to stop the kernel to stop the invocations" ] }, { "cell_type": "markdown", "id": "fiscal-circulation", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "##### List executions\n", "After you've set up the schedule, jobs start at specified intervals. Let's list the latest five executions. If you run this code after creating the hourly schedule, you might find the executions empty. You might have to wait until you cross the hour boundary (in UTC) to see executions start. The following code includes the logic for waiting." ] }, { "cell_type": "code", "execution_count": null, "id": "satisfied-application", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "mon_executions = my_default_monitor.list_executions()\n", "if len(mon_executions) == 0:\n", " print(\n", " \"We created a hourly schedule above and it will kick off executions ON the hour.\\nWe will have to wait till we hit the hour...\"\n", " )\n", "\n", "while len(mon_executions) == 0:\n", " print(\"Waiting for the 1st execution to happen...\")\n", " time.sleep(60)\n", " mon_executions = my_default_monitor.list_executions()" ] }, { "cell_type": "markdown", "id": "brilliant-minneapolis", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "##### Evaluate the latest execution and list the generated reports" ] }, { "cell_type": "code", "execution_count": null, "id": "lucky-guide", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "latest_execution = mon_executions[-1]\n", "latest_execution.wait()" ] }, { "cell_type": "code", "execution_count": null, "id": "designed-cleanup", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "print(\"Latest execution result: {}\".format(latest_execution.describe()[\"ExitMessage\"]))\n", "report_uri = latest_execution.output.destination\n", "\n", "print(\"Found Report Files:\")\n", "S3Downloader.list(report_uri)" ] }, { "cell_type": "markdown", "id": "natural-store", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "##### List violations\n", "\n", "If there are any violations compared to the baseline, they will be generated here. Let's list the violations." ] }, { "cell_type": "code", "execution_count": null, "id": "micro-center", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "violations = my_default_monitor.latest_monitoring_constraint_violations()\n", "pd.set_option(\"display.max_colwidth\", -1)\n", "constraints_df = pd.io.json.json_normalize(violations.body_dict[\"violations\"])\n", "constraints_df.head(10)" ] }, { "cell_type": "markdown", "id": "invisible-thesaurus", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "You can plug in the processing job arn for a single execution of the monitoring into [this notebook](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker_model_monitor/visualization/SageMaker-Model-Monitor-Visualize.ipynb) to see more detailed visualizations of the violations and distribution statistics of the data captue that was processed in that execution\n" ] }, { "cell_type": "code", "execution_count": null, "id": "productive-thumb", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "latest_execution.describe()[\"ProcessingJobArn\"]" ] }, { "cell_type": "markdown", "id": "considered-cathedral", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "source": [ "## Clean up\n", "\n", "If you're done with this notebook, run the following cell. This removes the hosted endpoint that you created and prevents you from being charged for any instances that might continue running. It also cleans up all artifacts related to the experiment. \n", "\n", "You might also want to delete the artifacts stored in the S3 bucket used with this notebook. Go to the Amazon S3 console, find the bucket named `sagemaker-studio--`, and delete the files associated with this notebook." ] }, { "cell_type": "code", "execution_count": null, "id": "retained-dance", "metadata": { "papermill": { "duration": null, "end_time": null, "exception": null, "start_time": null, "status": "pending" }, "tags": [] }, "outputs": [], "source": [ "try:\n", " sess.delete_monitoring_schedule(mon_schedule_name)\n", "except:\n", " pass\n", "while True:\n", " try:\n", " print(\"Waiting for schedule to be deleted\")\n", " sess.describe_monitoring_schedule(mon_schedule_name)\n", " sleep(15)\n", " except:\n", " print(\"Schedule deleted\")\n", " break\n", "\n", "sess.delete_endpoint(xgb_predictor.endpoint)\n", "\n", "\n", "def cleanup(experiment):\n", " \"\"\"Clean up everything in the given experiment object\"\"\"\n", " for trial_summary in experiment.list_trials():\n", " trial = Trial.load(trial_name=trial_summary.trial_name)\n", "\n", " for trial_comp_summary in trial.list_trial_components():\n", " trial_step = TrialComponent.load(\n", " trial_component_name=trial_comp_summary.trial_component_name\n", " )\n", " print(\"Starting to delete TrialComponent..\" + trial_step.trial_component_name)\n", " sm.disassociate_trial_component(\n", " TrialComponentName=trial_step.trial_component_name, TrialName=trial.trial_name\n", " )\n", " trial_step.delete()\n", " time.sleep(1)\n", "\n", " trial.delete()\n", "\n", " experiment.delete()\n", "\n", "\n", "cleanup(customer_churn_experiment)" ] }, { "attachments": {}, "cell_type": "markdown", "id": "e22c5a55", "metadata": {}, "source": [ "## Notebook CI Test Results\n", "\n", "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n", "\n", "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/aws_sagemaker_studio|getting_started|xgboost_customer_churn_studio.ipynb)\n", "\n", "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/aws_sagemaker_studio|getting_started|xgboost_customer_churn_studio.ipynb)\n", "\n", "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/aws_sagemaker_studio|getting_started|xgboost_customer_churn_studio.ipynb)\n", "\n", "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/aws_sagemaker_studio|getting_started|xgboost_customer_churn_studio.ipynb)\n", "\n", "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/aws_sagemaker_studio|getting_started|xgboost_customer_churn_studio.ipynb)\n", "\n", "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/aws_sagemaker_studio|getting_started|xgboost_customer_churn_studio.ipynb)\n", "\n", "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/aws_sagemaker_studio|getting_started|xgboost_customer_churn_studio.ipynb)\n", "\n", "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/aws_sagemaker_studio|getting_started|xgboost_customer_churn_studio.ipynb)\n", "\n", "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/aws_sagemaker_studio|getting_started|xgboost_customer_churn_studio.ipynb)\n", "\n", "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/aws_sagemaker_studio|getting_started|xgboost_customer_churn_studio.ipynb)\n", "\n", "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/aws_sagemaker_studio|getting_started|xgboost_customer_churn_studio.ipynb)\n", "\n", "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/aws_sagemaker_studio|getting_started|xgboost_customer_churn_studio.ipynb)\n", "\n", "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/aws_sagemaker_studio|getting_started|xgboost_customer_churn_studio.ipynb)\n", "\n", "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/aws_sagemaker_studio|getting_started|xgboost_customer_churn_studio.ipynb)\n", "\n", "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/aws_sagemaker_studio|getting_started|xgboost_customer_churn_studio.ipynb)\n" ] } ], "metadata": { "celltoolbar": "Tags", "kernelspec": { "display_name": "Python 3 (Data Science 2.0)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/sagemaker-data-science-38" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.13" }, "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.", "papermill": { "default_parameters": {}, "duration": 378.748954, "end_time": "2021-05-27T00:13:51.425700", "environment_variables": {}, "exception": true, "input_path": "xgboost_customer_churn_studio.ipynb", "output_path": "/opt/ml/processing/output/xgboost_customer_churn_studio-2021-05-27-00-03-43.ipynb", "parameters": { "kms_key": "arn:aws:kms:us-west-2:521695447989:key/6e9984db-50cf-4c7e-926c-877ec47a8b25" }, "start_time": "2021-05-27T00:07:32.676746", "version": "2.3.3" } }, "nbformat": 4, "nbformat_minor": 5 }