SageMaker Serverless Inference
This notebook’s CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.
XGBoost Regression Example
Amazon SageMaker Serverless Inference is a purpose-built inference option that makes it easy for customers to deploy and scale ML models. Serverless Inference is ideal for workloads which have idle periods between traffic spurts and can tolerate cold starts. Serverless endpoints also automatically launch compute resources and scale them in and out depending on traffic, eliminating the need to choose instance types or manage scaling policies.
For this notebook we’ll be working with the SageMaker XGBoost Algorithm to train a model and then deploy a serverless endpoint. We will be using the public S3 Abalone regression dataset for this example.
Notebook Setting - SageMaker Classic Notebook Instance: ml.m5.xlarge Notebook Instance & conda_python3
Kernel - SageMaker Studio: Python 3 (Data Science) - Regions Available: SageMaker Serverless Inference is currently available in the following regions: US East (Northern Virginia), US East (Ohio), US West (Oregon), EU (Ireland), Asia Pacific (Tokyo) and Asia Pacific (Sydney)
Table of Contents
Setup
Model Training
Deployment
Model Creation
Endpoint Configuration (Adjust for Serverless)
Serverless Endpoint Creation
Endpoint Invocation
Cleanup
Setup
For testing you need to properly configure your Notebook Role to have SageMaker Full Access.
Let’s start by upgrading the Python SDK, boto3
and AWS CLI
(Command Line Interface) packages.
[ ]:
! pip install sagemaker botocore boto3 awscli --upgrade
[ ]:
# Setup clients
import boto3
client = boto3.client(service_name="sagemaker")
runtime = boto3.client(service_name="sagemaker-runtime")
SageMaker Setup
To begin, we import the AWS SDK for Python (Boto3) and set up our environment, including an IAM role and an S3 bucket to store our data.
[ ]:
import boto3
import sagemaker
from sagemaker.estimator import Estimator
boto_session = boto3.session.Session()
region = boto_session.region_name
print(region)
sagemaker_session = sagemaker.Session()
base_job_prefix = "xgboost-example"
role = sagemaker.get_execution_role()
print(role)
default_bucket = sagemaker_session.default_bucket()
s3_prefix = base_job_prefix
training_instance_type = "ml.m5.xlarge"
Retrieve the Abalone dataset from a publicly hosted S3 bucket.
[ ]:
# retrieve data
s3 = boto3.client("s3")
s3.download_file(
f"sagemaker-example-files-prod-{region}",
"datasets/tabular/uci_abalone/train_csv/abalone_dataset1_train.csv",
"abalone_dataset1_train.csv",
)
Upload the Abalone dataset to the default S3 bucket.
[ ]:
# upload data to S3
!aws s3 cp abalone_dataset1_train.csv s3://{default_bucket}/xgboost-regression/train.csv
Model Training
Now, we train an ML model using the XGBoost Algorithm. In this example, we use a SageMaker-provided XGBoost container image and configure an estimator to train our model.
[ ]:
from sagemaker.inputs import TrainingInput
training_path = f"s3://{default_bucket}/xgboost-regression/train.csv"
train_input = TrainingInput(training_path, content_type="text/csv")
[ ]:
model_path = f"s3://{default_bucket}/{s3_prefix}/xgb_model"
# retrieve xgboost image
image_uri = sagemaker.image_uris.retrieve(
framework="xgboost",
region=region,
version="1.0-1",
py_version="py3",
instance_type=training_instance_type,
)
# Configure Training Estimator
xgb_train = Estimator(
image_uri=image_uri,
instance_type=training_instance_type,
instance_count=1,
output_path=model_path,
sagemaker_session=sagemaker_session,
role=role,
)
# Set Hyperparameters
xgb_train.set_hyperparameters(
objective="reg:linear",
num_round=50,
max_depth=5,
eta=0.2,
gamma=4,
min_child_weight=6,
subsample=0.7,
silent=0,
)
Train the model on the Abalone dataset.
[ ]:
# Fit model
xgb_train.fit({"train": train_input})
Deployment
After training the model, retrieve the model artifacts so that we can deploy the model to an endpoint.
[ ]:
# Retrieve model data from training job
model_artifacts = xgb_train.model_data
model_artifacts
Model Creation
Create a model by providing your model artifacts, the container image URI, environment variables for the container (if applicable), a model name, and the SageMaker IAM role.
[ ]:
from time import gmtime, strftime
model_name = "xgboost-serverless" + strftime("%Y-%m-%d-%H-%M-%S", gmtime())
print("Model name: " + model_name)
# dummy environment variables
byo_container_env_vars = {"SAGEMAKER_CONTAINER_LOG_LEVEL": "20", "SOME_ENV_VAR": "myEnvVar"}
create_model_response = client.create_model(
ModelName=model_name,
Containers=[
{
"Image": image_uri,
"Mode": "SingleModel",
"ModelDataUrl": model_artifacts,
"Environment": byo_container_env_vars,
}
],
ExecutionRoleArn=role,
)
print("Model Arn: " + create_model_response["ModelArn"])
Endpoint Configuration Creation
This is where you can adjust the Serverless Configuration for your endpoint. The current max concurrent invocations for a single endpoint, known as MaxConcurrency
, can be any value from 1 to 200, and MemorySize
can be any of the following: 1024 MB, 2048 MB, 3072 MB, 4096 MB, 5120 MB, or 6144 MB.
[ ]:
xgboost_epc_name = "xgboost-serverless-epc" + strftime("%Y-%m-%d-%H-%M-%S", gmtime())
endpoint_config_response = client.create_endpoint_config(
EndpointConfigName=xgboost_epc_name,
ProductionVariants=[
{
"VariantName": "byoVariant",
"ModelName": model_name,
"ServerlessConfig": {
"MemorySizeInMB": 4096,
"MaxConcurrency": 1,
},
},
],
)
print("Endpoint Configuration Arn: " + endpoint_config_response["EndpointConfigArn"])
Serverless Endpoint Creation
Now that we have an endpoint configuration, we can create a serverless endpoint and deploy our model to it. When creating the endpoint, provide the name of your endpoint configuration and a name for the new endpoint.
[ ]:
endpoint_name = "xgboost-serverless-ep" + strftime("%Y-%m-%d-%H-%M-%S", gmtime())
create_endpoint_response = client.create_endpoint(
EndpointName=endpoint_name,
EndpointConfigName=xgboost_epc_name,
)
print("Endpoint Arn: " + create_endpoint_response["EndpointArn"])
Wait until the endpoint status is InService
before invoking the endpoint.
[ ]:
# wait for endpoint to reach a terminal state (InService) using describe endpoint
import time
describe_endpoint_response = client.describe_endpoint(EndpointName=endpoint_name)
while describe_endpoint_response["EndpointStatus"] == "Creating":
describe_endpoint_response = client.describe_endpoint(EndpointName=endpoint_name)
print(describe_endpoint_response["EndpointStatus"])
time.sleep(15)
describe_endpoint_response
Endpoint Invocation
Invoke the endpoint by sending a request to it. The following is a sample data point grabbed from the CSV file downloaded from the public Abalone dataset.
[ ]:
response = runtime.invoke_endpoint(
EndpointName=endpoint_name,
Body=b".345,0.224414,.131102,0.042329,.279923,-0.110329,-0.099358,0.0",
ContentType="text/csv",
)
print(response["Body"].read())
Clean Up
Delete any resources you created in this notebook that you no longer wish to use.
[ ]:
client.delete_model(ModelName=model_name)
client.delete_endpoint_config(EndpointConfigName=xgboost_epc_name)
client.delete_endpoint(EndpointName=endpoint_name)
Notebook CI Test Results
This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.