Deploy and perform inference on Model Package from AWS Marketplace

---

This notebook’s CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

---

This notebook provides you instructions on how to deploy and perform inference on model packages from AWS Marketplace image classification model.

This notebook is compatible only with those image classification model packages which this notebook is linked to.

Pre-requisites:

1. Note: This notebook contains elements which render correctly in Jupyter interface. Open this notebook from an Amazon SageMaker Notebook Instance or Amazon SageMaker Studio.

2. Ensure that IAM role used has AmazonSageMakerFullAccess

3. To deploy this ML model successfully, ensure that:

   1. Either your IAM role has these three permissions and you have authority to make AWS Marketplace subscriptions in the AWS account used:

      1. aws-marketplace:ViewSubscriptions

      2. aws-marketplace:Unsubscribe

      3. aws-marketplace:Subscribe

   2. or your AWS account has a subscription to this image classification model. If so, skip step: Subscribe to the model package

Contents

1. Subscribe to the model package

2. Create an endpoint and perform real-time inference

   1. Create an endpoint

   2. Create input payload

   3. Perform real-time inference

   4. Visualize output

   5. Delete the endpoint

3. Perform batch inference

4. Clean-up

   1. Delete the model

   2. Unsubscribe to the listing (optional)

Usage instructions

You can run this notebook one cell at a time (By using Shift+Enter for running a cell).

Note - This notebook requires you to follow instructions and specify values for parameters, as instructed.

1. Subscribe to the model package

To subscribe to the model package: 1. Open the model package listing page you opened this notebook for. 1. On the AWS Marketplace listing, click on the Continue to subscribe button. 1. On the Subscribe to this software page, review and click on “Accept Offer” if you and your organization agrees with EULA, pricing, and support terms. 1. Once you click on Continue to configuration button and then choose a region, you will see a Product Arn displayed. This is the model package ARN that you need to specify while creating a deployable model using Boto3. Copy the ARN corresponding to your region and specify the same in the following cell.

model_package_arn = "<Customer to specify Model package ARN corresponding to their AWS region>"

import json
from sagemaker import ModelPackage
import sagemaker as sage
from sagemaker import get_execution_role
from IPython.core.display import Image, display

role = get_execution_role()
sagemaker_session = sage.Session()
boto3 = sagemaker_session.boto_session
bucket = sagemaker_session.default_bucket()
region = sagemaker_session.boto_region_name

s3 = boto3.client("s3")
runtime = boto3.client("runtime.sagemaker")

In next step, you would be deploying the model for real-time inference. For information on how real-time inference with Amazon SageMaker works, see Documentation.

2. Create an endpoint and perform real-time inference

model_name = "image-classification-model"

# The image classification model packages this notebook notebook is compatible with, support application/x-image as the
# content-type.
content_type = "application/x-image"

Review and update the compatible instance type for the model package in the following cell.

real_time_inference_instance_type = "ml.g4dn.xlarge"
batch_transform_inference_instance_type = "ml.p2.xlarge"

A. Create an endpoint

# create a deployable model from the model package.
model = ModelPackage(
    role=role, model_package_arn=model_package_arn, sagemaker_session=sagemaker_session
)

# Deploy the model
predictor = model.deploy(1, real_time_inference_instance_type, endpoint_name=model_name)

Once endpoint has been created, you would be able to perform real-time inference.

B. Prepare input file for performing real-time inference

In this step, we will download class_id_to_label_mapping from S3 bucket. The mapping files has been downloaded from TensorFlow. Apache 2.0 License.

s3_bucket = f"jumpstart-cache-prod-{region}"
key_prefix = "inference-notebook-assets"


def download_from_s3(key_filenames):
    for key_filename in key_filenames:
        s3.download_file(s3_bucket, f"{key_prefix}/{key_filename}", key_filename)


cat_jpg, dog_jpg, ImageNetLabels = "cat.jpg", "dog.jpg", "ImageNetLabels.txt"

# Download images and label-mapping file.
download_from_s3(key_filenames=[cat_jpg, dog_jpg, ImageNetLabels])

display(Image(filename="cat.jpg"))
display(Image(filename="dog.jpg"))

!head ImageNetLabels.txt

Next, open the downloaded images and load them in a variable.

with open(ImageNetLabels, "r") as file:
    class_id_to_label = file.read().splitlines()[1::]
# The label file has 1001 class labels starting with 'background' class
# but the model predicts 1000 classes sans the background class.

C. Query endpoint that you have created with the opened images

# perform_inference method performs inference on the endpoint and prints predictions.
def perform_inference(filename):
    with open(filename, "rb") as file:
        body = file.read()
        response = runtime.invoke_endpoint(
            EndpointName=model_name, ContentType=content_type, Body=body
        )
        prediction = json.loads(response["Body"].read())
        top5_prediction_ids = sorted(
            range(len(prediction)), key=lambda index: prediction[index], reverse=True
        )[:5]
        top5_class_labels = ", ".join([class_id_to_label[id] for id in top5_prediction_ids])
        print("Top-5 model predictions are: " + top5_class_labels)

perform_inference(cat_jpg)

perform_inference(dog_jpg)

D. Delete the endpoint

Now that you have successfully performed a real-time inference, you do not need the endpoint any more. You can terminate the endpoint to avoid being charged.

model.sagemaker_session.delete_endpoint(model_name)
model.sagemaker_session.delete_endpoint_config(model_name)

3. Perform batch inference

In this section, you will perform batch inference using multiple input payloads together. If you are not familiar with batch transform, and want to learn more, see How to run a batch transform job

# upload the batch-transform job input files to S3
transform_input_key_prefix = "image-classification-model-transform-input"
transform_input = sagemaker_session.upload_data(cat_jpg, key_prefix=transform_input_key_prefix)
print("Transform input uploaded to " + transform_input)

# Run the batch-transform job
transformer = model.transformer(1, batch_transform_inference_instance_type)
transformer.transform(transform_input, content_type=content_type)
transformer.wait()

# output is available on following path
transformer.output_path

output_bucket_name, output_path = transformer.output_path.replace("s3://", "").split("/", 1)
obj = s3.get_object(Bucket=output_bucket_name, Key=output_path + "/cat.jpg.out")
batch_prediction = obj["Body"].read().decode("utf-8")

# print out batch-transform job output
print(batch_prediction)

# print labels extracted from batch transform output
top5_prediction_ids = sorted(
    range(len(batch_prediction)), key=lambda index: batch_prediction[index], reverse=True
)[:5]
top5_class_labels = ", ".join([class_id_to_label[id] for id in top5_prediction_ids])
print("Top-5 model predictions are: " + top5_class_labels)

4. Clean-up

A. Delete the model

model.delete_model()

B. Unsubscribe to the listing (optional)

If you would like to unsubscribe to the model package, follow these steps. Before you cancel the subscription, ensure that you do not have any deployable model created from the model package or using the algorithm. Note - You can find this information by looking at the container name associated with the model.

Steps to unsubscribe to product from AWS Marketplace: 1. Navigate to Machine Learning tab on Your Software subscriptions page 2. Locate the listing that you want to cancel the subscription for, and then choose Cancel Subscription to cancel the subscription.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.