Introduction to JumpStart - Sentence Pair Classification
This notebook’s CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.
1. Set Up
Before executing the notebook, there are some initial steps required for setup. This notebook requires latest version of sagemaker and ipywidgets. ***
[ ]:
!pip install sagemaker ipywidgets --upgrade --quiet
To train and host on Amazon SageMaker, we need to setup and authenticate the use of AWS services. Here, we use the execution role associated with the current notebook instance as the AWS account role with SageMaker access. It has necessary permissions, including access to your data in S3.
[ ]:
import sagemaker, boto3, json
from sagemaker import get_execution_role
aws_role = get_execution_role()
aws_region = boto3.Session().region_name
sess = sagemaker.Session()
2. Select a pre-trained model
You can continue with the default model, or can choose a different model from the dropdown generated upon running the next cell. A complete list of JumpStart models can also be accessed at JumpStart Models. ***
[ ]:
model_id = "huggingface-spc-bert-base-uncased"
[Optional] Select a different JumpStart model. Here, we download jumpstart model_manifest file from the jumpstart s3 bucket, filter-out all the Sentence Pair Classification models and select a model. ***
[ ]:
import IPython
import ipywidgets as widgets
# download JumpStart model_manifest file.
boto3.client("s3").download_file(
f"jumpstart-cache-prod-{aws_region}", "models_manifest.json", "models_manifest.json"
)
with open("models_manifest.json", "rb") as json_file:
model_list = json.load(json_file)
# filter-out all the Sentence Pair Classification models from the manifest list.
spc_models_all_versions, spc_models = [
model["model_id"] for model in model_list if "-spc-" in model["model_id"]
], []
[spc_models.append(model) for model in spc_models_all_versions if model not in spc_models]
# display the model-ids in a dropdown, for user to select a model.
dropdown = widgets.Dropdown(
value=model_id,
options=spc_models,
description="JumpStart Sentence Pair Classification Models:",
style={"description_width": "initial"},
layout={"width": "max-content"},
)
display(IPython.display.Markdown("## Select a JumpStart pre-trained model from the dropdown below"))
display(dropdown)
3. Run inference on the pre-trained model
Using JumpStart, we can perform inference on the pre-trained model, even without fine-tuning it first on a custom dataset. For this example, that means on a pair of input sentences predicting the class label from one of the 2 classes of the QNLI dataset: entail, no-entail.
3.1. Retrieve JumpStart Artifacts & Deploy an Endpoint
We retrieve the deploy_image_uri, deploy_source_uri, and base_model_uri for the pre-trained model. To host the pre-trained model, we create an instance of `sagemaker.model.Model
<https://sagemaker.readthedocs.io/en/stable/api/inference/model.html>`__ and deploy it. ***
[ ]:
from sagemaker import image_uris, model_uris, script_uris
from sagemaker.model import Model
from sagemaker.predictor import Predictor
from sagemaker.utils import name_from_base
# model_version="*" fetches the latest version of the model.
infer_model_id, infer_model_version = dropdown.value, "*"
endpoint_name = name_from_base(f"jumpstart-example-{infer_model_id}")
inference_instance_type = "ml.m5.xlarge"
# Retrieve the inference docker container uri.
deploy_image_uri = image_uris.retrieve(
region=None,
framework=None,
image_scope="inference",
model_id=infer_model_id,
model_version=infer_model_version,
instance_type=inference_instance_type,
)
# Retrieve the inference script uri.
deploy_source_uri = script_uris.retrieve(
model_id=infer_model_id, model_version=infer_model_version, script_scope="inference"
)
# Retrieve the base model uri.
base_model_uri = model_uris.retrieve(
model_id=infer_model_id, model_version=infer_model_version, model_scope="inference"
)
# Create the SageMaker model instance. Note that we need to pass Predictor class when we deploy model through Model class,
# for being able to run inference through the SageMaker API.
model = Model(
image_uri=deploy_image_uri,
source_dir=deploy_source_uri,
model_data=base_model_uri,
entry_point="inference.py",
role=aws_role,
predictor_cls=Predictor,
name=endpoint_name,
)
# deploy the Model.
base_model_predictor = model.deploy(
initial_instance_count=1,
instance_type=inference_instance_type,
endpoint_name=endpoint_name,
)
3.2. Example input sentences for inference
Let’s put in some example sentence pairs. You can put in any pairs of sentences, the model will predict whether the second sentence entails the first sentence or not. These examples are taken from QNLI dataset downloaded from TensorFlow. Apache 2.0 License. Dataset Homepage. CC BY-SA 4.0 License. ***
[ ]:
sentence_pair1 = [
"How many octaves does Beyonce have?",
"Beyoncé's vocal range spans four octaves.",
]
sentence_pair2 = [
"How many octaves does Beyonce have?",
"While another critic says she is a "
"Vocal acrobat, being able to sing long and complex melismas and vocal runs effortlessly, and in key.",
]
3.3. Query endpoint and parse response
Input to the endpoint is a pair of sentences. Response from the endpoint is a dictionary containing the predicted class label, and a list of class label probabilities. ***
[ ]:
newline, bold, unbold = "\n", "\033[1m", "\033[0m"
def query_endpoint(encoded_text):
response = base_model_predictor.predict(
encoded_text, {"ContentType": "application/list-text", "Accept": "application/json;verbose"}
)
return response
def parse_response(query_response):
model_predictions = json.loads(query_response)
probabilities, labels, predicted_label = (
model_predictions["probabilities"],
model_predictions["labels"],
model_predictions["predicted_label"],
)
return probabilities, labels, predicted_label
for sentence_pair in [sentence_pair1, sentence_pair2]:
query_response = query_endpoint(json.dumps(sentence_pair).encode("utf-8"))
probabilities, labels, predicted_label = parse_response(query_response)
print(
f"Inference:{newline}"
f"Input text: '{sentence_pair}'{newline}"
f"Model prediction: {probabilities}{newline}"
f"Labels: {labels}{newline}"
f"Predicted Label: {bold}{predicted_label}{unbold}{newline}"
)
3.4. Clean up the endpoint
[ ]:
# Delete the SageMaker endpoint and the attached resources
base_model_predictor.delete_model()
base_model_predictor.delete_endpoint()
4. Finetune the pre-trained model on a custom dataset
Previously, we saw how to run inference on a pre-trained model, which was fine-tuned on QNLI dataset. Next, we discuss how a model can be finetuned to a custom dataset.
The Text Embedding model can be fine-tuned on any sentence pair classification dataset in the same way the model available for inference has been fine-tuned on the QNLI dataset. The model available for fine-tuning attaches a binary classification layer to the Text Embedding model and initializes the layer parameters to random values. The fine-tuning step fine-tunes all the model parameters to minimize prediction error on the input data and returns the fine-tuned model. The model returned by fine-tuning can be further deployed for inference. Below are the instructions for how the training data should be formatted for input to the model.
Input: A directory containing a ‘data.csv’ file.
Each row of the first column of ‘data.csv’ should have 0/1 integer class labels.
Each row of the second column should have the corresponding first sentence.
Each row of the third column should have the corresponding second sentence.
Output: A trained model that can be deployed for inference.
Below is an example of ‘data.csv’ file showing values in its first three columns. Note that the file should not have any header.
0 |
What is the Grotto at Notre Dame? |
Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. |
1 |
What is the Grotto at Notre Dame? |
It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. |
0 |
What sits on top of the Main Building at Notre Dame? |
Atop the Main Building’s gold dome is a golden statue of the Virgin Mary. |
… |
… |
… |
QNLI dataset is downloaded from TensorFlow. Apache 2.0 License. Dataset Homepage. CC BY-SA 4.0 License. ***
4.1. Retrieve JumpStart Training artifacts
Here, for the selected model, we retrieve the training docker container, the training algorithm source, the pre-trained model, and a python dictionary of the training hyper-parameters that the algorithm accepts with their default values. Note that the model_version=”*” fetches the latest model. Also, we do need to specify the training_instance_type to fetch train_image_uri.***
[ ]:
from sagemaker import image_uris, model_uris, script_uris, hyperparameters
model_id, model_version = dropdown.value, "*"
training_instance_type = "ml.p3.2xlarge"
# Retrieve the docker image
train_image_uri = image_uris.retrieve(
region=None,
framework=None,
model_id=model_id,
model_version=model_version,
image_scope="training",
instance_type=training_instance_type,
)
# Retrieve the training script
train_source_uri = script_uris.retrieve(
model_id=model_id, model_version=model_version, script_scope="training"
)
# Retrieve the pre-trained model tarball to further fine-tune
train_model_uri = model_uris.retrieve(
model_id=model_id, model_version=model_version, model_scope="training"
)
4.2. Set Training parameters
Now that we are done with all the setup that is needed, we are ready to fine-tune our Sentence Pair Classification model. To begin, let us create a `sagemaker.estimator.Estimator
<https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html>`__ object. This estimator will launch the training job.
There are two kinds of parameters that need to be set for training.
The first one are the parameters for the training job. These include: (i) Training data path. This is S3 folder in which the input data is stored, (ii) Output path: This the s3 folder in which the training output is stored. (iii) Training instance type: This indicates the type of machine on which to run the training. Typically, we use GPU instances for these training. We defined the training instance type above to fetch the correct train_image_uri.
The second set of parameters are algorithm specific training hyper-parameters. ***
[ ]:
# Sample training data is available in this bucket
training_data_bucket = f"jumpstart-cache-prod-{aws_region}"
# For a quick demonstration of training we have created a random subset of QNLI dataset.
# For complete QNLI dataset replace "QNLI-tiny" with "QNLI" in the line below.
training_data_prefix = "training-datasets/QNLI-tiny/"
training_dataset_s3_path = f"s3://{training_data_bucket}/{training_data_prefix}"
output_bucket = sess.default_bucket()
output_prefix = "jumpstart-example-spc-training"
s3_output_location = f"s3://{output_bucket}/{output_prefix}/output"
For algorithm specific hyper-parameters, we start by fetching python dictionary of the training hyper-parameters that the algorithm accepts with their default values. This can then be overridden to custom values. ***
[ ]:
from sagemaker import hyperparameters
# Retrieve the default hyper-parameters for fine-tuning the model
hyperparameters = hyperparameters.retrieve_default(model_id=model_id, model_version=model_version)
# [Optional] Override default hyperparameters with custom values
hyperparameters["batch-size"] = "64"
print(hyperparameters)
4.3. Train with Automatic Model Tuning (HPO)
Amazon SageMaker automatic model tuning, also known as hyperparameter tuning, finds the best version of a model by running many training jobs on your dataset using the algorithm and ranges of hyperparameters that you specify. It then chooses the hyperparameter values that result in a model that performs the best, as measured by a metric that you choose. We will use a HyperparameterTuner object to interact with Amazon SageMaker hyperparameter tuning APIs. ***
[ ]:
from sagemaker.tuner import ContinuousParameter
# Use AMT for tuning and selecting the best model
use_amt = True
# Define objective metric per framework, based on which the best model will be selected.
metric_definitions_per_model = {
"tensorflow": {
"metrics": [{"Name": "val_accuracy", "Regex": "val_accuracy: ([0-9\\.]+)"}],
"type": "Maximize",
},
"huggingface": {
"metrics": [{"Name": "eval_accuracy", "Regex": "'eval_accuracy': ([0-9\\.]+)"}],
"type": "Maximize",
},
}
# You can select from the hyperparameters supported by the model, and configure ranges of values to be searched for training the optimal model.(https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-define-ranges.html)
hyperparameter_ranges = {
"adam-learning-rate": ContinuousParameter(0.000001, 0.001, scaling_type="Logarithmic")
}
# Increase the total number of training jobs run by AMT, for increased accuracy (and training time).
max_jobs = 6
# Change parallel training jobs run by AMT to reduce total training time, constrained by your account limits.
# if max_jobs=max_parallel_jobs then Bayesian search turns to Random.
max_parallel_jobs = 2
4.4. Start Training
We start by creating the estimator object with all the required assets and then launch the training job. ***
[ ]:
from sagemaker.estimator import Estimator
from sagemaker.utils import name_from_base
from sagemaker.tuner import HyperparameterTuner
training_job_name = name_from_base(f"jumpstart-example-{model_id}-transfer-learning")
# Create SageMaker Estimator instance
spc_estimator = Estimator(
role=aws_role,
image_uri=train_image_uri,
source_dir=train_source_uri,
model_uri=train_model_uri,
entry_point="transfer_learning.py",
instance_count=1,
instance_type=training_instance_type,
max_run=360000,
hyperparameters=hyperparameters,
output_path=s3_output_location,
base_job_name=training_job_name,
)
if use_amt:
metric_definitions = next(
value for key, value in metric_definitions_per_model.items() if model_id.startswith(key)
)
hp_tuner = HyperparameterTuner(
spc_estimator,
metric_definitions["metrics"][0]["Name"],
hyperparameter_ranges,
metric_definitions["metrics"],
max_jobs=max_jobs,
max_parallel_jobs=max_parallel_jobs,
objective_type=metric_definitions["type"],
base_tuning_job_name=training_job_name,
)
# Launch a SageMaker Tuning job to search for the best hyperparameters
hp_tuner.fit({"training": training_dataset_s3_path})
else:
# Launch a SageMaker Training job by passing s3 path of the training data
spc_estimator.fit({"training": training_dataset_s3_path}, logs=True)
4.5. Deploy & run Inference on the fine-tuned model
A trained model does nothing on its own. We now want to use the model to perform inference. For this example, that means predicting the class label of an input sentence. We follow the same steps as in 3. Run inference on the pre-trained model. We start by retrieving the jumpstart artifacts for deploying an endpoint. However, instead of base_predictor, we deploy the spc_estimator
that we fine-tuned. ***
[ ]:
inference_instance_type = "ml.m5.xlarge"
# Retrieve the inference docker container uri
deploy_image_uri = image_uris.retrieve(
region=None,
framework=None,
image_scope="inference",
model_id=model_id,
model_version=model_version,
instance_type=inference_instance_type,
)
# Retrieve the inference script uri
deploy_source_uri = script_uris.retrieve(
model_id=model_id, model_version=model_version, script_scope="inference"
)
endpoint_name = name_from_base(f"jumpstart-example-FT-{model_id}-")
# Use the estimator from the previous step to deploy to a SageMaker endpoint
finetuned_predictor = (hp_tuner if use_amt else spc_estimator).deploy(
initial_instance_count=1,
instance_type=inference_instance_type,
entry_point="inference.py",
image_uri=deploy_image_uri,
source_dir=deploy_source_uri,
endpoint_name=endpoint_name,
)
[ ]:
sentence_pair1 = [
"How many octaves does Beyonce have?",
"Beyoncé's vocal range spans four octaves.",
]
sentence_pair2 = [
"How many octaves does Beyonce have?",
"While another critic says she is a "
"Vocal acrobat, being able to sing long and complex melismas and vocal runs effortlessly, and in key.",
]
[ ]:
newline, bold, unbold = "\n", "\033[1m", "\033[0m"
def query_endpoint(encoded_text):
response = finetuned_predictor.predict(
encoded_text, {"ContentType": "application/list-text", "Accept": "application/json;verbose"}
)
return response
def parse_response(query_response):
model_predictions = json.loads(query_response)
probabilities, labels, predicted_label = (
model_predictions["probabilities"],
model_predictions["labels"],
model_predictions["predicted_label"],
)
return probabilities, labels, predicted_label
for sentence_pair in [sentence_pair1, sentence_pair2]:
query_response = query_endpoint(json.dumps(sentence_pair).encode("utf-8"))
probabilities, labels, predicted_label = parse_response(query_response)
print(
f"Inference:{newline}"
f"Input text: '{sentence_pair}'{newline}"
f"Model prediction: {probabilities}{newline}"
f"Labels: {labels}{newline}"
f"Predicted Label: {bold}{predicted_label}{unbold}{newline}"
)
[ ]:
# Delete the SageMaker endpoint and the attached resources
finetuned_predictor.delete_model()
finetuned_predictor.delete_endpoint()
Notebook CI Test Results
This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.