SageMaker JumpStart Foundation Models - HuggingFace Text2Text Generation
This notebook’s CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.
Note: This notebook was tested on ml.t3.medium instance in Amazon SageMaker Studio with Python 3 (Data Science) kernel and in Amazon SageMaker Notebook instance with conda_python3 kernel.
1. Set Up
[ ]:
!pip install --upgrade sagemaker --quiet
2. Select a pre-trained model & deploy
You can continue with the default model, or can choose a different model from the list. A complete list of SageMaker pre-trained models can also be accessed at SageMaker pre-trained Models. ***
[ ]:
model_id, model_version = (
"huggingface-text2text-flan-t5-xl",
"2.*",
)
[ ]:
from ipywidgets import Dropdown
from sagemaker.jumpstart.notebook_utils import list_jumpstart_models
# Retrieves all Text Generation models available by SageMaker Built-In Algorithms.
filter_value = "task == text2text"
text_generation_models = list_jumpstart_models(filter=filter_value)
# display the model-ids in a dropdown to select a model for inference.
model_dropdown = Dropdown(
options=text_generation_models,
value=model_id,
description="Select a model",
style={"description_width": "initial"},
layout={"width": "max-content"},
)
display(model_dropdown)
[ ]:
from sagemaker.jumpstart.model import JumpStartModel
model = JumpStartModel(model_id=model_id, model_version=model_version)
predictor = model.deploy()
3. Query endpoint and parse response
Supported parameters
This model supports many parameters while performing inference. They include:
max_length: Model generates text until the output length (which includes the input context length) reaches
max_length. If specified, it must be a positive integer.max_new_tokens: Model generates text until the output length (excluding the input context length) reaches
max_new_tokens. If specified, it must be a positive integer.num_beams: Number of beams used in the greedy search. If specified, it must be integer greater than or equal to
num_return_sequences.no_repeat_ngram_size: Model ensures that a sequence of words of
no_repeat_ngram_sizeis not repeated in the output sequence. If specified, it must be a positive integer greater than 1.temperature: Controls the randomness in the output. Higher temperature results in output sequence with low-probability words and lower temperature results in output sequence with high-probability words. If
temperature-> 0, it results in greedy decoding. If specified, it must be a positive float.early_stopping: If True, text generation is finished when all beam hypotheses reach the end of sentence token. If specified, it must be boolean.
do_sample: If True, sample the next word as per the likelihood. If specified, it must be boolean.
top_k: In each step of text generation, sample from only the
top_kmost likely words. If specified, it must be a positive integer.top_p: In each step of text generation, sample from the smallest possible set of words with cumulative probability
top_p. If specified, it must be a float between 0 and 1.return_full_text: If True, input text will be part of the output generated text. If specified, it must be boolean. The default value for it is False.
**stop*: If specified, it must a list of strings. Text generation stops if any one of the specified strings is generated.
We may specify any subset of the parameters mentioned above while invoking an endpoint. Next, we show an example of how to invoke endpoint with these arguments. ***
[ ]:
# Input must be a json
# Recipe generation
payload = {
"inputs": "Tell me the steps to make a pizza",
"parameters": {"max_new_tokens": 50, "top_k": 50, "top_p": 0.95, "do_sample": True},
}
[ ]:
predictor.predict(payload)
4. Advanced features: How to use prompts engineering to solve different tasks
Below we demonstrate solving 5 key tasks with Flan T5 model. The tasks are: text summarization, common sense reasoning / question answering, sentence classification, translation, pronoun resolution.
Note . The notebook in the following sections are particularly designed for Flan T5 models (small, base, large, xl). There are other models like T5-one-line-summary which are designed for text summarization in particular. In that case, such models cannot perform all the following tasks.
4.1. Summarization
Define the text article you want to summarize.
[ ]:
text = """Amazon Comprehend uses natural language processing (NLP) to extract insights about the content of documents. It develops insights by recognizing the entities, key phrases, language, sentiments, and other common elements in a document. Use Amazon Comprehend to create new products based on understanding the structure of documents. For example, using Amazon Comprehend you can search social networking feeds for mentions of products or scan an entire document repository for key phrases.
You can access Amazon Comprehend document analysis capabilities using the Amazon Comprehend console or using the Amazon Comprehend APIs. You can run real-time analysis for small workloads or you can start asynchronous analysis jobs for large document sets. You can use the pre-trained models that Amazon Comprehend provides, or you can train your own custom models for classification and entity recognition.
All of the Amazon Comprehend features accept UTF-8 text documents as the input. In addition, custom classification and custom entity recognition accept image files, PDF files, and Word files as input.
Amazon Comprehend can examine and analyze documents in a variety of languages, depending on the specific feature. For more information, see Languages supported in Amazon Comprehend. Amazon Comprehend's Dominant language capability can examine documents and determine the dominant language for a far wider selection of languages."""
[ ]:
newline, bold, unbold = "\n", "\033[1m", "\033[0m"
prompts = [
"Briefly summarize this sentence: {text}",
"Write a short summary for this text: {text}",
"Generate a short summary this sentence:\n{text}",
"{text}\n\nWrite a brief summary in a sentence or less",
"{text}\nSummarize the aforementioned text in a single phrase.",
"{text}\nCan you generate a short summary of the above paragraph?",
"Write a sentence based on this summary: {text}",
"Write a sentence based on '{text}'",
"Summarize this article:\n\n{text}",
]
parameters = {
"max_new_tokens": 50,
"top_k": 50,
"top_p": 0.95,
"do_sample": True,
}
for each_prompt in prompts:
payload = {"inputs": each_prompt.replace("{text}", text), "parameters": parameters}
generated_texts = predictor.predict(payload)
print(f"{bold} For prompt: '{each_prompt}'{unbold}{newline}")
print(f"{bold} The result is{unbold}:{generated_texts[0]['generated_text']}{newline}")
4.2. Common sense reasoning / natural language inference
In the common sense reasoning, you can design a prompt and combine it with the premise, hypothesis, and options, send the combined text into the endpoint to get an answer. Examples are demonstrated as below.
Define the premise, hypothesis, and options that you hope the model to reason.
[ ]:
premise = "The world cup has kicked off in Los Angeles, United States."
hypothesis = "The world cup takes place in United States."
options = """["yes", "no"]"""
[ ]:
prompts = [
"""{premise}\n\nBased on the paragraph above can we conclude that "\"{hypothesis}\"?\n\n{options_}""",
"""{premise}\n\nBased on that paragraph can we conclude that this sentence is true?\n{hypothesis}\n\n{options_}""",
"""{premise}\n\nCan we draw the following conclusion?\n{hypothesis}\n\n{options_}""",
"""{premise}\nDoes this next sentence follow, given the preceding text?\n{hypothesis}\n\n{options_}""",
"""{premise}\nCan we infer the following?\n{hypothesis}\n\n{options_}""",
"""Read the following paragraph and determine if the hypothesis is true:\n\n{premise}\n\nHypothesis: {hypothesis}\n\n{options_}""",
"""Read the text and determine if the sentence is true:\n\n{premise}\n\nSentence: {hypothesis}\n\n{options_}""",
"""Can we draw the following hypothesis from the context? \n\nContext:\n\n{premise}\n\nHypothesis: {hypothesis}\n\n{options_}""",
"""Determine if the sentence is true based on the text below:\n{hypothesis}\n\n{premise}\n{options_}""",
]
parameters = {
"max_new_tokens": 50,
"top_k": 50,
"top_p": 0.95,
"do_sample": True,
}
for each_prompt in prompts:
input_text = each_prompt.replace("{premise}", premise)
input_text = input_text.replace("{hypothesis}", hypothesis)
input_text = input_text.replace("{options_}", options)
print(f"{bold} For prompt{unbold}: '{input_text}'{newline}")
payload = {"inputs": input_text, "parameters": parameters}
generated_texts = predictor.predict(payload)
print(
f"{bold} The reasoning result is{unbold}: '{generated_texts[0]['generated_text']}'{newline}"
)
4.3. Question and Answering
Now, let’s try another reasoning task with a different type of prompt template. You can simply provide context and question as shown below.
[ ]:
context = """The newest and most innovative Kindle yet lets you take notes on millions of books and documents, write lists and journals, and more.
For readers who have always wished they could write in their eBooks, Amazon’s new Kindle lets them do just that. The Kindle Scribe is the first Kindle for reading and writing and allows users to supplement their books and documents with notes, lists, and more.
Here’s everything you need to know about the Kindle Scribe, including frequently asked questions.
The Kindle Scribe makes it easy to read and write like you would on paper
The Kindle Scribe features a 10.2-inch, glare-free screen (the largest of all Kindle devices), crisp 300 ppi resolution, and 35 LED front lights that automatically adjust to your environment. Further personalize your experience with the adjustable warm light, font sizes, line spacing, and more.
It comes with your choice of the Basic Pen or the Premium Pen, which you use to write on the screen like you would on paper. They also attach magnetically to your Kindle and never need to be charged. The Premium Pen includes a dedicated eraser and a customizable shortcut button.
The Kindle Scribe has the most storage options of all Kindle devices: choose from 8 GB, 16 GB, or 32 GB to suit your level of reading and writing.
"""
question = "what are the key features of new Kindle?"
[ ]:
prompts = [
"""Answer based on context:\n\n{context}\n\n{question}""",
"""{context}\n\nAnswer this question based on the article: {question}""",
"""{context}\n\n{question}""",
"""{context}\nAnswer this question: {question}""",
"""Read this article and answer this question {context}\n{question}""",
"""{context}\n\nBased on the above article, answer a question. {question}""",
"""Write an article that answers the following question: {question} {context}""",
]
parameters = {
"max_new_tokens": 50,
"top_k": 50,
"top_p": 0.95,
"do_sample": True,
}
for each_prompt in prompts:
input_text = each_prompt.replace("{context}", context)
input_text = input_text.replace("{question}", question)
print(f"{bold} For prompt{unbold}: '{each_prompt}'{newline}")
payload = {"inputs": input_text, "parameters": parameters}
generated_texts = predictor.predict(payload)
print(
f"{bold} The reasoning result is{unbold}: '{generated_texts[0]['generated_text']}'{newline}"
)
4.4. Sentence / Sentiment Classification
Define the sentence you want to classify and the corresponding options.
[ ]:
sentence = "This moive is so great and once again dazzles and delights us"
options_ = """OPTIONS:\n-positive \n-negative """
[ ]:
prompts = [
"""Review:\n{sentence}\nIs this movie review sentence negative or positive?\n{options_}""",
"""Short movie review: {sentence}\nDid the critic think positively or negatively of the movie?\n{options_}""",
"""Sentence from a movie review: {sentence}\nWas the movie seen positively or negatively based on the preceding review? \n\n{options_}""",
"""\"{sentence}\"\nHow would the sentiment of this sentence be perceived?\n\n{options_}""",
"""Is the sentiment of the following sentence positive or negative?\n{sentence}\n{options_}""",
"""What is the sentiment of the following movie review sentence?\n{sentence}\n{options_}""",
]
parameters = {
"max_new_tokens": 50,
"top_k": 50,
"top_p": 0.95,
"do_sample": True,
}
for each_prompt in prompts:
input_text = each_prompt.replace("{sentence}", sentence)
input_text = input_text.replace("{options_}", options_)
print(f"{bold} For prompt{unbold}: '{input_text}'{newline}")
payload = {"inputs": input_text, "parameters": parameters}
generated_texts = predictor.predict(payload)
print(
f"{bold} The reasoning result is{unbold}: '{generated_texts[0]['generated_text']}'{newline}"
)
4.5. Translation
Define the sentence and the language you want to translate the sentence to.
[ ]:
sent1 = "My name is Arthur"
lang2 = """German"""
[ ]:
prompts = [
"""{sent1}\n\nTranslate to {lang2}""",
"""{sent1}\n\nCould you please translate this to {lang2}?""",
"""Translate to {lang2}:\n\n{sent1}""",
"""Translate the following sentence to {lang2}:\n{sent1}""",
"""How is \"{sent1}\" said in {lang2}?""",
"""Translate \"{sent1}\" to {lang2}?""",
]
parameters = {
"max_new_tokens": 50,
"top_k": 50,
"top_p": 0.95,
"do_sample": True,
}
for each_prompt in prompts:
input_text = each_prompt.replace("{sent1}", sent1)
input_text = input_text.replace("{lang2}", lang2)
print(f"{bold} For prompt{unbold}: '{input_text}'{newline}")
payload = {"inputs": input_text, "parameters": parameters}
generated_texts = predictor.predict(payload)
print(
f"{bold} The translated result is{unbold}: '{generated_texts[0]['generated_text']}'{newline}"
)
4.6. Pronoun resolution
Define the sentence, pronoun, and options you want to reason.
[ ]:
sentence = "George talked to Mike because he had experiences in many aspects."
pronoun = "he"
options_ = """\n(A)George \n(B)Mike """
[ ]:
prompts = [
"""sentence}\n\nWho is {pronoun} referring to?\n{options_}""",
"""{sentence}\n\nWho is \"{pronoun}\" in this prior sentence?\n{options_}""",
"""{sentence}\n\nWho is {pronoun} referring to in this sentence?\n{options_}""",
"""{sentence}\nTell me who {pronoun} is.\n{options_}""",
"""{sentence}\nBased on this sentence, who is {pronoun}?\n\n{options_}""",
"""Who is {pronoun} in the following sentence?\n\n{sentence}\n\n{options_}""",
"""Which entity is {pronoun} this sentence?\n\n{sentence}\n\n{options_}""",
]
parameters = {
"max_new_tokens": 50,
"top_k": 50,
"top_p": 0.95,
"do_sample": True,
}
for each_prompt in prompts:
input_text = each_prompt.replace("{sentence}", sentence)
input_text = input_text.replace("{pronoun}", pronoun)
input_text = input_text.replace("{options_}", options_)
print(f"{bold} For prompt{unbold}: '{input_text}'{newline}")
payload = {"inputs": input_text, "parameters": parameters}
generated_texts = predictor.predict(payload)
print(
f"{bold} The translated result is{unbold}: '{generated_texts[0]['generated_text']}'{newline}"
)
4.7. Imaginary article generation based on a title
[ ]:
title = "University has new facility coming up"
[ ]:
prompts = [
"""Title: \"{title}\"\\nGiven the above title of an imaginary article, imagine the article.\\n"""
]
parameters = {
"max_new_tokens": 2000,
"top_k": 50,
"top_p": 0.95,
"do_sample": True,
}
for each_prompt in prompts:
input_text = each_prompt.replace("{title}", title)
print(f"{bold} For prompt{unbold}: '{input_text}'{newline}")
payload = {"inputs": input_text, "parameters": parameters}
generated_texts = predictor.predict(payload)
print(
f"{bold} The reasoning result is{unbold}: '{generated_texts[0]['generated_text']}'{newline}"
)
4.8 Summarize a title based on the article
[ ]:
article = """The newest and most innovative Kindle yet lets you take notes on millions of books and documents, write lists and journals, and more.
For readers who have always wished they could write in their eBooks, Amazon’s new Kindle lets them do just that. The Kindle Scribe is the first Kindle for reading and writing and allows users to supplement their books and documents with notes, lists, and more.
Here’s everything you need to know about the Kindle Scribe, including frequently asked questions.
The Kindle Scribe makes it easy to read and write like you would on paper
The Kindle Scribe features a 10.2-inch, glare-free screen (the largest of all Kindle devices), crisp 300 ppi resolution, and 35 LED front lights that automatically adjust to your environment. Further personalize your experience with the adjustable warm light, font sizes, line spacing, and more.
It comes with your choice of the Basic Pen or the Premium Pen, which you use to write on the screen like you would on paper. They also attach magnetically to your Kindle and never need to be charged. The Premium Pen includes a dedicated eraser and a customizable shortcut button.
The Kindle Scribe has the most storage options of all Kindle devices: choose from 8 GB, 16 GB, or 32 GB to suit your level of reading and writing."""
[ ]:
prompts = ["""'\'{article} \n\n \\n\\nGive me a good title for the article above."""]
parameters = {
"max_length": 2000,
"top_k": 50,
"top_p": 0.95,
"do_sample": True,
}
for each_prompt in prompts:
input_text = each_prompt.replace("{article}", article)
print(f"{bold} For prompt{unbold}: '{input_text}'{newline}")
payload = {"inputs": input_text, "parameters": parameters}
generated_texts = predictor.predict(payload)
print(
f"{bold} The reasoning result is{unbold}: '{generated_texts[0]['generated_text']}'{newline}"
)
5. Clean up the endpoint
[ ]:
# Delete the SageMaker endpoint
predictor.delete_model()
predictor.delete_endpoint()
Notebook CI Test Results
This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.
