Machine Learning on a TabText Dataframe

This notebook’s CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

This us-west-2 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

An Example Based on the Paycheck Protection Program

The Paycheck Protection Program (PPP) was created by the U.S. government to enable employers struggling with COVID-related business adversities to make payments to their employees. For more information, see the Paycheck Protection Program. In this example notebook, you’ll learn how to run a machine learning model on a sample of companies in the program over the first two quarters of 2020.

In this notebook, we take U.S. Securities and Exchange Commission (SEC) filing data from some of the companies that partook of the loans under this program. We demonstrate how to merge the SEC filing data (text data) with stock price data (tabular data) using the SageMaker JumpStart Industry Python SDK. The build_tabText class of the library helps merge text dataframes with numeric dataframes to create a multimodal dataframe for machine learning.

A subset of the list of tickers of firms that took PPP loans obtained from authors of the following paper:
- Balyuk, T., Prabhala, N. and Puri, M. (November 2020, revised June 2021), Indirect Costs of Government Aid and Intermediary Supply Effects: Lessons from the Paycheck Protection Program. NBER Working Paper No. w28114, Available at SSRN:

Companies took funds under the PPP because they were financially constrained and needed to make payroll. However, some companies availed of the loans even though they were not financially constrained. The market penalized the borrowers when they revealed that they took the PPP money, as they read the information as bad news, even for the companies that had no credit quality issues. Subsequently, the better companies returned the money they had borrowed because they wanted to recoup their share value.

We are interested in seeing if an ML model is able to detect whether the text of filings of companies that returned PPP money is different from that of companies that retained PPP money.

Legal Disclaimer: This example notebook is for demonstrative purposes only. It is not financial advice and should not be relied on as financial or investment advice.

General Steps

This notebook takes the following steps:

  1. Read in over 400 tickers of companies that took the PPP loans.

  2. Read in the 10-K, 10-Q, and 8K filings for all paycheck protection tickers during Q1 and Q2 of 2020. Texts in the SEC filings are loaded using the smjsindustry.DataLoader class.

  3. Load a synthetic time series data of daily stock prices for the given tickers during Q1 and Q2 of 2020. Convert prices to returns. The simulated data is generated to be correlated with appropriate labels so that it can be meaningful. An analogous exercise with true data yields similar results.

  4. Merge text and tabular datasets using the smjsindustry.build_tabText API.

  5. Conduct machine learning analysis to obtain a baseline accuracy.

  6. Build an AutoGluon model to analyze how stock prices and texts in the SEC filings are related to each company’s decision to accept or return the money. This notebook shows how to flag all filings of companies that return the money with a 1 and the filings of companies that do not return the money with a 0. A good fit to the data implies the model can distinguish companies into two categories: the ones that return PPP funding versus those that do not based on the text.

  7. Deploy the endpoint for model inference.

  8. Test the endpoint.


The goal in this notebook is to investigate the companies that partook of funding from the Paycheck Protection Program (PPP) and to study how stock prices and returns and text from the SEC forms are related to their decisions to return the money.

The PPP program is reported in each company’s 8-K, an SEC filing which is required when a public company experiences a material change in business conditions. In addition to a 8-K filing, 10-K and 10-Q filings, which present a comprehensive summary of a company’s financial performance, are also used as source of inputs for this study. The stock data is synthetically generated to be correlated with the labels. You can repeat this exercise with actual data as needed.

Note: You can also access this notebook through SageMaker JumpStart that is executable on SageMaker Studio. For more information, see Amazon SageMaker JumpStart Industry in the Amazon SageMaker Developer Guide.

Kernel and SageMaker Setup

Recommended kernel is conda_python3.

Ensure AutoGluon images information is available in SageMaker Python SDK.

[ ]:
!pip install -q -U "sagemaker>=2.66"
[ ]:
import sagemaker

session = sagemaker.Session()
region = session._region_name
bucket = session.default_bucket()
role = sagemaker.get_execution_role()
ppp_folder = "jumpstart_industry_ppp"
train_instance_type = "ml.c5.2xlarge"
inference_instance_type = "ml.m5.xlarge"

Load Data, SDK, and Dependencies

The following code cells download the smjsindustry SDK, dependencies, and dataset from an Amazon S3 bucket prepared by SageMaker JumpStart Industry. You will learn how to use the smjsindustry SDK which contains various APIs to curate SEC datasets. The dataset in this example was synthetically generated using the smjsindustry package’s SEC Forms Retrieval tool.

[ ]:
notebook_artifact_bucket = f"jumpstart-cache-prod-{region}"
notebook_data_prefix = "smfinance-notebook-data/ppp"
notebook_sdk_prefix = "smfinance-notebook-dependency/smjsindustry"
[ ]:
# Download dataset
data_bucket = f"s3://{notebook_artifact_bucket}/{notebook_data_prefix}"
!aws s3 sync $data_bucket ./ --exclude "*" --include "*.csv" --include "*.xlsx"

Install smjsindustry package from whl artifact running the following code block. Alternatively, we can also use pip install smjsindustry==1.0.0.

[ ]:
# Install smjsindustry SDK
sdk_bucket = f"s3://{notebook_artifact_bucket}/{notebook_sdk_prefix}"
!aws s3 sync $sdk_bucket ./

!pip install --no-index smjsindustry-1.0.0-py3-none-any.whl

Import Packages

[ ]:
import os
import boto3
import json
import pandas as pd
import tarfile
import sagemaker
from sklearn.model_selection import train_test_split
from sagemaker.estimator import Estimator
from smjsindustry import build_tabText
from sklearn.metrics import classification_report, confusion_matrix

from ag_model import (

Step 1: Read in the Tickers

Over 400 tickers are used for this study.

[ ]:
%pylab inline

ppp_tickers = pd.read_excel("ppp_tickers.xlsx", index_col=None, sheet_name=0, engine="openpyxl")
print("Number of PPP tickers =", ppp_tickers.shape[0])
ticker_list = list(set(ppp_tickers.ticker))

Step 2: Read in the SEC Forms Filed by These Companies

  1. This notebook retrieves all 10-K/Q, 8-K forms from the SEC servers for Q1 and Q2 of 2020. This was done using the SageMaker JumpStart Industry Python SDK’s DataLoader class. For reference, the time taken by the data loader process was around 30 minutes for curating a dataframe of over 4000 filings.

  2. There is one 10K/Q form per quarter. These are quarterly reports.

  3. There can be multiple 8K forms per quarter, because these are filed for material changes in business conditions. Depending on how many such events there are, several 8Ks might need to be filed. As you will see, this notebook retrieves more than one form per quarter.

  4. The dataset was stored in a CSV file named ppp_10kq_8k_data.csv (351 MB).

Legal Disclaimer: This example notebook uses data obtained from the SEC EDGAR database. You are responsible for complying with EDGAR’s access terms and conditions located in the Accessing EDGAR Data page.

[ ]:
# Text data
df_sec = pd.read_csv("ppp_10kq_8k_data.csv")
[ ]:
print("Number of SEC filings: ", df_sec.shape[0])

Step 3: Collect Stock Prices and Convert to Returns

  • Given the list of tickers, we synthetically generated stock prices using simulation of geometric Brownian motion. The stock prices are generated to be consistent with the real market data. You can buy data for commercial use if needed.

  • Convert the stock prices to returns.

Some tickers might have been delisted since the time of the PPP program.

[ ]:
df_prices = pd.read_csv("ppp_stock_prices_synthetic.csv")
print("Total number of days for the stock time series: ", df_prices.shape[0])
print("Total number of stocks: ", df_prices.shape[1])

The following code cell converts the prices into percentage returns.

  • It converts prices into returns.

  • It calls helper function to convert prices to returns.

  • It removes the stock that only has NaN values, if any.

  • It converts prices to returns using the pct_change function.

[ ]:
def convert_price_to_return(df_prices):
    ticker_list = list(df_prices.columns[1:])
    df_returns = df_prices[ticker_list].pct_change()  # not using fill_method='ffill'
    df_returns = pd.concat([df_prices.Date, df_returns], axis=1)[1:]  # drop first row as it is NaN
    df_returns = df_returns.reset_index(drop=True)
    return df_returns

df_returns = convert_price_to_return(df_prices)
df_returns.dropna(axis=1, how="all", inplace=True)  # drop columns with partial data
df_returns.set_index("Date", inplace=True)
print("Total number of stocks: ", len(list(df_returns.columns[1:])))
[ ]:
df_returns.to_csv("ppp_returns.csv", index=True)

Step 4: Merge Text and Tabular Datasets

The stock returns and the SEC forms are saved in earlier code blocks into CSV files. In this step, you’ll learn how to read in the files and merge the text data with the tabular data.

  • Line up the returns from day -5 before the filing date to day +5 after the filing date. Including the return on the filing date itself, we get 11 days of returns around the filing date. Three types of returns are considered here: > Ret - stock return > MktRet - S&P 500 return > NetRet - difference between Ret and MktRet

  • Merge the SEC text data and the tabular data with the build_tabText API. We need to see how returns evolve around the filing date.

[ ]:

df_returns = pd.read_csv("ppp_returns.csv")  # Tabular/numeric data
df_sec = pd.read_csv("ppp_10kq_8k_data.csv")  # Text data

Define helper functions to create 3 types of returns for 5 days before and 5 days after the filing date. The functions fill in returns for the ticker and corresponding S&P return.

[ ]:

def fillReturn(df_returns, ticker, dt, displacement):
    if np.where(df_returns.columns == ticker)[0].size > 0:
        bwd = list(
            df_returns[ticker].loc[:dt][-(displacement + 1) :]
        )  # 5 days before filing plus filing date
        fwd = list(df_returns[ticker].loc[dt:][1 : (displacement + 1)])  # 5 days after filing
        if len(bwd) < displacement + 1:
            bwd = [np.nan] * (
                displacement + 1 - len(bwd)
            ) + bwd  # Add NaN at the beginning if less bwd
        if len(fwd) < displacement:
            fwd = fwd + [np.nan] * (displacement - len(fwd))  # Append NaN in the end if less fwd
        return bwd + fwd
        return [np.nan for idx in range(2 * displacement + 1)]

def create_df_5_days_return(df_returns):
    displace = 5
    cols = [
    df_trans_dict = {}
    idx = 0
    for ticker in df_returns.columns[1:]:
        for row in range(len(df_returns)):
            dt = df_returns.Date[row]
            rets = fillReturn(df_returns, ticker, dt, displace)
            mkt_rets = fillReturn(df_returns, "^GSPC", dt, displace)
            net_rets = [a - b for a, b in zip(rets, mkt_rets)]
            row_data = [dt, ticker] + rets + mkt_rets + net_rets
            df_trans_dict[idx] = row_data
            idx += 1
    df_returns_trans = pd.DataFrame.from_dict(df_trans_dict, orient="index", columns=cols)
    return df_returns_trans

df_returns_trans = create_df_5_days_return(df_returns)
pd.set_option("display.max_columns", 50)

Create a TabText dataframe

The following code cell calls the smjsindustry.build_tabText class to create a multimodal TabText dataframe, merging the tabular data and the text data together; the dataframe should have the Date column and a common column (‘ticker’ in this case) to generate a time series TabText dataset.

[ ]:
# Use build_tabText API to merge text and tabular datasets
tab_text = build_tabText(
    df_sec, "ticker", "filing_date", df_returns_trans, "ticker", "Date", freq="D"


Write the merged dataframe into a CSV file

[ ]:
tab_text.to_csv("ppp_10kq_8k_stock_data.csv", index=False)

Step 5: Machine Learning Analysis

Some of these companies subsequently returned the money, for various reasons, ranging from:
* They did not need the money,
* Their stock prices were penalized for taking the loans. When they returned the money, the market rewarded them, * They wished to avoid additional regulatory oversight.

Returning the money results in signaling an improvement in business conditions with a subsequent uptick in stock prices. Thus, an exercise to predict which firms would return the money based on their SEC filings might be of interest.

The following code cells prepare the dataset for ML studies with the following steps:
* It flags all filings of the companies that returned the PPP money with a 1 and the others with a 0. Therefore, an ML model fit to these labels teases out whether the text for companies that retain PPP money is distinguishable from text of companies that return PPP money.

The resultant dataframe from the previous steps is stored as a CSV file titled ppp_model_TabText.csv (354 MB). This file contains both text and numerical columns of data.

[ ]:
tab_text = pd.read_csv("ppp_10kq_8k_stock_data.csv")

ppp_tickers_returned = pd.read_excel(
    "ppp_tickers_returned.xlsx", index_col=None, sheet_name=0, engine="openpyxl"
print("Number of PPP Returned tickers =", ppp_tickers_returned.shape[0])
ticker_list_returned = list(set(ppp_tickers_returned.ticker))
[ ]:
tab_text["returned"] = [1 if j in ticker_list_returned else 0 for j in tab_text["ticker"]]
[ ]:
tab_text["returned"] = [1 if j in ticker_list_returned else 0 for j in tab_text["ticker"]]
tab_text = tab_text.drop(["Date"], axis=1)
tab_text.to_csv("ppp_model_TabText.csv", index=False)

You can start examining the mean return in the 5 days before the filing (-5,0) and 5 days after the filing (0,+5) to see how the firms that returned the money fared, compared to those that did not return the money. You’ll learn how the mean excess return (over the S&P return) between the two groups are calculated.

[ ]:
df = pd.read_csv("ppp_model_TabText.csv")

Next, the following cell curates the TabText dataframe by creating a cumulative (net of market) return for the 5 days before the filing (df["First5"]) and the 5 days after the filing (df["Second5"]). You can also see the various feature columns shown in the dataframe as shown in the following cell.

[ ]:
# Add up the returns for days (-5,0) denoted "First5" and days (0,5) denoted second 5
# Note that it is actually 6 days of returns.
df["First5"] = (
    + df["NetRet-4"]
    + df["NetRet-3"]
    + df["NetRet-2"]
    + df["NetRet-1"]
    + df["NetRet0"]
df["Second5"] = (
    df["NetRet5"] + df["NetRet4"] + df["NetRet3"] + df["NetRet2"] + df["NetRet1"] + df["NetRet0"]
[ ]:
res = df.groupby(["returned"]).count()["ticker"]
print("Baseline accuracy =", res[0] / sum(res))
[ ]:
df.groupby(["returned"]).mean()[["First5", "Second5"]]

From the output of the preceding cell, the mean return for the "First5" set is slightly worse for the "returned=0" case and the mean return for the "Second5" set is higher for the "returned=1" case. Maybe firms that returned the money were signalling to the market that they were in good shape and the market rewarded them with a stock price bounce.

Step 6: Machine Learning on the TabText Dataframe

In this notebook, an AutoGluon model is used to analyze how leading stock returns for 5 days (numerical data) and 10-K/Q, 8-K filings (text) are related to each company’s decision to accept or return the money.

Train an AutoGluon Model for Classification

Here, you’ll see how easy it is to undertake a seamless ML on multimodal data (TabText). In this section, you’ll learn how to use one of the open source AWS libraries known as AutoGluon, which is a part of the Gluon NLP family of tools. To learn more, see GluonNLP: NLP made easy.

In particular, we use the AutoGluon-Tabular model, which is designed for TabText and has superior performance. For more information about the model, see AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data.

For a quick start, see Predicting Columns in a Table - Quick Start. To find the AutoGluon-Tabular model in AWS Marketplace, see AutoGluon-Tabular.

The AutoGluon-Tabular model processes the data and trains a diverse ensemble of ML models to create a “predictor” which is able to predict the "returned" label in this data. This example uses both return and text data to build a model.

Create a sample dataset

For demonstration purposes, take a sample from the original dataset to reduce the time for training.

[ ]:
sample_df = pd.concat(
    [df[df["returned"] == 1].sample(n=500), df[df["returned"] == 0].sample(n=500)]

Save the dataframe into a CSV file.

[ ]:
sample_df.to_csv("ppp_model_sample_input.csv", index=False)

Split the sample dataset into a training dataset and a test dataset

[ ]:
from sklearn.model_selection import train_test_split

sample_df_ag = sample_df[["First5", "text", "returned"]]
train_data, test_data = train_test_split(sample_df_ag, test_size=0.2, random_state=123)
[ ]:
train_data.to_csv("train_data.csv", index=False)
test_data.to_csv("test_data.csv", index=False)

train_s3_path = session.upload_data(
    "train_data.csv", bucket=bucket, key_prefix=ppp_folder + "/" + "data"
test_s3_path = session.upload_data(
    "test_data.csv", bucket=bucket, key_prefix=ppp_folder + "/" + "data"
config_s3_path = session.upload_data(
    os.path.join("code", "config.yaml"), bucket=bucket, key_prefix=ppp_folder + "/" + "config"

Run a SageMaker training job

The training job takes around 10 minutes with the sample dataset. If you want to train a model with your own data, you might need to update the training script or configuration file config.yaml in thecode folder. If you want to use GPU instance to achieve a better accuracy, replace train_instance_type with the desired GPU instance.

[ ]:
ag = AutoGluonTraining(
    enable_network_isolation=True,  # Set enable_network_isolation=True to ensure a security running environment
    {"config": config_s3_path, "train": train_s3_path, "test": test_s3_path},

Download Model Outputs

Download the following files (training job artifacts) from the SageMaker session’s default S3 bucket: * leaderboard.csv * predictions.csv * feature_importance.csv * evaluation.json

[ ]:
s3_client = boto3.client("s3")
job_name = ag._current_job_name
s3_client.download_file(bucket, f"{job_name}/output/output.tar.gz", "output.tar.gz")

with"output.tar.gz", "r:gz") as so:

Score details of each model

[ ]:
leaderboard = pd.read_csv("leaderboard.csv")

The result of the training evaluation

[ ]:
with open("evaluation.json") as f:
    data = json.load(f)

The evaluation.json file reports all the usual metrics as well as the Matthews correlation coefficient (MCC). This is a more comprehensive metric for an unbalanced dataset. It ranges from \(-1\) to \(+1\), where \(-1\) implies perfect misclassification and \(+1\) is perfect classification.

Reference: Davide Chicco & Giuseppe Jurman (2020), The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation, BMC Genomics volume 21, Article number: 6

Note: Various metrics are discussed in Receiver operating characteristic in Wikipedia, the free encyclopedia.

Step 7: Deploy the endpoint

In this step, we deploy the model artifact from Step 6 and use for inference. We use AutoGluonInferenceModel defined in to create an AutoGluon model and SageMaker model deployment APIs to deploy an endpoint. If you bring your own data for inference, you may also need to update the inference script in the code folder.

[ ]:
training_job_name =
print("Training job name: ", training_job_name)
[ ]:
ag_estimator = Estimator.attach(training_job_name)
[ ]:
endpoint_name = "jumpstart-example-classic-gecko-ppp-endpoint"

ag_model = AutoGluonInferenceModel(

ppp_predictor = ag_model.deploy(

Step 8: Test the endpoint

We randomly select some data from the test dataset and test the endpoint.

[ ]:
test_endpoint_data = test_data.sample(n=5).drop(["returned"], axis=1)
[ ]:
[ ]:

Clean Up

After you are done using this notebook, delete the model artifacts and other resources to avoid any incurring charges.

Caution: You need to manually delete resources that you may have created while running the notebook, such as Amazon S3 buckets for model artifacts, training datasets, processing artifacts, and Amazon CloudWatch log groups.

For more information about cleaning up resources, see Clean Up in the Amazon SageMaker Developer Guide.

[ ]:


The SageMaker JumpStart Industry product and its related materials are under the Legal License Terms.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.

This us-east-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This us-east-2 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This us-west-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This ca-central-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This sa-east-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This eu-west-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This eu-west-2 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This eu-west-3 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This eu-central-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This eu-north-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This ap-southeast-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This ap-southeast-2 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This ap-northeast-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This ap-northeast-2 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This ap-south-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable