Amazon SageMaker Clarify Model Explainability Monitor for Batch Transform - JSON Format


This notebook’s CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

This us-west-2 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable


Runtime

This notebook takes approximately 60 minutes to run.

Contents

Introduction

Amazon SageMaker Model Monitor continuously monitors the quality of Amazon SageMaker machine learning models in production. It enables developers to set alerts for when there are deviations in the model quality. Early and pro-active detection of these deviations enables corrective actions, such as retraining models, auditing upstream systems, or fixing data quality issues without having to monitor models manually or build additional tooling.

Amazon SageMaker Clarify Model Explainability Monitor is a model monitor that helps data scientists and ML engineers monitor predictions for feature attribution drift on a regular basis. A drift in the distribution of live data for models in production can result in a corresponding drift in the feature attribution values. As the model is monitored, customers can view exportable reports and graphs detailing feature attributions in SageMaker Studio and configure alerts in Amazon CloudWatch to receive notifications if it is detected that the attribution values drift beyond a certain threshold.

This notebook demonstrates the process for setting up a SageMaker Clarify Feature Attribution Drift Monitor for continuous monitoring of feature attribution drift of the data and model used by a regularly running SageMaker Batch Transform job. The model input and output are in SageMaker JSON Lines dense format.

In general, you can use the model explainability monitor for batch transform in this way,

  1. Schedule a model explainability monitor to monitor a data capture S3 location

  2. Regularly run transform jobs with data capture enabled, the jobs save captured data to the data capture S3 URI

The monitor executes processing jobs regularly to do feature attribution analysis, and then generate analysis reports and publish metrics to CloudWatch.

General Setup

The notebook uses the SageMaker Python SDK. The following cell upgrades the SDK and its dependencies. Then you may need to restart the kernel and rerun the notebook to pick up the up-to-date APIs, if the notebook is executed in the SageMaker Studio.

[1]:
!pip install -U sagemaker
!pip install -U boto3
!pip install -U botocore
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
Requirement already satisfied: sagemaker in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (2.203.1)
Requirement already satisfied: fastapi==0.95.2 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (0.95.2)
Requirement already satisfied: platformdirs in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (3.10.0)
Requirement already satisfied: protobuf<5.0,>=3.12 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (3.20.3)
Requirement already satisfied: jsonschema in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (4.19.0)
Requirement already satisfied: psutil in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (5.9.4)
Requirement already satisfied: docker in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (6.1.3)
Requirement already satisfied: PyYAML~=6.0 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (6.0)
Requirement already satisfied: attrs<24,>=23.1.0 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (23.1.0)
Requirement already satisfied: requests in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (2.28.2)
Requirement already satisfied: smdebug-rulesconfig==1.0.1 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (1.0.1)
Requirement already satisfied: tqdm in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (4.66.1)
Requirement already satisfied: uvicorn==0.22.0 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (0.22.0)
Requirement already satisfied: numpy<2.0,>=1.9.0 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (1.24.3)
Requirement already satisfied: packaging>=20.0 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (23.1)
Requirement already satisfied: google-pasta in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (0.2.0)
Requirement already satisfied: pathos in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (0.3.1)
Requirement already satisfied: boto3<2.0,>=1.33.3 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (1.34.22)
Requirement already satisfied: urllib3<1.27 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (1.26.16)
Requirement already satisfied: cloudpickle==2.2.1 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (2.2.1)
Requirement already satisfied: pandas in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (2.1.0)
Requirement already satisfied: schema in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (0.7.5)
Requirement already satisfied: tblib<3,>=1.7.0 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (1.7.0)
Requirement already satisfied: importlib-metadata<7.0,>=1.4.0 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from sagemaker) (4.13.0)
Requirement already satisfied: pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from fastapi==0.95.2->sagemaker) (1.10.13)
Requirement already satisfied: starlette<0.28.0,>=0.27.0 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from fastapi==0.95.2->sagemaker) (0.27.0)
Requirement already satisfied: h11>=0.8 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from uvicorn==0.22.0->sagemaker) (0.14.0)
Requirement already satisfied: click>=7.0 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from uvicorn==0.22.0->sagemaker) (8.1.3)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from boto3<2.0,>=1.33.3->sagemaker) (1.0.1)
Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from boto3<2.0,>=1.33.3->sagemaker) (0.10.0)
Requirement already satisfied: botocore<1.35.0,>=1.34.22 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from boto3<2.0,>=1.33.3->sagemaker) (1.34.22)
Requirement already satisfied: zipp>=0.5 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from importlib-metadata<7.0,>=1.4.0->sagemaker) (3.17.0)
Requirement already satisfied: websocket-client>=0.32.0 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from docker->sagemaker) (1.5.1)
Requirement already satisfied: idna<4,>=2.5 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from requests->sagemaker) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from requests->sagemaker) (2022.12.7)
Requirement already satisfied: charset-normalizer<4,>=2 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from requests->sagemaker) (3.0.1)
Requirement already satisfied: six in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from google-pasta->sagemaker) (1.16.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from jsonschema->sagemaker) (2023.7.1)
Requirement already satisfied: referencing>=0.28.4 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from jsonschema->sagemaker) (0.30.2)
Requirement already satisfied: rpds-py>=0.7.1 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from jsonschema->sagemaker) (0.10.3)
Requirement already satisfied: pytz>=2020.1 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from pandas->sagemaker) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from pandas->sagemaker) (2023.3)
Requirement already satisfied: python-dateutil>=2.8.2 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from pandas->sagemaker) (2.8.2)
Requirement already satisfied: multiprocess>=0.70.15 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from pathos->sagemaker) (0.70.15)
Requirement already satisfied: ppft>=1.7.6.7 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from pathos->sagemaker) (1.7.6.7)
Requirement already satisfied: pox>=0.3.3 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from pathos->sagemaker) (0.3.3)
Requirement already satisfied: dill>=0.3.7 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from pathos->sagemaker) (0.3.7)
Requirement already satisfied: contextlib2>=0.5.5 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from schema->sagemaker) (21.6.0)
Requirement already satisfied: typing-extensions>=4.2.0 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2->fastapi==0.95.2->sagemaker) (4.8.0)
Requirement already satisfied: anyio<5,>=3.4.0 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from starlette<0.28.0,>=0.27.0->fastapi==0.95.2->sagemaker) (3.7.1)
Requirement already satisfied: exceptiongroup in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi==0.95.2->sagemaker) (1.1.0)
Requirement already satisfied: sniffio>=1.1 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi==0.95.2->sagemaker) (1.3.0)
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)

[notice] A new release of pip is available: 23.0.1 -> 23.3.2
[notice] To update, run: pip install --upgrade pip
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
Requirement already satisfied: boto3 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (1.34.22)
Requirement already satisfied: botocore<1.35.0,>=1.34.22 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from boto3) (1.34.22)
Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from boto3) (0.10.0)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from boto3) (1.0.1)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from botocore<1.35.0,>=1.34.22->boto3) (1.26.16)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from botocore<1.35.0,>=1.34.22->boto3) (2.8.2)
Requirement already satisfied: six>=1.5 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.35.0,>=1.34.22->boto3) (1.16.0)
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)

[notice] A new release of pip is available: 23.0.1 -> 23.3.2
[notice] To update, run: pip install --upgrade pip
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
Requirement already satisfied: botocore in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (1.34.22)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from botocore) (1.0.1)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from botocore) (1.26.16)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from botocore) (2.8.2)
Requirement already satisfied: six>=1.5 in /local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore) (1.16.0)
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)
WARNING: Ignoring invalid distribution -otocore (/local/home/zicanl/.virtualenvs/venv/lib/python3.9/site-packages)

[notice] A new release of pip is available: 23.0.1 -> 23.3.2
[notice] To update, run: pip install --upgrade pip

Imports

The following cell imports the APIs to be used by the notebook.

[2]:
import sagemaker
import pandas as pd
import copy
import datetime
import json
import os
import pprint
import time
sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml
sagemaker.config INFO - Not applying SDK defaults from location: /home/zicanl/.config/sagemaker/config.yaml

Handful of configuration

To begin, ensure that these prerequisites have been completed.

  • Specify an AWS Region to host the model.

  • Specify an IAM role to execute jobs.

  • Define the S3 URIs that stores the model file, input data and output data. For demonstration purposes, this notebook uses the same bucket for them. In reality, they could be separated with different security policies.

[3]:
sagemaker_session = sagemaker.Session()

region = sagemaker_session.boto_region_name
print(f"AWS region: {region}")

role = sagemaker.get_execution_role()
print(f"RoleArn: {role}")

# A different bucket can be used, but make sure the role for this notebook has
# the s3:PutObject permissions. This is the bucket into which the data is captured
bucket = sagemaker_session.default_bucket()
print(f"Demo Bucket: {bucket}")
prefix = sagemaker.utils.unique_name_from_base("sagemaker/DEMO-ClarifyModelMonitor")
print(f"Demo Prefix: {prefix}")
s3_key = f"s3://{bucket}/{prefix}"
print(f"Demo S3 key: {s3_key}")

data_capture_s3_uri = f"{s3_key}/data-capture"
transform_output_s3_uri = f"{s3_key}/transform-output"
baselining_output_s3_uri = f"{s3_key}/baselining-output"
monitor_output_s3_uri = f"{s3_key}/monitor-output"

print(f"The transform job will save the results to: {transform_output_s3_uri}")
print(f"The transform job will save the captured data to: {data_capture_s3_uri}")
print(f"The baselining job will save the analysis results to: {baselining_output_s3_uri}")
print(f"The monitor will save the analysis results to: {monitor_output_s3_uri}")
AWS region: us-west-2
RoleArn: arn:aws:iam::678264136642:role/Admin
Demo Bucket: sagemaker-us-west-2-678264136642
Demo Prefix: sagemaker/DEMO-ClarifyModelMonitor-1705692266-3764
Demo S3 key: s3://sagemaker-us-west-2-678264136642/sagemaker/DEMO-ClarifyModelMonitor-1705692266-3764
The transform job will save the results to: s3://sagemaker-us-west-2-678264136642/sagemaker/DEMO-ClarifyModelMonitor-1705692266-3764/transform-output
The transform job will save the captured data to: s3://sagemaker-us-west-2-678264136642/sagemaker/DEMO-ClarifyModelMonitor-1705692266-3764/data-capture
The baselining job will save the analysis results to: s3://sagemaker-us-west-2-678264136642/sagemaker/DEMO-ClarifyModelMonitor-1705692266-3764/baselining-output
The monitor will save the analysis results to: s3://sagemaker-us-west-2-678264136642/sagemaker/DEMO-ClarifyModelMonitor-1705692266-3764/monitor-output

Data files

This example includes two dataset files, both in the JSON format.

[4]:
train_dataset_path = "test_data/validation-dataset.json"
test_dataset_path = "test_data/test-dataset.json"
dataset_type = "application/json"

The train dataset has the features and the ground truth label (pointed to by the key “label”),

[5]:
!head -n 5 $train_dataset_path
{"instances":[{"features":[41,2,220531,14,15,2,9,0,4,1,0,0,60,38],"label":1},{"features":[33,2,35378,9,13,2,11,5,4,0,0,0,45,38],"label":1},{"features":[36,2,223433,12,14,2,11,0,4,1,7688,0,50,38],"label":1},{"features":[40,2,220589,7,12,4,0,1,4,0,0,0,40,38],"label":0},{"features":[30,2,231413,15,10,2,2,0,4,1,0,0,40,38],"label":1},{"features":[33,4,218164,11,9,2,2,0,4,1,0,0,40,38],"label":0},{"features":[42,2,213464,15,10,2,2,0,4,1,0,0,40,38],"label":0},{"features":[20,2,247794,11,9,4,11,1,4,0,0,0,84,38],"label":0},{"features":[43,2,174575,15,10,0,0,1,4,1,0,0,45,38],"label":0},{"features":[42,4,54202,14,15,2,9,0,4,1,0,0,50,38],"label":1},{"features":[27,2,126060,11,9,4,3,1,4,0,0,0,40,38],"label":0},{"features":[25,2,182866,11,9,4,5,3,4,1,0,0,40,38],"label":0},{"features":[43,2,302041,11,9,4,0,1,2,0,0,0,40,38],"label":0},{"features":[30,2,91145,11,9,4,5,4,4,1,0,0,55,38],"label":0},{"features":[41,2,648223,3,2,3,4,4,4,1,0,0,40,25],"label":0},{"features":[60,2,101096,10,16,4,9,1,4,0,0,0,65,38],"label":1},{"features":[45,3,197332,15,10,2,2,0,4,1,0,0,55,38],"label":1},{"features":[42,2,174112,12,14,4,9,1,4,0,0,0,40,38],"label":0},{"features":[36,2,183902,9,13,2,9,5,4,0,0,0,4,38],"label":1},{"features":[76,2,199949,9,13,2,0,0,4,1,20051,0,50,38],"label":1},{"features":[45,0,71823,15,10,2,0,0,2,1,0,0,20,38],"label":0},{"features":[37,2,147258,6,5,2,6,0,4,1,0,0,50,38],"label":1},{"features":[41,2,119079,11,9,2,11,0,4,1,0,0,49,38],"label":1},{"features":[38,2,193961,15,10,2,2,0,1,1,0,0,40,29],"label":1},{"features":[76,2,125784,9,13,2,3,0,4,1,0,0,40,38],"label":0},{"features":[45,2,155659,9,13,2,9,0,4,1,0,0,60,38],"label":1},{"features":[30,2,345122,14,15,2,9,0,4,1,0,0,50,38],"label":0},{"features":[30,2,171598,9,13,3,11,1,4,0,0,0,50,38],"label":0},{"features":[58,3,78104,15,10,2,3,0,4,1,7298,0,60,38],"label":1},{"features":[37,2,224541,15,10,2,13,0,4,1,0,0,40,38],"label":0},{"features":[17,2,369909,0,6,4,7,3,4,1,0,0,20,38],"label":0},{"features":[45,2,204205,5,4,0,6,1,4,1,0,0,48,38],"label":0},{"features":[64,2,180401,0,6,2,13,0,4,1,0,0,40,38],"label":1},{"features":[49,2,129513,11,9,2,13,0,4,1,0,0,50,38],"label":1},{"features":[23,2,125491,15,10,4,7,1,1,0,0,0,35,39],"label":0},{"features":[20,0,410446,11,9,4,0,2,4,1,0,0,20,38],"label":0},{"features":[51,2,259323,9,13,2,3,0,4,1,0,0,50,38],"label":1},{"features":[44,2,206686,15,10,0,0,4,4,0,0,0,40,38],"label":0},{"features":[22,2,106700,7,12,4,0,3,4,0,0,0,27,38],"label":0},{"features":[47,2,185041,15,10,2,2,0,4,1,7298,0,40,38],"label":1},{"features":[30,2,327202,2,8,4,2,1,2,1,0,0,40,38],"label":0},{"features":[35,2,136343,11,9,4,11,1,4,1,0,0,40,38],"label":0},{"features":[47,1,287320,12,14,4,9,1,4,1,0,0,40,38],"label":0},{"features":[27,5,553473,9,13,2,10,5,2,0,0,0,48,38],"label":0},{"features":[43,2,462180,14,15,2,9,0,4,1,99999,0,60,38],"label":1},{"features":[49,1,34021,9,13,4,9,3,4,0,0,0,50,38],"label":0},{"features":[43,2,350379,4,3,0,8,4,4,0,0,0,40,25],"label":0},{"features":[44,2,174283,11,9,2,2,0,4,1,0,0,40,38],"label":1},{"features":[39,2,164733,15,10,0,0,1,4,0,0,0,45,38],"label":0},{"features":[37,2,124293,15,10,2,0,0,4,1,0,0,50,38],"label":0},{"features":[36,1,110791,7,12,5,0,4,4,0,0,0,40,38],"label":0},{"features":[26,2,195994,15,10,4,11,1,4,0,0,0,15,38],"label":0},{"features":[52,4,72257,15,10,2,11,0,4,1,0,0,50,38],"label":0},{"features":[20,2,231981,15,10,4,13,1,4,1,0,0,32,38],"label":0},{"features":[43,2,346321,12,14,2,9,0,4,1,0,0,45,38],"label":1},{"features":[28,2,412149,0,6,4,4,2,4,1,0,0,35,25],"label":0},{"features":[61,2,128848,11,9,2,6,0,4,1,3471,0,40,38],"label":0},{"features":[46,3,168796,9,13,2,11,0,4,1,0,0,55,38],"label":0},{"features":[36,2,185099,14,15,2,9,0,4,1,0,0,55,38],"label":1},{"features":[40,3,50644,7,12,0,11,4,4,0,1506,0,40,38],"label":0},{"features":[32,2,340917,11,9,4,5,1,4,1,0,0,40,38],"label":0},{"features":[46,2,175625,14,15,0,9,4,4,0,0,0,40,38],"label":0},{"features":[43,2,216697,15,10,2,10,0,3,1,0,0,32,38],"label":0},{"features":[36,2,389725,15,10,0,0,1,4,1,0,0,45,38],"label":0},{"features":[28,4,192838,8,11,2,2,0,4,1,0,0,45,38],"label":0},{"features":[55,0,35723,12,14,2,3,0,4,1,0,0,60,38],"label":1},{"features":[39,2,270059,15,10,0,0,4,4,0,0,0,35,38],"label":0},{"features":[44,2,116825,14,15,2,9,0,4,1,15024,0,80,38],"label":1},{"features":[23,1,324637,15,10,4,0,1,4,1,0,0,30,38],"label":0},{"features":[28,2,160731,11,9,2,2,0,4,1,0,0,40,30],"label":1},{"features":[53,1,216931,15,10,2,10,0,4,1,4386,0,40,38],"label":1},{"features":[59,2,243226,0,6,0,6,1,4,0,0,0,40,38],"label":0},{"features":[19,2,63918,15,10,4,0,1,4,1,0,0,40,38],"label":0},{"features":[38,2,52963,9,13,4,0,1,4,0,0,0,50,38],"label":0},{"features":[17,2,268276,2,8,4,7,3,4,1,0,0,12,38],"label":0},{"features":[39,2,114079,7,12,4,2,1,4,1,0,0,40,38],"label":0},{"features":[61,2,130684,15,10,2,9,0,4,1,0,0,42,38],"label":0},{"features":[37,2,245053,15,10,0,5,3,4,1,0,1504,40,38],"label":0},{"features":[40,2,53835,9,13,2,11,0,4,1,0,0,50,38],"label":1},{"features":[41,2,225892,15,10,2,2,0,4,1,0,0,48,38],"label":1},{"features":[31,2,131425,9,13,2,2,0,4,1,0,0,40,38],"label":0},{"features":[40,2,71305,11,9,2,7,0,2,1,0,0,40,38],"label":0},{"features":[46,0,167381,11,9,2,0,5,4,0,0,0,40,38],"label":1},{"features":[45,2,187730,9,13,4,9,3,4,1,0,0,40,38],"label":0},{"features":[48,2,95661,15,10,4,0,1,4,0,0,0,43,38],"label":0},{"features":[39,2,150217,15,10,0,11,1,4,0,0,0,38,38],"label":0},{"features":[28,5,37250,9,13,4,9,3,4,1,0,0,16,38],"label":0},{"features":[18,2,27920,1,7,4,3,3,4,0,0,0,25,38],"label":0},{"features":[22,2,129172,15,10,4,7,3,4,1,0,0,16,38],"label":0},{"features":[28,2,138054,7,12,4,7,1,3,1,0,0,40,38],"label":0},{"features":[50,2,33304,11,9,2,2,0,4,1,0,0,40,38],"label":1},{"features":[52,2,110977,10,16,4,3,1,4,1,0,0,40,38],"label":1},{"features":[50,2,172175,14,15,2,9,0,4,1,0,0,50,38],"label":1},{"features":[37,3,107164,0,6,4,13,1,4,1,0,2559,50,38],"label":1},{"features":[38,2,160808,11,9,2,2,0,2,1,4386,0,48,38],"label":0},{"features":[57,3,51016,11,9,2,3,0,4,1,0,0,60,38],"label":1},{"features":[34,2,253438,15,10,2,3,0,4,1,0,0,60,38],"label":1},{"features":[38,2,185330,15,10,4,2,3,4,0,0,0,25,38],"label":0},{"features":[33,4,24504,11,9,5,2,2,4,1,0,0,50,38],"label":0},{"features":[37,2,278632,6,5,2,13,0,4,1,0,0,40,38],"label":0},{"features":[66,5,102640,11,9,6,9,4,2,0,0,0,35,38],"label":0},{"features":[35,2,168675,11,9,5,13,3,4,1,0,0,50,38],"label":0},{"features":[37,3,86459,7,12,5,3,4,4,1,0,0,50,38],"label":0},{"features":[51,2,138847,9,13,2,3,0,4,1,0,0,40,38],"label":1},{"features":[36,2,163290,15,10,0,11,4,4,0,0,0,40,38],"label":0},{"features":[33,2,134886,15,10,4,0,3,4,0,99999,0,30,38],"label":1},{"features":[50,2,271262,11,9,2,13,0,4,1,0,0,40,38],"label":1},{"features":[37,2,186191,11,9,2,6,0,4,1,0,0,46,38],"label":0},{"features":[59,2,261816,15,10,0,3,1,4,0,0,0,52,27],"label":0},{"features":[63,2,174018,15,10,2,11,0,2,1,0,0,40,38],"label":1},{"features":[33,2,124827,11,9,2,13,0,4,1,0,0,40,38],"label":0},{"features":[39,2,318416,0,6,5,7,3,2,0,0,0,12,38],"label":0},{"features":[36,2,214816,11,9,4,2,1,4,0,0,0,40,38],"label":0},{"features":[50,2,34832,9,13,2,12,0,4,1,15024,0,40,38],"label":1},{"features":[29,2,413297,7,12,4,11,1,4,1,0,0,45,25],"label":0},{"features":[44,2,68748,15,10,2,11,0,4,1,0,0,48,38],"label":0},{"features":[47,5,156417,15,10,0,9,4,4,1,0,0,20,38],"label":0},{"features":[26,2,302603,11,9,4,13,3,4,1,0,0,45,38],"label":0},{"features":[58,4,106942,15,10,0,2,4,4,1,0,0,40,38],"label":0},{"features":[28,2,203776,0,6,2,2,0,4,1,0,0,50,38],"label":0},{"features":[17,1,173497,1,7,4,9,3,2,1,0,0,15,38],"label":0},{"features":[66,0,47358,0,6,2,2,0,4,1,3471,0,40,38],"label":0},{"features":[50,2,174102,11,9,0,2,3,4,1,0,0,40,32],"label":0},{"features":[33,2,119176,15,10,6,0,4,4,0,0,0,40,38],"label":0},{"features":[36,4,219611,9,13,4,11,1,2,0,2174,0,50,38],"label":0},{"features":[48,2,102102,8,11,2,12,0,4,1,0,0,50,38],"label":1},{"features":[20,2,157541,15,10,4,2,3,4,1,0,0,40,38],"label":0},{"features":[68,2,218637,15,10,2,11,0,4,1,0,2377,55,38],"label":1},{"features":[27,2,198258,9,13,4,11,3,4,1,0,0,35,38],"label":0},{"features":[29,2,110134,15,10,0,6,1,4,1,0,0,40,38],"label":0},{"features":[65,5,29276,5,4,6,7,2,4,0,0,0,24,38],"label":0},{"features":[38,2,33001,9,13,2,3,0,4,1,0,0,55,38],"label":1},{"features":[43,4,277647,11,9,2,3,0,4,1,0,0,35,38],"label":0},{"features":[39,2,214816,9,13,2,3,0,4,1,0,0,60,38],"label":0},{"features":[52,4,237868,15,10,4,0,4,4,1,0,0,5,38],"label":0},{"features":[52,0,30731,9,13,2,3,0,4,1,0,0,45,38],"label":1},{"features":[29,2,228346,8,11,4,2,1,4,1,0,0,50,38],"label":0},{"features":[52,1,199995,12,14,2,3,0,4,1,7298,0,60,38],"label":1},{"features":[46,0,31141,15,10,0,13,1,4,1,0,0,40,38],"label":0},{"features":[42,2,231813,1,7,2,13,0,4,1,0,0,40,38],"label":0},{"features":[39,2,272950,9,13,2,2,0,4,1,0,0,45,38],"label":1},{"features":[36,2,182074,15,10,0,0,1,4,1,0,0,45,38],"label":0},{"features":[54,2,118793,11,9,2,0,0,4,1,0,0,45,38],"label":0},{"features":[28,2,207513,11,9,4,11,3,4,1,0,0,48,38],"label":0},{"features":[54,2,97778,5,4,2,2,0,4,1,0,0,40,38],"label":0},{"features":[33,2,217460,11,9,2,11,0,4,1,0,0,60,38],"label":1},{"features":[90,2,221832,9,13,2,3,0,4,1,0,0,45,38],"label":0},{"features":[57,5,109015,2,8,0,7,4,4,0,0,0,40,38],"label":0},{"features":[29,2,40083,10,16,4,9,1,4,1,0,0,40,1],"label":0},{"features":[25,2,188767,11,9,4,2,3,4,1,0,0,40,38],"label":0},{"features":[30,2,154568,9,13,2,2,0,1,1,0,0,36,39],"label":1},{"features":[38,2,161016,15,10,0,9,1,4,0,0,0,32,38],"label":0},{"features":[22,2,117789,15,10,4,9,3,4,0,0,0,10,38],"label":0},{"features":[26,5,294400,11,9,2,10,0,4,1,0,0,38,38],"label":0},{"features":[41,2,168293,12,14,0,3,4,4,0,0,0,45,38],"label":0},{"features":[29,4,164607,8,11,2,4,0,4,1,0,0,50,38],"label":0},{"features":[51,5,226885,11,9,4,13,1,4,1,0,0,40,38],"label":0},{"features":[76,4,117169,5,4,4,4,1,4,1,0,0,30,38],"label":0},{"features":[22,2,184756,15,10,4,11,3,4,0,0,0,30,38],"label":0},{"features":[49,2,248895,11,9,2,6,0,4,1,0,0,45,38],"label":0},{"features":[36,4,257250,8,11,2,4,0,4,1,0,0,99,38],"label":0},{"features":[61,4,133969,11,9,2,11,0,1,1,0,0,63,34],"label":0},{"features":[31,2,236599,9,13,2,3,0,4,1,0,0,45,38],"label":1},{"features":[22,2,150175,15,10,4,0,3,4,0,0,0,20,38],"label":0},{"features":[25,2,191921,15,10,4,13,3,4,1,0,0,40,38],"label":0},{"features":[56,2,170324,4,3,2,2,0,2,1,0,0,40,37],"label":0},{"features":[35,2,107125,9,13,2,9,0,4,1,0,0,16,38],"label":1},{"features":[62,2,103344,9,13,6,3,1,4,1,10520,0,50,38],"label":1},{"features":[24,1,317443,9,13,2,9,5,2,0,0,0,40,38],"label":0},{"features":[22,2,341227,15,10,4,0,1,4,1,0,0,20,38],"label":0},{"features":[25,2,290528,11,9,2,6,0,4,1,0,0,40,38],"label":0},{"features":[27,2,198286,15,10,4,7,1,4,0,0,0,34,38],"label":0},{"features":[64,2,256466,11,9,2,12,0,1,1,0,0,60,29],"label":1},{"features":[32,1,223267,11,9,2,13,0,4,1,0,0,40,38],"label":0},{"features":[32,2,388672,15,10,0,5,1,4,1,0,0,16,38],"label":0},{"features":[24,2,509629,11,9,4,7,3,4,0,0,0,25,38],"label":0},{"features":[21,2,191460,1,7,4,7,4,2,0,0,0,40,38],"label":0},{"features":[54,2,90363,7,12,2,3,0,4,1,0,0,40,38],"label":1},{"features":[49,2,192323,11,9,2,6,0,4,1,0,0,40,38],"label":0},{"features":[36,2,218490,8,11,2,11,0,4,1,0,0,60,38],"label":0},{"features":[24,2,159580,9,13,4,7,3,2,0,0,0,75,38],"label":0},{"features":[56,2,220187,15,10,2,11,0,4,1,0,0,45,38],"label":1},{"features":[52,2,218550,15,10,3,0,1,4,0,14084,0,16,38],"label":1},{"features":[68,2,195868,9,13,2,11,0,4,1,20051,0,40,38],"label":1},{"features":[44,2,151780,15,10,6,3,1,2,0,0,0,40,38],"label":0},{"features":[58,2,190747,11,9,2,6,0,4,1,0,0,40,38],"label":0},{"features":[29,4,142519,11,9,2,6,0,4,1,0,0,40,38],"label":0},{"features":[73,1,205580,4,3,2,9,0,4,1,0,0,6,38],"label":0},{"features":[58,3,78634,1,7,2,13,0,4,1,0,0,60,38],"label":0},{"features":[21,2,314182,11,9,4,7,1,4,0,0,0,40,38],"label":0},{"features":[44,2,297991,7,12,4,3,1,1,0,0,0,50,38],"label":0},{"features":[36,2,186110,15,10,2,13,0,4,1,0,0,40,38],"label":0},{"features":[46,4,31267,11,9,2,13,0,4,1,0,0,50,38],"label":0},{"features":[34,2,57426,9,13,4,11,1,4,1,0,0,45,38],"label":0},{"features":[21,2,107882,7,12,4,7,3,4,0,0,0,9,38],"label":0},{"features":[58,5,194068,12,14,2,9,0,4,1,0,1977,50,38],"label":1},{"features":[22,2,332194,15,10,4,7,3,2,1,0,0,40,38],"label":0},{"features":[65,3,115922,9,13,2,3,0,4,1,0,0,40,38],"label":1},{"features":[27,2,302406,15,10,2,11,0,4,1,0,0,40,38],"label":1},{"features":[37,2,270059,15,10,0,0,4,4,0,25236,0,25,38],"label":1},{"features":[40,2,375603,11,9,0,0,4,2,1,0,0,40,38],"label":0},{"features":[24,2,456460,7,12,2,0,5,4,0,0,0,40,38],"label":0},{"features":[35,2,202397,9,13,2,2,0,1,1,0,0,40,29],"label":1},{"features":[35,4,120066,15,10,2,2,0,0,1,0,0,60,38],"label":0},{"features":[33,2,197424,11,9,2,3,0,4,1,5013,0,40,38],"label":0},{"features":[36,4,67728,9,13,2,11,0,4,1,0,0,50,38],"label":1},{"features":[23,2,99543,2,8,4,13,1,4,1,0,0,46,38],"label":0},{"features":[49,3,229737,14,15,2,9,0,4,1,99999,0,37,38],"label":1},{"features":[62,2,194167,11,9,0,6,1,4,0,2174,0,40,38],"label":0},{"features":[34,2,188096,11,9,4,0,1,4,0,0,0,36,38],"label":0},{"features":[40,2,338740,11,9,2,3,0,4,1,0,0,40,38],"label":0},{"features":[24,2,275691,1,7,4,13,3,4,1,0,0,39,38],"label":0},{"features":[17,2,220384,1,7,4,0,3,4,1,0,0,15,38],"label":0},{"features":[51,2,302146,1,7,4,7,1,2,0,0,0,40,38],"label":0},{"features":[31,0,166626,11,9,2,0,0,4,1,0,0,40,38],"label":1},{"features":[52,2,145271,9,13,2,2,0,1,1,0,0,40,38],"label":0},{"features":[30,2,95299,11,9,2,6,0,1,1,0,0,40,39],"label":1},{"features":[28,2,31801,11,9,4,5,2,4,1,0,0,60,38],"label":0},{"features":[24,2,228613,1,7,4,6,4,4,0,0,0,40,38],"label":0},{"features":[40,2,234633,15,10,4,2,1,4,1,0,0,40,38],"label":0},{"features":[26,2,146343,15,10,2,11,5,2,0,0,0,40,38],"label":0},{"features":[42,2,331651,12,14,4,9,1,4,0,8614,0,50,38],"label":1},{"features":[26,2,167106,11,9,4,2,2,1,1,0,0,40,16],"label":0},{"features":[27,0,196386,7,12,2,0,0,4,1,4064,0,40,7],"label":0},{"features":[28,1,146949,11,9,2,5,0,4,1,0,0,40,38],"label":0},{"features":[36,2,47310,11,9,4,7,1,2,0,0,0,40,38],"label":0},{"features":[45,1,192793,15,10,2,10,0,4,1,0,0,40,38],"label":1},{"features":[29,2,535978,15,10,2,2,0,4,1,0,0,45,38],"label":0},{"features":[22,2,324922,11,9,4,6,1,4,1,0,0,50,38],"label":0},{"features":[47,2,155489,11,9,2,13,0,4,1,7688,0,55,38],"label":1},{"features":[39,5,85566,9,13,2,9,0,4,1,0,0,40,38],"label":0},{"features":[24,2,385540,11,9,2,11,0,4,1,0,0,40,25],"label":0},{"features":[39,2,167140,12,14,2,3,0,4,1,0,0,40,38],"label":0},{"features":[39,2,347960,14,15,4,9,1,4,0,14084,0,35,38],"label":1},{"features":[51,2,180807,15,10,0,3,4,4,0,0,0,40,38],"label":0},{"features":[24,2,310380,15,10,3,0,3,2,0,0,0,45,38],"label":0},{"features":[55,2,271710,15,10,4,0,1,4,1,0,0,45,38],"label":0},{"features":[32,0,191385,7,12,0,10,1,4,1,2174,0,40,38],"label":0},{"features":[22,2,320451,15,10,4,10,3,1,1,0,0,24,18],"label":0},{"features":[59,2,277034,11,9,0,12,4,4,1,0,0,60,38],"label":1},{"features":[24,2,403865,15,10,2,2,0,4,1,0,0,56,38],"label":0},{"features":[41,5,47170,9,13,2,9,5,0,0,0,0,48,38],"label":1},{"features":[40,2,273308,11,9,0,6,4,4,0,0,0,48,25],"label":0},{"features":[57,4,152030,15,10,2,11,5,4,0,0,0,25,38],"label":1},{"features":[36,2,194905,9,13,6,9,4,4,0,0,0,44,38],"label":0},{"features":[31,4,229946,11,9,2,9,0,4,1,0,0,40,3],"label":0},{"features":[28,2,119793,8,11,0,3,1,4,1,10520,0,50,38],"label":1},{"features":[38,2,143538,11,9,4,6,1,4,0,0,0,40,38],"label":0},{"features":[28,2,108574,15,10,2,0,5,4,0,0,0,15,38],"label":0},{"features":[32,2,194141,11,9,0,6,3,4,1,0,0,50,38],"label":0},{"features":[49,4,107597,11,9,0,3,4,4,0,14084,0,30,38],"label":1},{"features":[37,2,186035,7,12,2,2,0,4,1,0,0,55,38],"label":0},{"features":[50,2,263200,4,3,3,7,4,4,0,0,0,34,25],"label":0},{"features":[37,2,70562,3,2,4,7,4,4,0,0,0,48,7],"label":0},{"features":[38,2,195686,15,10,2,2,0,4,1,0,0,40,38],"label":1},{"features":[44,1,197919,15,10,0,7,4,4,0,0,0,40,38],"label":0},{"features":[30,4,261943,1,7,3,2,1,4,1,0,0,30,15],"label":0},{"features":[20,3,95997,11,9,4,4,3,4,1,0,0,70,38],"label":0},{"features":[32,2,151773,15,10,2,2,0,4,1,0,0,45,38],"label":0},{"features":[56,2,177271,8,11,2,12,0,4,1,0,0,40,38],"label":1},{"features":[24,2,537222,11,9,2,3,0,4,1,0,0,50,38],"label":0},{"features":[59,2,196482,11,9,6,0,4,4,0,0,0,40,38],"label":0},{"features":[24,2,43323,11,9,4,7,1,4,0,0,1762,40,38],"label":0},{"features":[40,2,259307,12,14,2,3,0,4,1,0,0,50,38],"label":1},{"features":[35,2,167990,6,5,2,6,0,4,1,0,0,40,1],"label":0},{"features":[32,2,158416,11,9,0,11,1,4,1,0,0,50,38],"label":0},{"features":[27,2,199903,9,13,4,9,1,4,0,0,0,40,38],"label":0},{"features":[44,2,210534,4,3,2,5,0,4,1,0,0,40,25],"label":0},{"features":[50,2,128798,9,13,2,12,0,4,1,0,0,40,38],"label":1},{"features":[17,2,176467,6,5,4,13,1,4,1,0,0,20,38],"label":0},{"features":[29,2,153805,11,9,4,6,2,3,1,0,0,40,6],"label":0},{"features":[23,2,238917,5,4,4,2,2,4,1,0,0,36,38],"label":0},{"features":[69,5,34339,11,9,2,10,0,4,1,0,0,40,38],"label":0},{"features":[34,2,205733,11,9,4,0,1,4,0,0,0,40,38],"label":0},{"features":[29,2,193152,11,9,4,5,1,4,1,0,1408,40,38],"label":0},{"features":[35,2,191628,15,10,2,9,0,4,1,0,0,40,38],"label":0},{"features":[17,2,51939,1,7,4,11,3,4,0,0,0,15,38],"label":0},{"features":[34,3,80249,15,10,2,4,0,4,1,0,0,72,38],"label":0},{"features":[50,2,162632,11,9,2,3,0,4,1,0,0,45,38],"label":0},{"features":[21,2,292264,11,9,4,2,1,4,1,0,0,35,38],"label":0},{"features":[40,2,224799,9,13,2,9,0,4,1,0,0,45,38],"label":0},{"features":[37,2,194004,1,7,2,2,0,4,1,0,0,25,38],"label":0},{"features":[32,2,188245,1,7,4,8,4,2,0,0,0,40,38],"label":0},{"features":[49,3,201498,11,9,2,2,0,4,1,0,0,40,38],"label":0},{"features":[33,5,313729,12,14,4,9,1,4,1,0,0,60,38],"label":0},{"features":[19,2,172893,15,10,4,3,3,4,0,0,0,30,38],"label":0},{"features":[41,2,252058,9,13,4,0,1,4,1,0,0,40,38],"label":0},{"features":[39,2,188540,11,9,0,3,1,4,1,0,0,45,38],"label":0},{"features":[47,2,168232,9,13,2,0,0,4,1,7298,0,40,38],"label":1},{"features":[58,2,199278,9,13,0,3,1,4,1,0,0,38,38],"label":0},{"features":[41,2,104334,15,10,2,11,0,4,1,0,0,50,38],"label":1},{"features":[24,2,281221,9,13,4,0,2,1,0,0,0,40,35],"label":0},{"features":[23,2,197613,15,10,4,0,1,4,0,0,0,40,38],"label":0},{"features":[33,2,229716,11,9,0,0,1,4,1,0,0,38,38],"label":0},{"features":[30,2,255279,11,9,0,0,4,4,0,0,0,20,38],"label":0},{"features":[25,2,282063,5,4,2,5,0,4,1,0,0,40,25],"label":0},{"features":[40,2,105936,9,13,0,9,1,4,0,0,0,40,38],"label":0},{"features":[39,2,32146,15,10,4,2,1,4,1,0,0,40,38],"label":0},{"features":[29,2,118230,11,9,4,11,1,4,0,0,0,35,38],"label":0},{"features":[43,5,115005,11,9,0,12,1,4,0,0,0,40,38],"label":0},{"features":[26,2,190469,9,13,4,12,1,4,1,0,0,40,38],"label":0},{"features":[35,2,347491,8,11,4,2,1,4,1,0,0,40,38],"label":0},{"features":[23,2,45834,9,13,4,3,1,4,0,0,0,50,38],"label":0},{"features":[20,2,237305,15,10,4,6,2,2,0,0,0,35,38],"label":0},{"features":[48,2,160647,15,10,4,3,1,4,0,0,0,40,20],"label":1},{"features":[31,2,241885,11,9,4,4,4,4,1,0,0,45,38],"label":0},{"features":[47,2,108510,0,6,2,11,0,4,1,0,0,65,38],"label":0},{"features":[55,0,189985,15,10,0,0,4,2,0,0,0,40,38],"label":0},{"features":[23,2,201145,11,9,4,2,1,4,1,0,0,65,38],"label":0},{"features":[45,2,167187,9,13,4,9,1,4,0,0,0,40,38],"label":1},{"features":[63,3,272425,8,11,2,3,0,4,1,0,0,40,38],"label":1},{"features":[41,2,49797,11,9,2,2,0,4,1,0,0,40,38],"label":0},{"features":[30,2,381153,11,9,4,2,1,4,1,0,0,40,38],"label":0},{"features":[33,2,170148,11,9,0,0,4,4,0,0,0,45,38],"label":0},{"features":[27,2,113054,11,9,5,6,1,4,1,0,0,43,38],"label":0},{"features":[62,2,319582,11,9,6,11,1,4,0,0,0,32,38],"label":0},{"features":[24,2,289448,8,11,4,0,3,1,0,0,0,40,29],"label":0},{"features":[44,2,277488,15,10,2,6,0,4,1,3103,0,40,38],"label":1},{"features":[25,2,371987,11,9,0,0,1,4,0,0,0,40,38],"label":0},{"features":[39,2,509060,15,10,0,7,1,4,1,0,0,40,38],"label":0},{"features":[17,2,211870,6,5,4,7,1,4,1,0,0,6,38],"label":0},{"features":[29,2,131088,11,9,4,5,3,4,1,0,0,25,38],"label":0},{"features":[42,5,222884,9,13,0,0,1,4,1,0,0,40,38],"label":0},{"features":[25,2,124590,11,9,4,3,2,4,1,0,0,40,38],"label":0},{"features":[60,2,88055,0,6,2,13,0,4,1,0,0,40,38],"label":0},{"features":[23,2,184255,11,9,2,11,5,4,0,0,0,40,38],"label":0},{"features":[28,2,66434,0,6,4,7,4,4,0,0,0,15,38],"label":0},{"features":[31,2,118551,6,5,0,0,1,4,0,0,0,40,38],"label":0},{"features":[41,4,26598,11,9,0,2,1,4,1,0,0,40,38],"label":0},{"features":[28,2,157391,9,13,4,11,3,4,0,0,0,40,38],"label":0},{"features":[45,4,275445,9,13,0,3,4,4,1,0,0,50,38],"label":0},{"features":[19,2,100999,9,13,4,9,3,4,0,0,0,30,38],"label":0},{"features":[19,4,206599,15,10,4,7,3,4,0,0,0,22,38],"label":0},{"features":[25,1,197728,9,13,4,3,1,4,0,0,0,20,38],"label":0},{"features":[48,2,123075,10,16,2,9,0,4,1,0,0,45,38],"label":1},{"features":[37,1,117760,8,11,4,10,1,4,1,4650,0,40,38],"label":0},{"features":[44,2,230684,9,13,2,3,0,4,1,7688,0,50,38],"label":1},{"features":[24,2,22201,11,9,2,10,0,1,1,0,0,40,36],"label":0},{"features":[62,4,159939,11,9,2,4,0,4,1,0,0,35,38],"label":0},{"features":[57,1,118481,9,13,2,9,0,4,1,0,1902,40,38],"label":1},{"features":[51,2,239155,8,11,0,7,1,4,1,0,0,40,38],"label":0},{"features":[37,2,67125,11,9,0,11,1,4,1,0,0,60,38],"label":0},{"features":[19,2,255161,11,9,4,11,3,4,1,0,0,25,38],"label":0},{"features":[30,2,243841,11,9,0,7,2,1,0,0,0,40,34],"label":0},{"features":[27,2,91501,11,9,2,12,5,4,0,0,0,40,38],"label":0},{"features":[60,2,232242,11,9,2,11,0,4,1,0,0,40,38],"label":0},{"features":[26,2,104746,11,9,2,2,0,4,1,5013,0,60,38],"label":0},{"features":[19,2,72355,15,10,4,7,1,4,1,0,0,20,38],"label":0},{"features":[22,2,203182,9,13,4,3,4,4,0,0,0,30,38],"label":0},{"features":[50,5,173020,15,10,2,2,0,4,1,0,0,40,38],"label":0},{"features":[17,2,276718,11,9,4,0,3,4,1,0,0,20,38],"label":0},{"features":[61,1,95450,9,13,2,3,0,4,1,5178,0,50,38],"label":1},{"features":[28,2,312588,0,6,0,7,1,4,0,0,0,40,38],"label":0},{"features":[22,2,284317,7,12,4,0,1,4,0,0,0,40,38],"label":0},{"features":[35,2,185325,9,13,2,9,0,4,1,0,0,50,38],"label":1},{"features":[40,2,149466,11,9,0,5,1,2,1,0,0,35,38],"label":0},{"features":[32,2,114746,11,9,5,5,4,1,0,0,0,60,34],"label":0},{"features":[23,4,208503,15,10,0,0,3,4,1,0,0,40,38],"label":0},{"features":[33,2,290763,15,10,4,11,1,4,0,0,0,40,38],"label":0},{"features":[34,2,37646,7,12,2,2,0,4,1,0,0,65,38],"label":0},{"features":[47,2,334039,9,13,2,3,0,4,1,7298,0,44,38],"label":1},{"features":[51,2,219599,11,9,2,6,5,4,0,0,0,40,38],"label":0},{"features":[36,2,206521,11,9,4,6,1,4,1,0,0,40,38],"label":0},{"features":[46,2,45288,9,13,4,7,1,4,1,0,0,40,38],"label":0},{"features":[17,2,60562,6,5,4,7,3,4,0,0,0,20,38],"label":0},{"features":[47,3,79627,14,15,0,9,1,4,1,27828,0,50,38],"label":1},{"features":[31,2,213002,2,8,4,11,1,4,1,4650,0,50,38],"label":0},{"features":[23,1,210029,15,10,4,0,3,4,0,0,0,20,38],"label":0},{"features":[53,2,79324,11,9,2,2,0,4,1,0,0,40,38],"label":1},{"features":[50,2,137815,11,9,2,13,0,4,1,0,0,60,38],"label":1},{"features":[23,1,157331,9,13,4,9,1,4,0,0,0,40,38],"label":0},{"features":[45,2,43479,15,10,2,13,0,4,1,0,0,48,38],"label":0},{"features":[38,2,183279,15,10,2,3,0,4,1,0,0,44,38],"label":1},{"features":[41,4,150533,14,15,2,9,0,4,1,0,0,50,38],"label":1},{"features":[32,2,27856,15,10,4,0,1,4,0,0,0,40,38],"label":0},{"features":[44,2,123983,9,13,0,7,1,1,1,0,0,40,2],"label":0},{"features":[38,2,198216,15,10,0,3,4,4,0,0,0,40,38],"label":0},{"features":[42,2,33002,11,9,2,3,0,4,1,0,0,48,38],"label":0},{"features":[43,2,115562,9,13,2,9,0,4,1,0,0,42,38],"label":1},{"features":[34,2,300687,11,9,2,2,0,2,1,0,0,40,38],"label":0},{"features":[48,2,287480,12,14,2,12,0,4,1,0,0,40,38],"label":1},{"features":[61,2,146788,5,4,2,13,0,4,1,0,0,40,38],"label":0},{"features":[29,2,452205,11,9,0,7,4,4,0,0,0,36,38],"label":0},{"features":[23,2,182812,15,10,4,7,3,4,0,0,0,40,5],"label":0},{"features":[48,2,192791,11,9,2,6,0,4,1,0,0,40,38],"label":0},{"features":[68,3,182131,15,10,2,3,0,4,1,10605,0,20,38],"label":1},{"features":[23,2,200973,11,9,4,0,1,4,0,0,0,40,38],"label":0},{"features":[45,3,271901,11,9,2,11,0,4,1,0,0,32,38],"label":1},{"features":[22,2,110946,15,10,4,7,1,4,0,0,0,40,38],"label":0},{"features":[49,2,206947,11,9,0,0,1,4,0,0,0,40,38],"label":0},{"features":[25,2,154863,11,9,4,0,4,2,1,0,0,35,38],"label":0},{"features":[56,2,102106,11,9,2,5,0,4,1,0,0,40,38],"label":0},{"features":[53,2,120839,2,8,0,4,3,4,1,0,0,40,38],"label":0},{"features":[29,5,106972,12,14,4,9,1,4,0,0,0,35,38],"label":0},{"features":[60,2,227468,15,10,6,10,1,2,0,0,0,40,38],"label":0},{"features":[25,2,179462,5,4,4,5,4,4,1,0,0,40,38],"label":0},{"features":[46,2,201595,11,9,2,13,0,4,1,0,0,70,38],"label":0},{"features":[17,2,137042,0,6,4,9,3,4,1,0,0,20,38],"label":0},{"features":[50,4,213654,11,9,2,11,0,2,1,0,0,40,38],"label":0},{"features":[54,5,119565,9,13,2,3,0,4,1,0,0,40,32],"label":1},{"features":[28,2,60288,11,9,4,0,3,4,0,0,0,40,38],"label":0},{"features":[34,2,229732,8,11,2,2,0,4,1,0,0,40,38],"label":0},{"features":[22,2,133833,15,10,4,7,3,4,0,0,0,25,38],"label":0},{"features":[29,2,290740,7,12,4,8,1,4,0,0,0,50,38],"label":0},{"features":[49,2,123584,1,7,2,13,0,4,1,0,0,75,38],"label":0},{"features":[40,2,206066,11,9,2,2,0,4,1,0,0,50,38],"label":0},{"features":[38,2,183279,15,10,2,2,0,4,1,0,0,43,38],"label":0},{"features":[34,2,287737,15,10,2,3,5,4,0,0,1485,40,38],"label":1},{"features":[52,2,90189,5,4,0,8,3,2,0,0,0,16,38],"label":0},{"features":[51,2,128143,15,10,2,2,0,4,1,0,0,40,38],"label":1},{"features":[20,2,184779,15,10,4,12,3,4,0,0,0,20,38],"label":0},{"features":[28,2,54243,11,9,0,13,1,4,1,0,0,60,38],"label":0},{"features":[21,2,213015,11,9,4,5,2,2,1,2176,0,40,38],"label":0},{"features":[43,2,240504,11,9,2,5,0,4,1,0,0,40,38],"label":0},{"features":[43,2,236985,11,9,2,2,0,2,1,0,0,40,38],"label":0},{"features":[43,2,154538,7,12,0,2,1,4,1,0,0,40,38],"label":0},{"features":[33,2,159247,9,13,2,9,0,4,1,0,0,40,38],"label":1},{"features":[35,2,171327,11,9,2,2,0,4,1,0,0,40,38],"label":0},{"features":[36,2,342642,12,14,4,3,1,4,1,0,0,15,38],"label":0},{"features":[50,2,34233,11,9,2,4,0,4,1,0,0,50,38],"label":0},{"features":[26,2,196805,15,10,2,13,0,2,1,0,0,65,38],"label":0},{"features":[27,2,262478,11,9,4,4,3,2,1,0,0,30,38],"label":0},{"features":[34,2,184147,11,9,5,11,4,2,0,0,0,20,38],"label":0},{"features":[36,2,29984,2,8,2,13,0,4,1,0,0,40,38],"label":0},{"features":[44,2,210525,9,13,2,9,0,4,1,0,0,40,38],"label":1},{"features":[51,2,237729,15,10,0,0,4,4,0,0,0,40,38],"label":0},{"features":[32,4,173854,9,13,0,9,2,4,1,0,0,35,38],"label":1},{"features":[23,4,184370,11,9,0,7,1,4,0,0,0,40,38],"label":0},{"features":[49,2,281647,12,14,2,3,0,4,1,0,0,45,38],"label":1},{"features":[61,2,54373,15,10,2,11,0,4,1,0,0,40,38],"label":0},{"features":[41,2,154194,11,9,4,11,3,4,0,0,0,40,38],"label":0},{"features":[30,2,48829,11,9,4,11,1,4,0,0,1602,30,38],"label":0},{"features":[52,1,255927,15,10,6,0,1,4,0,0,0,24,38],"label":0},{"features":[41,2,120277,9,13,2,9,0,4,1,0,0,40,38],"label":1},{"features":[39,2,129495,15,10,5,0,4,2,0,0,0,40,38],"label":0},{"features":[30,2,310889,15,10,4,5,1,4,1,0,0,55,38],"label":0},{"features":[72,2,284080,3,2,0,7,1,2,1,0,0,40,38],"label":0},{"features":[27,2,132191,11,9,4,2,1,4,1,0,0,40,38],"label":0},{"features":[45,2,49298,9,13,4,12,3,4,1,0,0,40,38],"label":0},{"features":[42,2,106900,8,11,4,12,1,4,1,0,0,40,38],"label":0},{"features":[23,2,140462,11,9,4,6,3,4,1,0,0,40,38],"label":0},{"features":[37,2,272950,11,9,0,2,1,4,1,0,0,40,38],"label":0},{"features":[43,5,345969,14,15,2,9,0,4,1,0,0,50,38],"label":1},{"features":[46,2,318259,8,11,0,12,2,4,0,0,0,36,38],"label":0},{"features":[32,2,296282,9,13,2,11,0,4,1,0,0,40,38],"label":0},{"features":[20,2,238685,15,10,4,7,1,4,0,0,0,32,38],"label":0},{"features":[21,2,197583,15,10,4,0,3,4,0,0,0,20,38],"label":0},{"features":[34,2,342709,12,14,2,3,0,4,1,0,0,40,38],"label":0},{"features":[27,1,209109,12,14,4,9,3,4,1,0,0,35,38],"label":0},{"features":[38,2,331395,5,4,2,4,0,4,1,3942,0,84,31],"label":0},{"features":[41,1,107327,8,11,0,9,4,4,0,0,0,40,38],"label":0},{"features":[47,4,237731,11,9,2,4,0,4,1,2829,0,65,38],"label":0},{"features":[43,2,260761,11,9,2,6,0,4,1,0,0,40,25],"label":0},{"features":[42,2,154374,9,13,2,3,0,4,1,0,2415,60,38],"label":1},{"features":[27,2,243569,1,7,2,5,0,4,1,3942,0,40,38],"label":0},{"features":[54,1,31533,12,14,2,0,0,4,1,7298,0,40,38],"label":1},{"features":[37,2,36425,11,9,4,7,1,4,0,0,0,40,38],"label":0},{"features":[46,5,192779,9,13,2,3,0,4,1,7688,0,40,38],"label":1},{"features":[52,5,314627,12,14,0,9,1,1,0,0,0,40,38],"label":0},{"features":[74,4,146929,11,9,2,11,0,4,1,0,0,55,38],"label":0},{"features":[55,2,49996,1,7,4,6,1,2,0,0,0,40,38],"label":0},{"features":[35,1,190964,9,13,2,2,0,4,1,0,0,40,38],"label":0},{"features":[66,2,185336,11,9,6,11,2,4,0,0,0,35,38],"label":0},{"features":[51,1,175750,11,9,0,13,4,2,1,0,0,40,38],"label":0},{"features":[56,2,219762,11,9,2,11,5,4,0,0,0,35,38],"label":0},{"features":[33,2,155343,11,9,2,11,0,4,1,3103,0,40,38],"label":1},{"features":[36,1,28996,11,9,2,13,0,4,1,0,0,40,38],"label":0},{"features":[46,2,98012,8,11,0,0,1,4,0,0,0,40,38],"label":0},{"features":[50,4,105010,11,9,2,4,0,4,1,0,2051,20,38],"label":0},{"features":[52,2,29658,11,9,2,0,0,4,1,0,0,40,38],"label":0},{"features":[56,2,275236,9,13,2,6,0,4,1,0,0,40,38],"label":0},{"features":[29,2,161155,7,12,2,9,0,4,1,0,0,50,38],"label":0},{"features":[20,2,235442,15,10,4,7,1,4,1,0,0,35,38],"label":0},{"features":[30,2,206051,11,9,2,13,0,4,1,0,0,40,38],"label":0},{"features":[55,2,37438,8,11,2,2,0,4,1,0,0,40,38],"label":1},{"features":[60,2,162947,4,3,0,6,1,4,0,0,0,40,32],"label":0},{"features":[39,2,147548,11,9,2,2,0,4,1,0,0,40,38],"label":0},{"features":[50,2,159650,15,10,2,12,0,4,1,0,0,60,38],"label":1},{"features":[35,2,86648,14,15,2,9,0,4,1,7688,0,50,38],"label":1},{"features":[24,5,61737,9,13,4,9,1,4,1,0,0,40,38],"label":0},{"features":[33,1,70164,9,13,4,9,1,0,1,0,0,60,38],"label":0},{"features":[39,2,129597,9,13,2,11,0,4,1,3464,0,40,38],"label":0},{"features":[27,0,47907,9,13,4,0,1,4,0,0,0,40,38],"label":0},{"features":[39,2,150061,12,14,0,3,4,2,0,15020,0,60,38],"label":1},{"features":[51,2,55507,11,9,2,2,0,2,1,0,0,40,38],"label":0},{"features":[53,0,271544,11,9,2,0,0,2,1,0,1977,40,38],"label":1},{"features":[22,2,188950,15,10,4,12,3,4,1,0,0,40,38],"label":0},{"features":[44,2,252202,11,9,0,0,1,4,0,0,0,40,38],"label":0},{"features":[42,2,173590,15,10,2,0,0,4,1,0,1628,40,38],"label":0},{"features":[33,2,105370,11,9,0,10,1,4,1,0,0,70,38],"label":0},{"features":[46,2,162030,11,9,6,0,4,4,0,0,0,43,38],"label":0},{"features":[19,2,86150,1,7,4,11,3,1,0,0,0,19,29],"label":0},{"features":[18,2,25837,1,7,4,9,3,4,1,0,0,15,38],"label":0},{"features":[62,4,173631,15,10,2,3,0,4,1,0,0,70,38],"label":0},{"features":[81,2,100675,3,2,2,9,0,4,1,0,0,15,30],"label":0},{"features":[24,5,184216,15,10,4,0,3,4,0,0,0,40,38],"label":0},{"features":[20,2,38001,15,10,4,7,3,4,0,0,0,20,38],"label":0},{"features":[18,2,123714,1,7,4,5,1,2,1,0,0,40,38],"label":0},{"features":[21,2,256356,1,7,4,8,2,4,0,0,0,40,25],"label":0},{"features":[30,2,75573,9,13,4,3,1,4,0,0,0,45,10],"label":0},{"features":[53,2,31588,9,13,2,9,0,4,1,0,0,52,38],"label":1},{"features":[45,2,265097,11,9,2,7,0,4,1,0,1902,40,38],"label":1},{"features":[61,5,159908,1,7,6,7,4,4,0,0,0,32,38],"label":1},{"features":[24,3,142404,9,13,2,3,0,4,1,0,0,40,38],"label":1},{"features":[29,2,55390,7,12,4,12,1,4,1,0,0,45,38],"label":0},{"features":[20,2,49179,15,10,4,9,1,4,1,0,0,35,38],"label":0},{"features":[31,2,209448,0,6,2,4,0,4,1,2105,0,40,25],"label":0},{"features":[54,2,138944,11,9,2,11,0,4,1,0,0,44,38],"label":0},{"features":[24,2,181820,15,10,4,0,3,4,1,0,0,40,38],"label":0},{"features":[46,2,101430,1,7,0,5,4,2,0,0,0,40,38],"label":0},{"features":[27,2,238859,8,11,4,2,1,4,1,0,0,40,38],"label":0},{"features":[19,2,318822,15,10,4,0,2,4,0,0,0,40,38],"label":0},{"features":[30,2,174789,7,12,2,3,0,4,1,0,1848,50,38],"label":1},{"features":[17,2,146268,0,6,4,7,3,4,0,0,0,10,38],"label":0},{"features":[58,2,142158,9,13,0,3,4,4,0,0,0,35,38],"label":0},{"features":[42,2,510072,11,9,2,2,0,4,1,0,0,40,38],"label":1},{"features":[32,2,257043,11,9,4,0,1,4,0,0,0,42,38],"label":0},{"features":[58,2,127264,0,6,2,2,0,4,1,0,0,50,38],"label":0},{"features":[27,2,93021,11,9,4,0,4,3,0,0,0,40,38],"label":0},{"features":[56,2,282023,14,15,2,9,0,4,1,0,0,45,38],"label":1},{"features":[35,2,162601,11,9,0,0,4,4,0,0,0,40,38],"label":0},{"features":[41,4,147110,11,9,2,6,0,4,1,0,0,25,38],"label":0},{"features":[45,2,72844,11,9,0,3,1,4,0,0,0,46,38],"label":0},{"features":[36,3,306156,15,10,2,11,0,4,1,15024,0,60,38],"label":1},{"features":[32,1,286101,11,9,4,13,4,2,0,0,0,37,38],"label":0},{"features":[35,3,202027,15,10,0,3,1,4,1,0,0,60,38],"label":0},{"features":[24,2,174461,9,13,4,11,1,4,0,0,0,50,38],"label":0},{"features":[39,1,189911,1,7,0,0,4,4,0,0,0,40,38],"label":0},{"features":[57,4,95280,15,10,2,11,0,4,1,99999,0,45,38],"label":1},{"features":[24,1,249101,11,9,0,10,4,2,0,0,0,40,38],"label":0},{"features":[36,2,749636,15,10,0,0,4,4,0,0,0,40,38],"label":0},{"features":[35,2,187119,15,10,0,3,1,4,0,0,0,70,38],"label":0},{"features":[19,2,184207,15,10,4,11,1,4,1,0,0,40,38],"label":0},{"features":[42,2,176286,7,12,2,3,0,4,1,0,0,40,38],"label":1},{"features":[51,4,35295,11,9,4,4,4,4,1,0,0,45,38],"label":0},{"features":[44,2,165599,11,9,2,6,0,4,1,0,0,48,38],"label":0},{"features":[29,2,162312,8,11,4,6,1,3,1,0,0,40,38],"label":0},{"features":[36,5,137421,8,11,2,12,0,1,1,0,0,37,16],"label":0},{"features":[41,5,100800,12,14,0,9,1,4,1,0,0,35,38],"label":0},{"features":[66,2,142723,4,3,3,5,4,4,0,0,0,40,32],"label":0},{"features":[28,2,199903,9,13,4,0,1,4,0,0,0,20,38],"label":0},{"features":[38,2,210438,5,4,0,11,4,4,0,0,0,40,38],"label":0},{"features":[39,2,216149,14,15,0,9,1,4,1,0,0,70,38],"label":1},{"features":[34,2,355571,11,9,0,6,4,2,0,0,0,40,38],"label":0},{"features":[52,4,42984,14,15,2,9,0,4,1,0,0,70,38],"label":1},{"features":[52,2,226084,11,9,6,8,2,4,0,0,0,40,38],"label":0},{"features":[29,4,229842,11,9,4,13,4,2,1,0,0,45,38],"label":0},{"features":[40,4,29036,15,10,4,6,1,4,1,0,0,35,38],"label":0},{"features":[36,2,102864,11,9,4,6,3,4,0,0,0,40,38],"label":0},{"features":[27,4,334132,7,12,4,9,1,4,0,0,0,78,38],"label":0},{"features":[65,2,172906,11,9,6,0,4,4,0,0,0,40,38],"label":0},{"features":[41,2,163287,11,9,2,9,0,4,1,7688,0,43,38],"label":1},{"features":[41,4,83411,11,9,2,3,0,4,1,0,0,40,38],"label":1},{"features":[45,3,160440,11,9,0,3,1,4,1,0,0,42,38],"label":0},{"features":[65,2,143554,15,10,5,0,1,4,0,0,0,38,38],"label":0},{"features":[49,2,242987,9,13,2,9,0,4,1,0,0,40,3],"label":0},{"features":[25,2,166971,11,9,2,11,0,4,1,0,0,52,38],"label":0},{"features":[28,4,204984,9,13,4,12,1,4,1,0,0,45,38],"label":0},{"features":[24,2,267706,15,10,4,2,3,4,0,0,0,45,38],"label":0},{"features":[20,0,191878,15,10,4,0,3,2,0,0,0,20,38],"label":0},{"features":[33,5,175023,11,9,2,10,0,4,1,0,0,37,38],"label":0},{"features":[23,2,179423,9,13,4,0,1,4,0,0,0,5,38],"label":0},{"features":[78,3,188044,9,13,2,3,0,4,1,0,2392,40,38],"label":1},{"features":[30,2,427474,6,5,2,7,0,4,1,0,0,40,25],"label":0},{"features":[55,4,189933,5,4,2,4,0,4,1,0,0,50,38],"label":0},{"features":[20,2,219211,15,10,4,7,3,4,1,0,0,20,38],"label":0},{"features":[30,2,87561,7,12,4,12,1,4,0,0,0,40,38],"label":0},{"features":[38,2,203836,11,9,2,11,0,4,1,3464,0,40,3],"label":0},{"features":[34,2,157289,15,10,2,2,0,4,1,0,0,40,38],"label":0},{"features":[30,2,175856,12,14,2,9,0,4,1,0,0,38,38],"label":0},{"features":[40,2,240124,11,9,2,3,0,4,1,0,0,40,38],"label":1},{"features":[39,2,201410,9,13,2,13,0,4,1,0,1977,45,29],"label":1},{"features":[42,2,190179,9,13,2,9,0,4,1,99999,0,40,38],"label":1},{"features":[47,2,357848,11,9,2,2,0,4,1,0,0,40,38],"label":1},{"features":[33,2,120201,11,9,0,0,3,3,0,0,0,65,38],"label":0},{"features":[29,2,170301,11,9,2,0,5,4,0,2829,0,40,38],"label":0},{"features":[35,2,183898,8,11,2,3,0,4,1,7298,0,50,38],"label":1},{"features":[45,2,123681,11,9,2,11,0,4,1,0,0,40,38],"label":1},{"features":[33,2,169496,9,13,2,3,0,4,1,0,0,50,38],"label":1},{"features":[34,2,152246,11,9,2,13,0,0,1,0,0,52,38],"label":0},{"features":[47,3,101926,9,13,0,3,1,4,1,0,0,70,38],"label":1},{"features":[30,2,142977,15,10,0,2,1,4,1,0,0,65,38],"label":0},{"features":[34,2,260560,11,9,2,6,0,4,1,0,0,40,38],"label":0},{"features":[39,2,315291,11,9,4,0,4,2,0,0,0,40,38],"label":0},{"features":[24,2,306779,8,11,4,3,3,4,1,0,0,35,38],"label":0},{"features":[47,2,339863,11,9,2,11,0,4,1,0,0,45,38],"label":1},{"features":[77,4,71676,15,10,6,0,1,4,0,0,1944,1,38],"label":0},{"features":[53,2,250034,9,13,2,3,0,2,1,0,0,50,38],"label":1},{"features":[33,2,91666,2,8,0,3,1,4,1,0,0,40,38],"label":0},{"features":[36,2,113397,11,9,2,5,0,4,1,0,0,40,38],"label":0},{"features":[51,2,56915,11,9,2,2,0,0,1,0,0,40,38],"label":0},{"features":[17,2,99462,1,7,4,7,3,0,0,0,0,20,38],"label":0},{"features":[44,5,167265,12,14,2,9,0,4,1,0,0,60,38],"label":1},{"features":[43,2,124919,11,9,2,7,0,1,1,0,0,60,23],"label":0},{"features":[35,2,247750,11,9,6,7,4,2,1,0,0,40,38],"label":0},{"features":[46,1,36228,11,9,2,2,0,4,1,0,1902,40,38],"label":0},{"features":[39,0,314822,15,10,2,0,0,2,1,0,0,40,38],"label":0},{"features":[38,2,168407,15,10,0,0,4,4,0,5721,0,44,38],"label":0},{"features":[50,2,105010,9,13,2,4,0,4,1,0,0,45,38],"label":1},{"features":[47,2,72880,12,14,4,9,1,4,0,0,0,40,38],"label":0},{"features":[47,4,318593,11,9,2,3,0,4,1,0,0,25,38],"label":0},{"features":[26,2,201481,9,13,4,3,1,4,0,0,0,40,38],"label":0},{"features":[36,2,139743,15,10,6,9,3,4,0,0,0,40,38],"label":0},{"features":[46,2,216934,9,13,0,0,1,4,1,0,0,40,31],"label":0},{"features":[17,1,191910,1,7,4,11,3,4,1,0,0,20,38],"label":0},{"features":[19,2,229431,15,10,4,9,3,4,1,0,0,11,38],"label":0},{"features":[36,2,43712,0,6,2,2,0,4,1,0,0,40,38],"label":0},{"features":[41,2,320984,14,15,2,9,0,4,1,99999,0,65,38],"label":1},{"features":[51,2,126010,11,9,2,2,0,4,1,0,0,40,38],"label":0},{"features":[41,0,564135,12,14,2,3,0,4,1,0,0,40,38],"label":1},{"features":[37,2,305259,7,12,0,3,1,4,0,0,0,48,38],"label":0},{"features":[41,2,320744,11,9,4,2,1,4,1,3325,0,50,38],"label":0},{"features":[45,2,166929,1,7,2,2,0,4,1,0,0,40,38],"label":0},{"features":[57,3,123053,14,15,2,9,0,1,1,15024,0,50,18],"label":1},{"features":[32,2,154120,11,9,2,13,0,4,1,7298,0,40,38],"label":1},{"features":[48,2,109832,12,14,2,9,0,4,1,0,1902,40,38],"label":1},{"features":[45,3,84324,7,12,2,9,0,4,1,0,0,50,38],"label":1},{"features":[24,2,233280,7,12,4,11,3,4,0,0,0,37,38],"label":0},{"features":[43,1,174491,11,9,0,12,1,2,0,0,0,40,38],"label":0},{"features":[26,2,39014,2,8,2,8,5,3,0,0,0,40,5],"label":0},{"features":[48,2,273828,4,3,4,5,1,4,1,0,0,40,25],"label":0},{"features":[53,2,53197,12,14,2,9,0,4,1,3103,0,40,38],"label":1},{"features":[34,2,286020,11,9,2,6,0,4,1,0,0,45,38],"label":0},{"features":[48,2,235646,15,10,2,11,0,4,1,3103,0,40,38],"label":1},{"features":[61,2,160942,12,14,2,11,0,4,1,3103,0,50,38],"label":0},{"features":[42,4,177937,9,13,3,3,1,4,1,0,0,45,30],"label":0},{"features":[37,2,98941,12,14,4,3,1,4,1,0,0,40,38],"label":1},{"features":[32,2,169589,8,11,2,5,0,4,1,0,0,40,38],"label":1},{"features":[35,2,219902,11,9,5,13,4,2,0,0,0,48,38],"label":0},{"features":[38,2,107125,15,10,4,11,1,4,1,0,0,60,38],"label":0},{"features":[59,2,453067,15,10,2,9,0,4,1,0,0,36,38],"label":1},{"features":[43,2,222971,4,3,4,6,4,4,0,0,0,40,25],"label":0},{"features":[34,2,294064,12,14,2,3,0,4,1,0,0,50,9],"label":0},{"features":[21,2,56582,1,7,4,7,3,4,1,0,0,50,38],"label":0},{"features":[61,2,166124,11,9,2,2,0,4,1,0,0,40,38],"label":1},{"features":[32,2,107218,9,13,4,0,1,1,1,0,0,40,38],"label":0},{"features":[72,2,56559,11,9,2,11,0,4,1,0,0,12,38],"label":0},{"features":[45,2,198759,10,16,2,3,0,4,1,0,0,60,38],"label":0},{"features":[38,2,119741,12,14,2,2,0,2,1,0,0,40,38],"label":1},{"features":[26,2,117217,9,13,0,7,1,4,0,0,0,45,38],"label":0},{"features":[48,2,115585,9,13,2,11,0,4,1,0,0,40,38],"label":0},{"features":[22,5,311512,15,10,2,7,0,2,1,0,0,15,38],"label":0},{"features":[34,2,164190,15,10,2,9,0,4,1,0,1902,38,38],"label":1},{"features":[37,2,387430,15,10,2,0,0,4,1,0,0,37,38],"label":0},{"features":[62,2,214288,11,9,2,6,0,4,1,0,0,40,38],"label":0},{"features":[28,2,190911,11,9,2,2,0,4,1,0,0,40,38],"label":0},{"features":[35,2,267798,11,9,0,2,4,4,1,0,0,40,38],"label":0},{"features":[28,2,204516,0,6,4,13,1,4,1,0,0,45,38],"label":0},{"features":[19,2,125591,1,7,4,7,1,4,0,0,0,40,38],"label":0},{"features":[31,2,113364,7,12,2,6,0,4,1,0,0,55,38],"label":0},{"features":[64,2,133166,11,9,2,3,0,4,1,0,0,5,38],"label":0},{"features":[21,2,178255,15,10,4,0,1,4,0,0,0,30,3],"label":0},{"features":[21,2,116788,11,9,4,2,3,4,1,0,0,40,38],"label":0},{"features":[20,2,141481,1,7,2,11,2,4,0,0,0,50,38],"label":0},{"features":[33,2,138142,15,10,5,7,4,2,0,0,0,25,38],"label":0},{"features":[25,2,254613,11,9,4,2,3,4,1,0,0,40,4],"label":0},{"features":[54,4,200960,9,13,2,11,0,4,1,0,0,50,38],"label":1},{"features":[24,2,200593,11,9,2,5,0,4,1,0,0,50,38],"label":0},{"features":[62,2,200332,11,9,2,6,0,4,1,0,0,40,38],"label":0},{"features":[20,4,197207,11,9,0,11,1,4,0,0,0,30,38],"label":0},{"features":[53,2,133436,5,4,0,6,1,4,0,0,0,40,38],"label":0},{"features":[17,4,228786,0,6,4,7,3,4,0,0,0,24,38],"label":0},{"features":[27,2,404421,15,10,4,5,1,2,1,0,0,40,38],"label":0},{"features":[55,2,61708,11,9,2,0,0,4,1,6418,0,50,38],"label":1},{"features":[21,2,147655,11,9,4,0,3,4,0,0,0,40,38],"label":0},{"features":[35,1,103966,12,14,0,0,4,4,0,0,0,41,38],"label":0}]}

The test dataset only has features.

[6]:
!head -n 5 $test_dataset_path
{"instances":[{"features":[28,2,133937,9,13,2,0,0,4,1,15024,0,55,37]},{"features":[43,2,72338,12,14,2,12,0,1,1,0,0,40,37]},{"features":[34,2,162604,11,9,4,2,2,2,1,0,0,40,37]},{"features":[20,2,258509,11,9,4,6,3,2,1,0,0,40,37]},{"features":[27,2,446947,9,13,4,0,4,2,0,0,0,55,37]},{"features":[20,2,95552,11,9,4,11,3,4,1,0,0,40,37]},{"features":[46,2,145636,11,9,2,3,0,4,1,3103,0,50,37]},{"features":[18,2,150675,0,6,4,11,3,4,1,0,0,40,37]},{"features":[22,2,197050,11,9,4,7,3,4,0,0,0,20,37]},{"features":[20,2,246635,15,10,4,11,3,4,0,2597,0,20,37]},{"features":[65,0,200764,11,9,6,0,1,4,0,0,0,40,37]},{"features":[38,2,175665,15,10,2,9,5,4,0,0,0,40,37]},{"features":[34,3,337995,9,13,0,3,4,2,1,15020,0,50,37]},{"features":[42,2,86912,9,13,0,7,1,4,1,0,0,40,37]},{"features":[40,2,100451,15,10,4,2,1,4,1,0,0,40,37]},{"features":[45,2,192360,12,14,2,3,0,4,1,0,1902,50,37]},{"features":[55,2,150507,15,10,2,0,0,4,1,0,0,40,37]},{"features":[36,2,48976,9,13,2,11,5,4,0,0,0,40,37]},{"features":[34,2,111567,15,10,4,3,1,4,1,0,0,40,37]},{"features":[26,2,167350,15,10,2,6,0,4,1,3137,0,50,37]},{"features":[29,2,485944,9,13,4,11,3,2,1,0,0,40,37]},{"features":[44,1,112763,12,14,0,9,4,4,0,0,0,38,37]},{"features":[37,5,195843,11,9,2,2,0,4,1,5013,0,40,37]},{"features":[22,5,181096,9,13,4,9,3,2,1,0,0,20,37]},{"features":[53,2,119170,11,9,2,13,0,2,1,0,1740,40,37]},{"features":[61,1,205711,11,9,2,9,0,4,1,0,0,30,37]},{"features":[46,0,260549,15,10,2,0,0,4,1,0,0,80,37]},{"features":[18,2,129053,1,7,4,7,3,4,1,0,0,28,37]},{"features":[22,2,209034,15,10,4,7,1,4,0,0,0,35,37]},{"features":[29,2,266583,11,9,2,11,0,2,1,2829,0,38,37]},{"features":[30,2,96480,8,11,4,0,3,4,0,0,0,32,37]},{"features":[66,4,331960,11,9,2,2,0,4,1,0,0,20,37]},{"features":[44,2,83891,9,13,0,0,3,1,1,5455,0,40,37]},{"features":[61,5,103575,15,10,0,2,1,4,1,0,0,40,10]},{"features":[38,2,589809,9,13,2,0,0,4,1,0,0,45,37]},{"features":[33,2,214288,11,9,2,6,0,4,1,0,1848,48,37]},{"features":[31,2,280927,9,13,4,3,1,4,0,0,0,40,37]},{"features":[49,2,380922,12,14,2,3,0,4,1,15024,0,80,37]},{"features":[34,2,361497,1,7,2,13,0,4,1,0,0,40,37]},{"features":[37,2,306868,11,9,0,2,4,4,1,0,0,38,37]},{"features":[17,2,364952,0,6,3,7,2,4,1,0,0,40,37]},{"features":[60,2,338833,11,9,4,0,1,2,0,0,0,38,37]},{"features":[30,4,70985,11,9,2,4,0,4,1,0,0,75,37]},{"features":[22,2,240229,11,9,4,0,3,4,0,0,0,40,37]},{"features":[51,2,173987,11,9,2,2,0,4,1,0,0,40,37]},{"features":[29,2,157103,8,11,4,12,3,2,1,0,1974,40,37]},{"features":[42,2,205195,11,9,2,2,0,4,1,0,0,40,37]},{"features":[25,5,120268,15,10,2,2,3,4,1,0,0,50,37]},{"features":[64,2,104973,11,9,2,0,0,4,1,0,0,45,37]},{"features":[38,4,248694,15,10,2,2,0,4,1,0,0,36,37]},{"features":[54,1,108739,1,7,6,10,4,2,0,0,0,40,37]},{"features":[57,2,151874,11,9,2,7,5,2,0,0,0,50,37]},{"features":[27,2,150767,15,10,4,6,3,4,1,0,0,48,37]},{"features":[53,2,239155,15,10,2,3,0,4,1,0,0,50,37]},{"features":[35,2,166497,14,15,2,9,0,4,1,0,1902,60,37]},{"features":[22,2,50610,15,10,4,7,1,4,0,0,0,40,37]},{"features":[52,2,335997,9,13,2,12,0,4,1,7688,0,38,37]},{"features":[27,4,209301,11,9,2,2,0,4,1,0,0,60,37]},{"features":[26,2,247196,15,10,4,5,3,4,1,0,0,35,37]},{"features":[23,2,213902,15,10,4,7,4,4,0,0,0,20,37]},{"features":[25,1,281412,11,9,4,7,3,4,0,0,0,35,37]},{"features":[17,2,154337,1,7,4,7,3,4,0,0,0,13,37]},{"features":[22,2,95647,1,7,4,13,3,1,1,0,0,40,28]},{"features":[32,2,177695,9,13,2,2,0,1,1,0,0,45,17]},{"features":[54,2,64421,15,10,6,12,4,4,0,0,0,40,37]},{"features":[45,2,176341,11,9,0,7,4,4,0,0,0,32,37]},{"features":[20,2,203914,2,8,4,7,3,4,0,0,0,25,37]},{"features":[22,2,23940,11,9,4,3,1,1,1,0,0,40,37]},{"features":[32,2,169768,9,13,5,12,1,2,1,0,0,40,37]},{"features":[36,2,109133,9,13,2,11,0,4,1,0,0,50,37]},{"features":[33,2,41610,11,9,5,2,1,4,1,0,0,40,37]},{"features":[37,2,33440,11,9,5,7,4,4,0,0,0,40,37]},{"features":[46,2,151325,0,6,2,2,0,4,1,0,0,40,37]},{"features":[54,1,182429,11,9,6,13,4,4,0,0,0,38,37]},{"features":[34,2,195748,7,12,4,0,3,2,0,0,0,38,37]},{"features":[22,2,248446,4,3,4,8,1,4,1,0,0,50,12]},{"features":[42,2,188789,5,4,6,5,1,4,0,0,0,35,37]},{"features":[34,2,185480,7,12,4,0,3,4,0,0,0,40,37]},{"features":[39,2,30875,9,13,0,11,4,4,0,0,0,40,37]},{"features":[21,2,116489,15,10,4,9,3,4,0,0,0,40,37]},{"features":[18,2,99591,1,7,4,7,3,4,0,0,0,16,37]},{"features":[43,2,282678,11,9,0,3,1,4,0,0,0,60,37]},{"features":[56,1,238405,11,9,6,0,1,4,0,0,0,40,37]},{"features":[32,1,247156,11,9,2,7,0,2,1,3103,0,38,37]},{"features":[19,2,73461,11,9,4,12,1,2,1,0,0,40,37]},{"features":[35,2,98776,11,9,4,3,1,4,1,0,0,60,37]},{"features":[30,2,232766,11,9,0,7,4,4,0,0,0,40,37]},{"features":[32,2,220333,11,9,2,2,0,4,1,7298,0,46,37]},{"features":[27,2,321456,15,10,2,10,0,4,1,0,0,40,37]},{"features":[41,2,173307,11,9,2,13,0,4,1,0,0,43,37]},{"features":[22,2,351952,15,10,4,0,3,4,0,0,0,38,37]},{"features":[33,2,108438,15,10,2,3,0,4,1,0,0,60,37]},{"features":[30,2,171483,11,9,4,2,3,4,1,0,0,38,37]},{"features":[32,2,453983,11,9,2,5,0,4,1,0,0,44,37]},{"features":[37,2,48779,11,9,4,3,1,4,1,0,0,50,37]},{"features":[42,2,222756,9,13,0,9,4,4,1,7430,0,40,37]},{"features":[49,2,118520,11,9,0,0,1,4,0,0,0,45,37]},{"features":[34,2,199539,8,11,2,2,0,4,1,0,0,48,37]},{"features":[42,2,201343,11,9,2,2,0,4,1,2885,0,40,37]},{"features":[49,2,99340,4,3,5,6,4,4,0,0,0,40,5]},{"features":[48,2,163706,9,13,2,3,0,4,1,15024,0,70,37]},{"features":[59,2,176118,12,14,2,9,0,4,1,0,0,7,37]},{"features":[67,3,147377,11,9,2,3,0,4,1,0,0,45,37]},{"features":[36,2,225330,11,9,0,7,4,4,0,0,0,40,37]},{"features":[32,2,147921,14,15,4,7,1,4,0,0,0,35,37]},{"features":[36,2,110013,12,14,4,11,1,4,0,0,0,40,37]},{"features":[76,4,130585,15,10,2,7,5,4,0,0,0,12,37]},{"features":[41,4,134724,8,11,2,7,5,4,0,3103,0,40,37]},{"features":[44,2,160369,15,10,2,8,0,4,1,0,0,2,37]},{"features":[24,2,172169,15,10,4,5,4,4,1,0,0,30,37]},{"features":[35,2,106471,9,13,4,2,1,4,1,0,0,35,37]},{"features":[25,1,336320,9,13,0,10,1,4,0,0,0,40,37]},{"features":[62,2,186446,15,10,0,12,4,4,0,0,0,43,37]},{"features":[39,2,183279,9,13,2,11,0,4,1,7298,0,40,37]},{"features":[65,4,135517,5,4,2,2,0,4,1,0,0,40,37]},{"features":[48,0,72808,1,7,0,0,1,4,0,0,0,42,37]},{"features":[56,2,197577,11,9,0,7,1,4,0,0,0,40,37]},{"features":[51,3,110327,1,7,2,2,0,4,1,0,0,60,37]},{"features":[23,2,237811,15,10,4,0,4,2,0,0,0,40,36]},{"features":[18,2,632271,15,10,3,0,2,4,0,0,0,40,27]},{"features":[18,2,220754,1,7,4,5,3,4,1,0,0,24,37]},{"features":[61,2,29797,11,9,0,11,2,4,0,0,0,40,37]},{"features":[32,2,183470,8,11,2,2,0,0,1,0,0,42,37]},{"features":[36,2,127388,7,12,2,11,5,4,0,0,0,40,37]},{"features":[19,2,78401,11,9,4,7,3,4,1,0,0,40,37]},{"features":[37,2,385330,5,4,5,7,4,2,1,0,0,40,37]},{"features":[53,2,161691,12,14,0,3,1,4,0,4865,0,40,37]},{"features":[31,2,301251,9,13,2,2,0,4,1,0,0,50,37]},{"features":[30,2,198660,11,9,2,5,0,4,1,0,0,40,37]},{"features":[44,2,105896,9,13,0,9,1,4,0,0,0,36,37]},{"features":[23,2,132220,11,9,2,5,0,4,1,0,0,40,37]},{"features":[45,1,317846,7,12,0,3,4,4,1,0,0,47,37]},{"features":[32,2,33117,8,11,2,7,0,4,1,0,0,40,37]},{"features":[41,2,192602,15,10,2,2,0,4,1,0,0,40,37]},{"features":[30,2,408328,13,1,3,5,4,4,1,0,0,40,24]},{"features":[34,2,233729,7,12,2,9,0,2,1,0,0,50,37]},{"features":[21,2,174063,8,11,4,7,3,4,0,0,0,20,37]},{"features":[30,2,175323,8,11,2,3,5,4,0,0,0,52,37]},{"features":[20,2,460356,2,8,4,7,1,4,1,0,0,30,24]},{"features":[33,2,119422,11,9,2,3,0,4,1,0,0,40,37]},{"features":[26,2,269168,15,10,2,3,0,1,1,0,0,40,37]},{"features":[21,5,173534,15,10,4,9,3,4,0,0,0,40,6]},{"features":[48,2,235891,11,9,4,7,1,4,1,0,0,40,31]},{"features":[70,3,217801,9,13,2,11,0,4,1,0,0,15,37]},{"features":[52,1,251841,12,14,4,9,1,4,0,0,0,50,37]},{"features":[24,2,196943,8,11,2,9,0,4,1,0,0,40,37]},{"features":[41,2,204415,1,7,0,5,1,4,1,0,0,48,37]},{"features":[23,2,130959,9,13,2,9,0,4,1,2407,0,6,1]},{"features":[46,2,316271,4,3,2,2,0,4,1,0,0,55,37]},{"features":[59,2,124137,11,9,0,11,1,4,1,2202,0,40,37]},{"features":[36,4,140676,9,13,4,11,1,4,1,0,0,50,37]},{"features":[52,2,91506,11,9,2,5,0,4,1,0,0,45,37]},{"features":[40,2,300195,15,10,0,12,4,2,0,0,0,40,37]},{"features":[51,3,119570,9,13,2,2,0,4,1,0,0,50,37]},{"features":[43,2,303155,9,13,2,3,0,4,1,0,0,50,37]},{"features":[30,2,210541,11,9,0,2,1,4,0,0,0,40,37]},{"features":[48,2,153312,15,10,2,11,0,2,1,0,0,60,37]},{"features":[50,5,137815,9,13,2,2,0,4,1,0,0,40,37]},{"features":[38,4,179824,11,9,4,4,1,4,1,0,0,50,37]},{"features":[41,2,106159,11,9,4,6,3,4,1,14344,0,48,37]},{"features":[69,2,104827,11,9,6,12,4,4,0,0,0,8,37]},{"features":[21,2,278254,15,10,4,5,3,2,1,0,0,40,37]},{"features":[33,3,287372,15,10,2,3,0,4,1,0,0,50,37]},{"features":[51,5,152810,8,11,2,12,0,4,1,0,0,40,37]},{"features":[46,2,106662,9,13,5,11,1,4,1,99999,0,55,37]},{"features":[35,2,108140,11,9,0,2,1,4,1,0,0,40,37]},{"features":[29,2,231507,11,9,4,2,1,4,1,0,0,35,37]},{"features":[34,4,114074,8,11,6,3,4,4,0,0,0,40,37]},{"features":[52,2,163776,11,9,2,11,0,4,1,0,1902,60,37]},{"features":[45,2,123219,4,3,4,6,1,4,1,0,0,40,37]},{"features":[25,2,391591,11,9,4,2,1,4,1,0,0,50,37]},{"features":[61,1,202384,9,13,2,9,5,4,0,0,0,30,37]},{"features":[58,2,282023,9,13,2,3,0,4,1,0,0,50,37]},{"features":[51,5,22211,11,9,0,3,1,4,1,0,0,37,37]},{"features":[27,2,192936,9,13,4,9,1,4,0,0,0,45,37]},{"features":[51,1,106365,7,12,0,0,4,4,0,0,0,40,37]},{"features":[51,2,166461,1,7,0,6,4,2,0,5455,0,40,37]},{"features":[52,2,251585,0,6,2,13,0,4,1,0,0,55,37]},{"features":[61,1,149981,11,9,6,0,1,4,0,0,0,40,37]},{"features":[23,2,161092,9,13,4,0,3,4,1,0,0,40,37]},{"features":[40,2,21755,15,10,4,2,2,0,1,0,0,30,37]},{"features":[20,2,174436,11,9,4,2,3,4,1,0,0,60,37]},{"features":[26,4,33016,8,11,0,7,4,4,0,0,0,55,37]},{"features":[55,1,134042,12,14,2,3,5,4,0,0,0,40,37]},{"features":[32,2,259425,15,10,0,2,1,4,1,0,0,40,37]},{"features":[26,2,359854,9,13,4,8,2,4,0,0,0,35,24]},{"features":[44,2,217039,14,15,2,9,0,4,1,99999,0,60,37]},{"features":[61,2,194804,13,1,5,13,1,2,1,14344,0,40,37]},{"features":[34,4,198068,11,9,2,2,0,4,1,0,0,40,37]},{"features":[42,4,52131,15,10,4,3,1,4,1,0,0,40,37]},{"features":[23,2,239539,11,9,4,6,3,1,1,0,0,40,28]},{"features":[25,2,54298,11,9,2,11,0,4,1,0,0,30,37]},{"features":[17,2,35603,2,8,4,11,3,4,0,0,0,20,37]},{"features":[31,2,241880,8,11,4,0,1,2,1,0,0,45,37]},{"features":[35,2,46947,15,10,0,0,1,4,0,0,0,45,37]},{"features":[28,2,203171,15,10,0,2,1,4,1,0,0,40,37]},{"features":[37,2,199739,15,10,0,2,3,4,1,0,0,40,37]},{"features":[23,2,215395,15,10,4,2,1,4,1,0,0,40,37]},{"features":[53,2,117932,11,9,0,6,1,4,0,0,0,40,37]},{"features":[30,5,107142,9,13,2,9,0,4,1,0,0,37,37]},{"features":[33,2,173730,8,11,2,6,0,4,1,0,0,40,37]},{"features":[53,3,200400,10,16,0,3,1,4,1,0,0,60,37]},{"features":[50,2,158948,11,9,2,9,0,4,1,0,0,84,37]},{"features":[39,2,206888,15,10,0,0,1,4,0,0,0,40,37]},{"features":[26,2,124483,9,13,4,9,1,1,1,0,0,25,17]},{"features":[34,5,62327,9,13,2,9,0,4,1,0,0,40,37]},{"features":[26,2,366889,11,9,4,13,1,4,1,0,0,40,37]},{"features":[21,2,30796,15,10,4,7,3,4,0,0,0,25,37]},{"features":[46,2,130667,11,9,2,13,0,2,1,0,0,40,37]},{"features":[67,0,231604,11,9,4,0,1,4,1,0,0,40,37]},{"features":[25,2,332409,8,11,2,2,0,4,1,0,0,40,37]},{"features":[34,2,51854,11,9,4,6,1,4,1,0,0,40,37]},{"features":[50,2,62593,8,11,2,4,0,1,1,0,0,40,37]},{"features":[47,2,78954,1,7,0,11,4,4,0,0,0,28,37]},{"features":[39,2,205997,15,10,2,11,5,4,0,0,0,21,37]},{"features":[51,2,231230,11,9,2,6,0,4,1,0,0,45,37]},{"features":[62,2,291904,11,9,0,8,1,2,0,0,0,20,37]},{"features":[58,2,49893,12,14,2,3,0,4,1,0,0,50,37]},{"features":[36,2,141584,15,10,2,9,0,4,1,0,0,50,37]},{"features":[28,2,259609,11,9,4,2,3,4,1,0,0,50,37]},{"features":[22,2,125010,9,13,4,0,1,4,0,0,0,20,37]},{"features":[59,5,136819,12,14,2,9,0,4,1,0,0,8,37]},{"features":[69,4,199829,9,13,2,3,0,4,1,0,1258,40,37]},{"features":[33,4,100580,15,10,2,7,5,4,0,0,0,10,37]},{"features":[56,2,257555,12,14,2,9,0,4,1,0,0,40,37]},{"features":[47,2,100113,5,4,2,13,0,4,1,0,2051,40,37]},{"features":[38,0,236648,11,9,2,2,0,4,1,0,0,40,37]},{"features":[41,2,99679,0,6,2,2,0,4,1,0,0,40,37]},{"features":[32,2,339482,12,14,4,3,1,4,1,0,0,48,37]},{"features":[28,2,120475,11,9,4,2,1,4,1,0,0,35,37]},{"features":[22,2,137876,15,10,4,10,1,4,1,0,0,20,37]},{"features":[36,4,110861,11,9,0,2,3,4,1,0,0,20,37]},{"features":[55,4,225623,15,10,2,4,0,4,1,0,0,40,37]},{"features":[47,2,323212,11,9,6,7,1,4,0,0,0,40,37]},{"features":[59,2,157831,11,9,0,0,1,4,0,0,0,16,37]},{"features":[25,2,25497,15,10,4,13,1,4,1,4101,0,40,37]},{"features":[42,4,114580,12,14,0,3,4,4,0,0,0,70,37]},{"features":[22,2,273675,11,9,3,7,2,2,0,0,0,35,31]},{"features":[31,0,40909,15,10,2,12,0,2,1,0,0,40,37]},{"features":[42,3,557349,9,13,2,3,0,4,1,0,0,70,37]},{"features":[18,2,219256,15,10,4,11,3,4,0,0,0,25,37]},{"features":[39,2,126569,11,9,4,2,1,4,1,0,0,40,29]},{"features":[37,2,108282,9,13,2,3,0,4,1,0,0,45,37]},{"features":[31,2,147270,15,10,4,0,3,4,0,0,0,35,37]},{"features":[44,2,90582,9,13,2,2,0,4,1,0,0,50,37]},{"features":[51,2,379797,0,6,2,6,0,2,1,0,0,40,37]},{"features":[37,1,136749,11,9,4,0,3,4,0,0,0,35,37]},{"features":[25,0,198813,9,13,4,0,4,2,0,0,1590,40,37]},{"features":[30,2,159123,11,9,2,2,0,4,1,0,0,45,37]},{"features":[36,3,196554,11,9,2,2,0,4,1,0,0,46,37]},{"features":[31,2,238002,9,13,2,13,0,4,1,0,0,55,24]},{"features":[43,2,125577,11,9,5,0,4,2,0,0,0,40,37]},{"features":[22,2,97212,11,9,4,7,1,4,0,0,0,15,37]},{"features":[19,2,222866,0,6,4,4,2,4,1,0,0,40,37]},{"features":[18,2,175752,11,9,4,5,3,4,1,0,0,30,37]},{"features":[28,2,77009,15,10,4,11,2,4,0,0,0,40,37]},{"features":[54,2,162745,11,9,2,2,0,4,1,0,0,55,37]},{"features":[30,2,94235,9,13,2,9,0,4,1,0,1977,50,37]},{"features":[19,2,158343,15,10,4,7,3,4,0,0,0,12,37]},{"features":[49,2,201127,1,7,2,13,0,4,1,0,1902,70,37]},{"features":[39,2,118429,15,10,0,11,1,4,1,0,0,40,37]},{"features":[36,2,334365,1,7,2,13,0,4,1,0,0,60,37]},{"features":[42,2,89226,8,11,2,13,0,4,1,0,0,45,37]},{"features":[33,2,56121,11,9,4,13,1,4,1,0,0,60,37]},{"features":[61,5,140851,9,13,2,9,0,4,1,0,0,40,37]},{"features":[36,2,86643,2,8,2,6,0,4,1,0,0,48,37]},{"features":[20,2,175808,11,9,4,2,3,4,1,0,0,40,37]},{"features":[19,2,58471,11,9,4,2,3,4,0,0,0,40,37]},{"features":[55,2,118057,11,9,6,2,4,4,1,0,0,51,37]},{"features":[30,2,192002,15,10,2,2,0,4,1,0,0,40,37]},{"features":[61,2,43904,11,9,0,7,1,2,1,0,0,40,37]},{"features":[39,3,31709,15,10,2,0,5,4,0,0,0,20,37]},{"features":[39,2,286026,9,13,2,2,0,4,1,0,0,52,37]},{"features":[55,4,110844,11,9,2,3,5,4,0,0,0,40,37]},{"features":[32,2,200401,11,9,4,3,1,4,1,0,0,40,3]},{"features":[44,5,101603,9,13,2,3,0,4,1,0,0,40,37]},{"features":[58,2,49159,11,9,2,0,5,4,0,0,0,40,37]},{"features":[52,5,168035,15,10,2,12,0,4,1,0,0,45,37]},{"features":[18,2,260977,2,8,4,11,3,4,0,0,0,20,37]},{"features":[47,2,33794,11,9,2,2,0,4,1,0,0,56,37]},{"features":[26,2,242464,8,11,4,3,1,4,1,0,0,50,37]},{"features":[35,2,97554,7,12,2,3,0,4,1,0,0,50,37]},{"features":[39,4,245361,15,10,4,9,3,4,0,0,0,10,37]},{"features":[26,2,178478,15,10,4,11,3,4,0,0,0,40,37]},{"features":[31,2,104509,15,10,5,7,4,4,0,0,0,35,37]},{"features":[31,2,159187,15,10,2,2,0,4,1,0,0,25,37]},{"features":[67,4,167015,9,13,6,11,1,4,1,0,0,30,37]},{"features":[40,2,199668,11,9,0,11,3,4,0,0,0,25,37]},{"features":[35,2,37778,11,9,2,2,0,4,1,0,0,50,37]},{"features":[54,4,139023,15,10,2,11,0,4,1,0,0,40,37]},{"features":[45,3,188694,14,15,2,9,0,4,1,0,0,50,37]},{"features":[50,2,178251,12,14,2,0,5,4,0,0,0,40,37]},{"features":[51,2,81534,1,7,4,7,2,1,1,0,0,35,37]},{"features":[37,2,353550,12,14,2,3,0,4,1,15024,0,60,37]},{"features":[54,1,231482,11,9,2,2,0,4,1,0,0,40,30]},{"features":[22,2,228394,11,9,4,7,1,4,0,0,0,50,37]},{"features":[38,1,94529,11,9,2,5,5,4,0,3103,0,50,37]},{"features":[35,2,135289,8,11,0,2,1,4,1,0,0,50,37]},{"features":[37,0,32950,7,12,0,3,4,2,0,0,0,40,37]},{"features":[45,2,165346,15,10,0,3,4,4,0,0,0,64,37]},{"features":[57,1,62701,15,10,6,3,1,4,1,6849,0,40,37]},{"features":[30,2,49358,2,8,4,11,3,2,0,0,0,40,37]},{"features":[52,2,227832,9,13,2,9,0,4,1,0,0,50,37]},{"features":[67,2,188903,9,13,2,9,0,4,1,0,0,40,37]},{"features":[28,4,183151,11,9,2,2,0,4,1,0,0,40,37]},{"features":[42,5,116493,9,13,2,10,0,4,1,0,0,52,37]},{"features":[48,1,93449,14,15,2,9,0,1,1,99999,0,40,28]},{"features":[18,2,211683,2,8,4,5,3,4,1,0,0,20,37]},{"features":[47,2,155107,11,9,2,12,0,4,1,0,0,40,37]},{"features":[55,3,150917,15,10,2,3,0,4,1,0,1977,45,37]},{"features":[51,2,135388,2,8,6,6,1,4,1,0,1564,40,37]},{"features":[38,2,183683,0,6,3,7,1,4,1,0,0,45,37]},{"features":[47,4,185859,11,9,2,4,0,4,1,3103,0,60,37]},{"features":[44,4,22933,11,9,2,3,0,4,1,0,0,40,37]},{"features":[40,2,356934,14,15,2,3,0,4,1,0,0,50,37]},{"features":[52,2,94448,8,11,2,9,0,4,1,0,0,40,37]},{"features":[59,2,107318,5,4,2,2,0,4,1,5178,0,50,37]},{"features":[31,2,83413,11,9,4,11,3,4,1,0,0,40,37]},{"features":[34,2,162312,9,13,2,0,0,1,1,0,0,40,28]},{"features":[44,2,118212,0,6,2,6,0,4,1,0,0,40,37]},{"features":[35,1,132879,11,9,2,13,0,4,1,0,0,40,37]},{"features":[25,4,121285,9,13,4,11,1,4,0,0,0,40,37]},{"features":[22,2,341760,9,13,4,3,3,4,0,0,0,40,37]},{"features":[35,2,216473,11,9,0,2,4,4,1,0,0,40,37]},{"features":[25,2,179255,15,10,4,0,3,4,0,0,0,25,37]},{"features":[36,2,298635,9,13,2,7,0,3,1,0,0,40,18]},{"features":[20,2,204596,15,10,4,11,3,4,0,0,0,32,37]},{"features":[27,2,285897,11,9,2,13,0,4,1,0,1887,40,37]},{"features":[19,2,386492,15,10,4,5,3,4,1,0,0,16,37]},{"features":[29,2,178610,15,10,0,7,4,4,0,0,0,21,37]},{"features":[49,2,96854,11,9,0,7,4,4,1,0,0,40,37]},{"features":[45,2,293628,15,10,2,9,0,4,1,0,0,50,28]},{"features":[67,2,192995,11,9,6,0,4,4,0,6723,0,40,37]},{"features":[30,2,235847,9,13,4,7,3,4,0,0,0,24,37]}]}

Here are the headers of the train dataset. “Target” is the header of the ground truth label, and the others are the feature headers. They will be used to beautify the analysis report.

[7]:
all_headers = [
    "Age",
    "Workclass",
    "fnlwgt",
    "Education",
    "Education-Num",
    "Marital Status",
    "Occupation",
    "Relationship",
    "Ethnic group",
    "Sex",
    "Capital Gain",
    "Capital Loss",
    "Hours per week",
    "Country",
    "Target",
]
label_header = all_headers[-1]

To verify that the execution role for this notebook has the necessary permissions to proceed, put a simple test object into the S3 bucket specified above. If this command fails, update the role to have s3:PutObject permission on the bucket and try again.

[8]:
sagemaker.s3.S3Uploader.upload_string_as_file_body(
    body="hello",
    desired_s3_uri=f"{s3_key}/upload-test-file.txt",
    sagemaker_session=sagemaker_session,
)
print("Success! We are all set to proceed with uploading to S3.")
Success! We are all set to proceed with uploading to S3.

Then upload the data files to S3 so that they can be used by SageMaker jobs.

[9]:
train_data_s3_uri = sagemaker.s3.S3Uploader.upload(
    local_path=train_dataset_path,
    desired_s3_uri=s3_key,
    sagemaker_session=sagemaker_session,
)
print(f"Train data is uploaded to: {train_data_s3_uri}")
test_data_s3_uri = sagemaker.s3.S3Uploader.upload(
    local_path=test_dataset_path,
    desired_s3_uri=s3_key,
    sagemaker_session=sagemaker_session,
)
print(f"Test data is uploaded to: {test_data_s3_uri}")
Train data is uploaded to: s3://sagemaker-us-west-2-678264136642/sagemaker/DEMO-ClarifyModelMonitor-1705692266-3764/validation-dataset.json
Test data is uploaded to: s3://sagemaker-us-west-2-678264136642/sagemaker/DEMO-ClarifyModelMonitor-1705692266-3764/test-dataset.json

SageMaker model

This example includes a prebuilt SageMaker Linear Learner model trained by a SageMaker Clarify offline processing example notebook. The model supports SageMaker JSON Lines dense format (MIME type "application/jsonlines").

  • The model input can one or more lines, each line is a JSON object that has a “features” key pointing to a list of feature values concerning demographic characteristics of individuals. For example,

{"features":[28,2,133937,9,13,2,0,0,4,1,15024,0,55,37]}
{"features":[43,2,72338,12,14,2,12,0,1,1,0,0,40,37]}
  • The model output has the predictions of whether a person has a yearly income that is more than $50,000. Each prediction is a JSON object that has a “predicted_label” key pointing to the predicted label, and the “score” key pointing to the confidence score. For example,

{"predicted_label":1,"score":0.989977359771728}
{"predicted_label":1,"score":0.504138827323913}
[10]:
model_file = "model/ll-adult-prediction-model.tar.gz"
model_url = sagemaker.s3.S3Uploader.upload(
    local_path=model_file,
    desired_s3_uri=s3_key,
    sagemaker_session=sagemaker_session,
)
print(f"Model file has been uploaded to {model_url}")

model_name = sagemaker.utils.unique_name_from_base("DEMO-xgb-churn-pred-model-monitor")
print(f"SageMaker model name: {model_name}")

image_uri = sagemaker.image_uris.retrieve("linear-learner", region, "1")
print(f"SageMaker Linear Learner image: {image_uri}")

model = sagemaker.model.Model(image_uri=image_uri, model_data=model_url, role=role)
container_def = model.prepare_container_def()
sagemaker_session.create_model(model_name, role, container_def)
print("SageMaker model created")
Model file has been uploaded to s3://sagemaker-us-west-2-678264136642/sagemaker/DEMO-ClarifyModelMonitor-1705692266-3764/ll-adult-prediction-model.tar.gz
SageMaker model name: DEMO-xgb-churn-pred-model-monitor-1705692267-227f
SageMaker Linear Learner image: 174872318107.dkr.ecr.us-west-2.amazonaws.com/linear-learner:1
SageMaker model created

Batch Transform Job

For continuous monitoring, batch transform jobs should be executed regularly with the latest data. But for demonstration purpose, the following cell only executes the job once before the monitor is scheduled, so that the first monitoring execution has captured data to process.

See Transformer for the API reference. The destination_s3_uri is used to specify the data capture S3 URI which is a key connection between the job and the monitor.

NOTE: The following cell takes about 5 minutes to run.

[11]:
transfomer = model.transformer(
    instance_count=1,
    instance_type="ml.m5.xlarge",
    accept=dataset_type,  # The transform output data format
    assemble_with=None,  # JSON records are under a single JSON structure
    output_path=transform_output_s3_uri,
)

transfomer.transform(
    data=test_data_s3_uri,
    content_type=dataset_type,  # The transform input format
    split_type=None,  # JSON records are under a single JSON structure
    batch_data_capture_config=sagemaker.inputs.BatchDataCaptureConfig(
        destination_s3_uri=data_capture_s3_uri,
    ),
    wait=True,  # In real world you don't have to wait, but for demo purpose we wait for the output
    logs=False,  # You can change it to True to view job logs inline
)
INFO:sagemaker:Creating transform job with name: linear-learner-2024-01-19-19-24-28-808
....................................................................!

Captured data

Once the transform job completed, an “input” folders is created under data_capture_s3_uri, to includes the captured data files of transform input. Note that, batch transform data capture is unlike endpoint data capture, it does not capture the data for real as it will create tremendous amount of duplications. Instead, it generates manifest files which refer to the transform output S3 location.

Now list the captured data files stored in Amazon S3. There should be different files from different time periods organized based on the hour in which the batch transformation occurred. The format of the Amazon S3 path is:

s3://{data_capture_s3_uri}/input/yyyy/mm/dd/hh/filename.jsonl

[12]:
data_capture_output = f"{data_capture_s3_uri}/input"
captured_data_files = sorted(
    sagemaker.s3.S3Downloader.list(
        s3_uri=data_capture_output,
        sagemaker_session=sagemaker_session,
    )
)
print("Found capture data files:")
print("\n ".join(captured_data_files[-5:]))
Found capture data files:
s3://sagemaker-us-west-2-678264136642/sagemaker/DEMO-ClarifyModelMonitor-1705692266-3764/data-capture/input/2024/01/19/19/1b4bda25-59a8-476c-9bb4-65495d56a050.json
[13]:
captured_data_file = captured_data_files[-1]
captured_data_file_content = sagemaker.s3.S3Downloader.read_file(
    s3_uri=captured_data_files[-1],
    sagemaker_session=sagemaker_session,
)
data_capture_input_dict = json.loads(captured_data_file_content)
print(json.dumps(data_capture_input_dict, indent=4))
[
    {
        "prefix": "s3://sagemaker-us-west-2-678264136642/sagemaker/DEMO-ClarifyModelMonitor-1705692266-3764/test-dataset.json"
    },
    ""
]
[14]:
def upload_captured_data(offset):
    yyyy_mm_dd_hh = "%Y/%m/%d/%H"
    file_path, file_name = os.path.split(captured_data_file)
    this_hour_str = file_path[len(data_capture_output) + 1 :]  # like "2023/01/18/22"
    this_hour = datetime.datetime.strptime(this_hour_str, yyyy_mm_dd_hh)
    next_hour = this_hour + datetime.timedelta(hours=offset)
    next_hour_str = next_hour.strftime(yyyy_mm_dd_hh)  # like "2023/01/18/23"
    sagemaker.s3.S3Uploader.upload_string_as_file_body(
        body=captured_data_file_content,
        desired_s3_uri=f"{data_capture_output}/{next_hour_str}/{file_name}",
        sagemaker_session=sagemaker_session,
    )


# For demostration purpose, only needed for this example
# copy the captured file to the last hour's folder, just in case the first monitoring execution is started in this hour.
upload_captured_data(-1)
# copy the captured file to the next hour's folder, just in case the first monitoring execution is started after next hour.
upload_captured_data(1)

Transform input

The captured data file refers to the transform input file. The cell below shows the first few records of the file.

[15]:
transform_input = data_capture_input_dict[0]["prefix"]
transform_output_content = sagemaker.s3.S3Downloader.read_file(
    s3_uri=transform_input,
    sagemaker_session=sagemaker_session,
).splitlines()
print(*transform_output_content[-5:], sep="\n")
{"instances":[{"features":[28,2,133937,9,13,2,0,0,4,1,15024,0,55,37]},{"features":[43,2,72338,12,14,2,12,0,1,1,0,0,40,37]},{"features":[34,2,162604,11,9,4,2,2,2,1,0,0,40,37]},{"features":[20,2,258509,11,9,4,6,3,2,1,0,0,40,37]},{"features":[27,2,446947,9,13,4,0,4,2,0,0,0,55,37]},{"features":[20,2,95552,11,9,4,11,3,4,1,0,0,40,37]},{"features":[46,2,145636,11,9,2,3,0,4,1,3103,0,50,37]},{"features":[18,2,150675,0,6,4,11,3,4,1,0,0,40,37]},{"features":[22,2,197050,11,9,4,7,3,4,0,0,0,20,37]},{"features":[20,2,246635,15,10,4,11,3,4,0,2597,0,20,37]},{"features":[65,0,200764,11,9,6,0,1,4,0,0,0,40,37]},{"features":[38,2,175665,15,10,2,9,5,4,0,0,0,40,37]},{"features":[34,3,337995,9,13,0,3,4,2,1,15020,0,50,37]},{"features":[42,2,86912,9,13,0,7,1,4,1,0,0,40,37]},{"features":[40,2,100451,15,10,4,2,1,4,1,0,0,40,37]},{"features":[45,2,192360,12,14,2,3,0,4,1,0,1902,50,37]},{"features":[55,2,150507,15,10,2,0,0,4,1,0,0,40,37]},{"features":[36,2,48976,9,13,2,11,5,4,0,0,0,40,37]},{"features":[34,2,111567,15,10,4,3,1,4,1,0,0,40,37]},{"features":[26,2,167350,15,10,2,6,0,4,1,3137,0,50,37]},{"features":[29,2,485944,9,13,4,11,3,2,1,0,0,40,37]},{"features":[44,1,112763,12,14,0,9,4,4,0,0,0,38,37]},{"features":[37,5,195843,11,9,2,2,0,4,1,5013,0,40,37]},{"features":[22,5,181096,9,13,4,9,3,2,1,0,0,20,37]},{"features":[53,2,119170,11,9,2,13,0,2,1,0,1740,40,37]},{"features":[61,1,205711,11,9,2,9,0,4,1,0,0,30,37]},{"features":[46,0,260549,15,10,2,0,0,4,1,0,0,80,37]},{"features":[18,2,129053,1,7,4,7,3,4,1,0,0,28,37]},{"features":[22,2,209034,15,10,4,7,1,4,0,0,0,35,37]},{"features":[29,2,266583,11,9,2,11,0,2,1,2829,0,38,37]},{"features":[30,2,96480,8,11,4,0,3,4,0,0,0,32,37]},{"features":[66,4,331960,11,9,2,2,0,4,1,0,0,20,37]},{"features":[44,2,83891,9,13,0,0,3,1,1,5455,0,40,37]},{"features":[61,5,103575,15,10,0,2,1,4,1,0,0,40,10]},{"features":[38,2,589809,9,13,2,0,0,4,1,0,0,45,37]},{"features":[33,2,214288,11,9,2,6,0,4,1,0,1848,48,37]},{"features":[31,2,280927,9,13,4,3,1,4,0,0,0,40,37]},{"features":[49,2,380922,12,14,2,3,0,4,1,15024,0,80,37]},{"features":[34,2,361497,1,7,2,13,0,4,1,0,0,40,37]},{"features":[37,2,306868,11,9,0,2,4,4,1,0,0,38,37]},{"features":[17,2,364952,0,6,3,7,2,4,1,0,0,40,37]},{"features":[60,2,338833,11,9,4,0,1,2,0,0,0,38,37]},{"features":[30,4,70985,11,9,2,4,0,4,1,0,0,75,37]},{"features":[22,2,240229,11,9,4,0,3,4,0,0,0,40,37]},{"features":[51,2,173987,11,9,2,2,0,4,1,0,0,40,37]},{"features":[29,2,157103,8,11,4,12,3,2,1,0,1974,40,37]},{"features":[42,2,205195,11,9,2,2,0,4,1,0,0,40,37]},{"features":[25,5,120268,15,10,2,2,3,4,1,0,0,50,37]},{"features":[64,2,104973,11,9,2,0,0,4,1,0,0,45,37]},{"features":[38,4,248694,15,10,2,2,0,4,1,0,0,36,37]},{"features":[54,1,108739,1,7,6,10,4,2,0,0,0,40,37]},{"features":[57,2,151874,11,9,2,7,5,2,0,0,0,50,37]},{"features":[27,2,150767,15,10,4,6,3,4,1,0,0,48,37]},{"features":[53,2,239155,15,10,2,3,0,4,1,0,0,50,37]},{"features":[35,2,166497,14,15,2,9,0,4,1,0,1902,60,37]},{"features":[22,2,50610,15,10,4,7,1,4,0,0,0,40,37]},{"features":[52,2,335997,9,13,2,12,0,4,1,7688,0,38,37]},{"features":[27,4,209301,11,9,2,2,0,4,1,0,0,60,37]},{"features":[26,2,247196,15,10,4,5,3,4,1,0,0,35,37]},{"features":[23,2,213902,15,10,4,7,4,4,0,0,0,20,37]},{"features":[25,1,281412,11,9,4,7,3,4,0,0,0,35,37]},{"features":[17,2,154337,1,7,4,7,3,4,0,0,0,13,37]},{"features":[22,2,95647,1,7,4,13,3,1,1,0,0,40,28]},{"features":[32,2,177695,9,13,2,2,0,1,1,0,0,45,17]},{"features":[54,2,64421,15,10,6,12,4,4,0,0,0,40,37]},{"features":[45,2,176341,11,9,0,7,4,4,0,0,0,32,37]},{"features":[20,2,203914,2,8,4,7,3,4,0,0,0,25,37]},{"features":[22,2,23940,11,9,4,3,1,1,1,0,0,40,37]},{"features":[32,2,169768,9,13,5,12,1,2,1,0,0,40,37]},{"features":[36,2,109133,9,13,2,11,0,4,1,0,0,50,37]},{"features":[33,2,41610,11,9,5,2,1,4,1,0,0,40,37]},{"features":[37,2,33440,11,9,5,7,4,4,0,0,0,40,37]},{"features":[46,2,151325,0,6,2,2,0,4,1,0,0,40,37]},{"features":[54,1,182429,11,9,6,13,4,4,0,0,0,38,37]},{"features":[34,2,195748,7,12,4,0,3,2,0,0,0,38,37]},{"features":[22,2,248446,4,3,4,8,1,4,1,0,0,50,12]},{"features":[42,2,188789,5,4,6,5,1,4,0,0,0,35,37]},{"features":[34,2,185480,7,12,4,0,3,4,0,0,0,40,37]},{"features":[39,2,30875,9,13,0,11,4,4,0,0,0,40,37]},{"features":[21,2,116489,15,10,4,9,3,4,0,0,0,40,37]},{"features":[18,2,99591,1,7,4,7,3,4,0,0,0,16,37]},{"features":[43,2,282678,11,9,0,3,1,4,0,0,0,60,37]},{"features":[56,1,238405,11,9,6,0,1,4,0,0,0,40,37]},{"features":[32,1,247156,11,9,2,7,0,2,1,3103,0,38,37]},{"features":[19,2,73461,11,9,4,12,1,2,1,0,0,40,37]},{"features":[35,2,98776,11,9,4,3,1,4,1,0,0,60,37]},{"features":[30,2,232766,11,9,0,7,4,4,0,0,0,40,37]},{"features":[32,2,220333,11,9,2,2,0,4,1,7298,0,46,37]},{"features":[27,2,321456,15,10,2,10,0,4,1,0,0,40,37]},{"features":[41,2,173307,11,9,2,13,0,4,1,0,0,43,37]},{"features":[22,2,351952,15,10,4,0,3,4,0,0,0,38,37]},{"features":[33,2,108438,15,10,2,3,0,4,1,0,0,60,37]},{"features":[30,2,171483,11,9,4,2,3,4,1,0,0,38,37]},{"features":[32,2,453983,11,9,2,5,0,4,1,0,0,44,37]},{"features":[37,2,48779,11,9,4,3,1,4,1,0,0,50,37]},{"features":[42,2,222756,9,13,0,9,4,4,1,7430,0,40,37]},{"features":[49,2,118520,11,9,0,0,1,4,0,0,0,45,37]},{"features":[34,2,199539,8,11,2,2,0,4,1,0,0,48,37]},{"features":[42,2,201343,11,9,2,2,0,4,1,2885,0,40,37]},{"features":[49,2,99340,4,3,5,6,4,4,0,0,0,40,5]},{"features":[48,2,163706,9,13,2,3,0,4,1,15024,0,70,37]},{"features":[59,2,176118,12,14,2,9,0,4,1,0,0,7,37]},{"features":[67,3,147377,11,9,2,3,0,4,1,0,0,45,37]},{"features":[36,2,225330,11,9,0,7,4,4,0,0,0,40,37]},{"features":[32,2,147921,14,15,4,7,1,4,0,0,0,35,37]},{"features":[36,2,110013,12,14,4,11,1,4,0,0,0,40,37]},{"features":[76,4,130585,15,10,2,7,5,4,0,0,0,12,37]},{"features":[41,4,134724,8,11,2,7,5,4,0,3103,0,40,37]},{"features":[44,2,160369,15,10,2,8,0,4,1,0,0,2,37]},{"features":[24,2,172169,15,10,4,5,4,4,1,0,0,30,37]},{"features":[35,2,106471,9,13,4,2,1,4,1,0,0,35,37]},{"features":[25,1,336320,9,13,0,10,1,4,0,0,0,40,37]},{"features":[62,2,186446,15,10,0,12,4,4,0,0,0,43,37]},{"features":[39,2,183279,9,13,2,11,0,4,1,7298,0,40,37]},{"features":[65,4,135517,5,4,2,2,0,4,1,0,0,40,37]},{"features":[48,0,72808,1,7,0,0,1,4,0,0,0,42,37]},{"features":[56,2,197577,11,9,0,7,1,4,0,0,0,40,37]},{"features":[51,3,110327,1,7,2,2,0,4,1,0,0,60,37]},{"features":[23,2,237811,15,10,4,0,4,2,0,0,0,40,36]},{"features":[18,2,632271,15,10,3,0,2,4,0,0,0,40,27]},{"features":[18,2,220754,1,7,4,5,3,4,1,0,0,24,37]},{"features":[61,2,29797,11,9,0,11,2,4,0,0,0,40,37]},{"features":[32,2,183470,8,11,2,2,0,0,1,0,0,42,37]},{"features":[36,2,127388,7,12,2,11,5,4,0,0,0,40,37]},{"features":[19,2,78401,11,9,4,7,3,4,1,0,0,40,37]},{"features":[37,2,385330,5,4,5,7,4,2,1,0,0,40,37]},{"features":[53,2,161691,12,14,0,3,1,4,0,4865,0,40,37]},{"features":[31,2,301251,9,13,2,2,0,4,1,0,0,50,37]},{"features":[30,2,198660,11,9,2,5,0,4,1,0,0,40,37]},{"features":[44,2,105896,9,13,0,9,1,4,0,0,0,36,37]},{"features":[23,2,132220,11,9,2,5,0,4,1,0,0,40,37]},{"features":[45,1,317846,7,12,0,3,4,4,1,0,0,47,37]},{"features":[32,2,33117,8,11,2,7,0,4,1,0,0,40,37]},{"features":[41,2,192602,15,10,2,2,0,4,1,0,0,40,37]},{"features":[30,2,408328,13,1,3,5,4,4,1,0,0,40,24]},{"features":[34,2,233729,7,12,2,9,0,2,1,0,0,50,37]},{"features":[21,2,174063,8,11,4,7,3,4,0,0,0,20,37]},{"features":[30,2,175323,8,11,2,3,5,4,0,0,0,52,37]},{"features":[20,2,460356,2,8,4,7,1,4,1,0,0,30,24]},{"features":[33,2,119422,11,9,2,3,0,4,1,0,0,40,37]},{"features":[26,2,269168,15,10,2,3,0,1,1,0,0,40,37]},{"features":[21,5,173534,15,10,4,9,3,4,0,0,0,40,6]},{"features":[48,2,235891,11,9,4,7,1,4,1,0,0,40,31]},{"features":[70,3,217801,9,13,2,11,0,4,1,0,0,15,37]},{"features":[52,1,251841,12,14,4,9,1,4,0,0,0,50,37]},{"features":[24,2,196943,8,11,2,9,0,4,1,0,0,40,37]},{"features":[41,2,204415,1,7,0,5,1,4,1,0,0,48,37]},{"features":[23,2,130959,9,13,2,9,0,4,1,2407,0,6,1]},{"features":[46,2,316271,4,3,2,2,0,4,1,0,0,55,37]},{"features":[59,2,124137,11,9,0,11,1,4,1,2202,0,40,37]},{"features":[36,4,140676,9,13,4,11,1,4,1,0,0,50,37]},{"features":[52,2,91506,11,9,2,5,0,4,1,0,0,45,37]},{"features":[40,2,300195,15,10,0,12,4,2,0,0,0,40,37]},{"features":[51,3,119570,9,13,2,2,0,4,1,0,0,50,37]},{"features":[43,2,303155,9,13,2,3,0,4,1,0,0,50,37]},{"features":[30,2,210541,11,9,0,2,1,4,0,0,0,40,37]},{"features":[48,2,153312,15,10,2,11,0,2,1,0,0,60,37]},{"features":[50,5,137815,9,13,2,2,0,4,1,0,0,40,37]},{"features":[38,4,179824,11,9,4,4,1,4,1,0,0,50,37]},{"features":[41,2,106159,11,9,4,6,3,4,1,14344,0,48,37]},{"features":[69,2,104827,11,9,6,12,4,4,0,0,0,8,37]},{"features":[21,2,278254,15,10,4,5,3,2,1,0,0,40,37]},{"features":[33,3,287372,15,10,2,3,0,4,1,0,0,50,37]},{"features":[51,5,152810,8,11,2,12,0,4,1,0,0,40,37]},{"features":[46,2,106662,9,13,5,11,1,4,1,99999,0,55,37]},{"features":[35,2,108140,11,9,0,2,1,4,1,0,0,40,37]},{"features":[29,2,231507,11,9,4,2,1,4,1,0,0,35,37]},{"features":[34,4,114074,8,11,6,3,4,4,0,0,0,40,37]},{"features":[52,2,163776,11,9,2,11,0,4,1,0,1902,60,37]},{"features":[45,2,123219,4,3,4,6,1,4,1,0,0,40,37]},{"features":[25,2,391591,11,9,4,2,1,4,1,0,0,50,37]},{"features":[61,1,202384,9,13,2,9,5,4,0,0,0,30,37]},{"features":[58,2,282023,9,13,2,3,0,4,1,0,0,50,37]},{"features":[51,5,22211,11,9,0,3,1,4,1,0,0,37,37]},{"features":[27,2,192936,9,13,4,9,1,4,0,0,0,45,37]},{"features":[51,1,106365,7,12,0,0,4,4,0,0,0,40,37]},{"features":[51,2,166461,1,7,0,6,4,2,0,5455,0,40,37]},{"features":[52,2,251585,0,6,2,13,0,4,1,0,0,55,37]},{"features":[61,1,149981,11,9,6,0,1,4,0,0,0,40,37]},{"features":[23,2,161092,9,13,4,0,3,4,1,0,0,40,37]},{"features":[40,2,21755,15,10,4,2,2,0,1,0,0,30,37]},{"features":[20,2,174436,11,9,4,2,3,4,1,0,0,60,37]},{"features":[26,4,33016,8,11,0,7,4,4,0,0,0,55,37]},{"features":[55,1,134042,12,14,2,3,5,4,0,0,0,40,37]},{"features":[32,2,259425,15,10,0,2,1,4,1,0,0,40,37]},{"features":[26,2,359854,9,13,4,8,2,4,0,0,0,35,24]},{"features":[44,2,217039,14,15,2,9,0,4,1,99999,0,60,37]},{"features":[61,2,194804,13,1,5,13,1,2,1,14344,0,40,37]},{"features":[34,4,198068,11,9,2,2,0,4,1,0,0,40,37]},{"features":[42,4,52131,15,10,4,3,1,4,1,0,0,40,37]},{"features":[23,2,239539,11,9,4,6,3,1,1,0,0,40,28]},{"features":[25,2,54298,11,9,2,11,0,4,1,0,0,30,37]},{"features":[17,2,35603,2,8,4,11,3,4,0,0,0,20,37]},{"features":[31,2,241880,8,11,4,0,1,2,1,0,0,45,37]},{"features":[35,2,46947,15,10,0,0,1,4,0,0,0,45,37]},{"features":[28,2,203171,15,10,0,2,1,4,1,0,0,40,37]},{"features":[37,2,199739,15,10,0,2,3,4,1,0,0,40,37]},{"features":[23,2,215395,15,10,4,2,1,4,1,0,0,40,37]},{"features":[53,2,117932,11,9,0,6,1,4,0,0,0,40,37]},{"features":[30,5,107142,9,13,2,9,0,4,1,0,0,37,37]},{"features":[33,2,173730,8,11,2,6,0,4,1,0,0,40,37]},{"features":[53,3,200400,10,16,0,3,1,4,1,0,0,60,37]},{"features":[50,2,158948,11,9,2,9,0,4,1,0,0,84,37]},{"features":[39,2,206888,15,10,0,0,1,4,0,0,0,40,37]},{"features":[26,2,124483,9,13,4,9,1,1,1,0,0,25,17]},{"features":[34,5,62327,9,13,2,9,0,4,1,0,0,40,37]},{"features":[26,2,366889,11,9,4,13,1,4,1,0,0,40,37]},{"features":[21,2,30796,15,10,4,7,3,4,0,0,0,25,37]},{"features":[46,2,130667,11,9,2,13,0,2,1,0,0,40,37]},{"features":[67,0,231604,11,9,4,0,1,4,1,0,0,40,37]},{"features":[25,2,332409,8,11,2,2,0,4,1,0,0,40,37]},{"features":[34,2,51854,11,9,4,6,1,4,1,0,0,40,37]},{"features":[50,2,62593,8,11,2,4,0,1,1,0,0,40,37]},{"features":[47,2,78954,1,7,0,11,4,4,0,0,0,28,37]},{"features":[39,2,205997,15,10,2,11,5,4,0,0,0,21,37]},{"features":[51,2,231230,11,9,2,6,0,4,1,0,0,45,37]},{"features":[62,2,291904,11,9,0,8,1,2,0,0,0,20,37]},{"features":[58,2,49893,12,14,2,3,0,4,1,0,0,50,37]},{"features":[36,2,141584,15,10,2,9,0,4,1,0,0,50,37]},{"features":[28,2,259609,11,9,4,2,3,4,1,0,0,50,37]},{"features":[22,2,125010,9,13,4,0,1,4,0,0,0,20,37]},{"features":[59,5,136819,12,14,2,9,0,4,1,0,0,8,37]},{"features":[69,4,199829,9,13,2,3,0,4,1,0,1258,40,37]},{"features":[33,4,100580,15,10,2,7,5,4,0,0,0,10,37]},{"features":[56,2,257555,12,14,2,9,0,4,1,0,0,40,37]},{"features":[47,2,100113,5,4,2,13,0,4,1,0,2051,40,37]},{"features":[38,0,236648,11,9,2,2,0,4,1,0,0,40,37]},{"features":[41,2,99679,0,6,2,2,0,4,1,0,0,40,37]},{"features":[32,2,339482,12,14,4,3,1,4,1,0,0,48,37]},{"features":[28,2,120475,11,9,4,2,1,4,1,0,0,35,37]},{"features":[22,2,137876,15,10,4,10,1,4,1,0,0,20,37]},{"features":[36,4,110861,11,9,0,2,3,4,1,0,0,20,37]},{"features":[55,4,225623,15,10,2,4,0,4,1,0,0,40,37]},{"features":[47,2,323212,11,9,6,7,1,4,0,0,0,40,37]},{"features":[59,2,157831,11,9,0,0,1,4,0,0,0,16,37]},{"features":[25,2,25497,15,10,4,13,1,4,1,4101,0,40,37]},{"features":[42,4,114580,12,14,0,3,4,4,0,0,0,70,37]},{"features":[22,2,273675,11,9,3,7,2,2,0,0,0,35,31]},{"features":[31,0,40909,15,10,2,12,0,2,1,0,0,40,37]},{"features":[42,3,557349,9,13,2,3,0,4,1,0,0,70,37]},{"features":[18,2,219256,15,10,4,11,3,4,0,0,0,25,37]},{"features":[39,2,126569,11,9,4,2,1,4,1,0,0,40,29]},{"features":[37,2,108282,9,13,2,3,0,4,1,0,0,45,37]},{"features":[31,2,147270,15,10,4,0,3,4,0,0,0,35,37]},{"features":[44,2,90582,9,13,2,2,0,4,1,0,0,50,37]},{"features":[51,2,379797,0,6,2,6,0,2,1,0,0,40,37]},{"features":[37,1,136749,11,9,4,0,3,4,0,0,0,35,37]},{"features":[25,0,198813,9,13,4,0,4,2,0,0,1590,40,37]},{"features":[30,2,159123,11,9,2,2,0,4,1,0,0,45,37]},{"features":[36,3,196554,11,9,2,2,0,4,1,0,0,46,37]},{"features":[31,2,238002,9,13,2,13,0,4,1,0,0,55,24]},{"features":[43,2,125577,11,9,5,0,4,2,0,0,0,40,37]},{"features":[22,2,97212,11,9,4,7,1,4,0,0,0,15,37]},{"features":[19,2,222866,0,6,4,4,2,4,1,0,0,40,37]},{"features":[18,2,175752,11,9,4,5,3,4,1,0,0,30,37]},{"features":[28,2,77009,15,10,4,11,2,4,0,0,0,40,37]},{"features":[54,2,162745,11,9,2,2,0,4,1,0,0,55,37]},{"features":[30,2,94235,9,13,2,9,0,4,1,0,1977,50,37]},{"features":[19,2,158343,15,10,4,7,3,4,0,0,0,12,37]},{"features":[49,2,201127,1,7,2,13,0,4,1,0,1902,70,37]},{"features":[39,2,118429,15,10,0,11,1,4,1,0,0,40,37]},{"features":[36,2,334365,1,7,2,13,0,4,1,0,0,60,37]},{"features":[42,2,89226,8,11,2,13,0,4,1,0,0,45,37]},{"features":[33,2,56121,11,9,4,13,1,4,1,0,0,60,37]},{"features":[61,5,140851,9,13,2,9,0,4,1,0,0,40,37]},{"features":[36,2,86643,2,8,2,6,0,4,1,0,0,48,37]},{"features":[20,2,175808,11,9,4,2,3,4,1,0,0,40,37]},{"features":[19,2,58471,11,9,4,2,3,4,0,0,0,40,37]},{"features":[55,2,118057,11,9,6,2,4,4,1,0,0,51,37]},{"features":[30,2,192002,15,10,2,2,0,4,1,0,0,40,37]},{"features":[61,2,43904,11,9,0,7,1,2,1,0,0,40,37]},{"features":[39,3,31709,15,10,2,0,5,4,0,0,0,20,37]},{"features":[39,2,286026,9,13,2,2,0,4,1,0,0,52,37]},{"features":[55,4,110844,11,9,2,3,5,4,0,0,0,40,37]},{"features":[32,2,200401,11,9,4,3,1,4,1,0,0,40,3]},{"features":[44,5,101603,9,13,2,3,0,4,1,0,0,40,37]},{"features":[58,2,49159,11,9,2,0,5,4,0,0,0,40,37]},{"features":[52,5,168035,15,10,2,12,0,4,1,0,0,45,37]},{"features":[18,2,260977,2,8,4,11,3,4,0,0,0,20,37]},{"features":[47,2,33794,11,9,2,2,0,4,1,0,0,56,37]},{"features":[26,2,242464,8,11,4,3,1,4,1,0,0,50,37]},{"features":[35,2,97554,7,12,2,3,0,4,1,0,0,50,37]},{"features":[39,4,245361,15,10,4,9,3,4,0,0,0,10,37]},{"features":[26,2,178478,15,10,4,11,3,4,0,0,0,40,37]},{"features":[31,2,104509,15,10,5,7,4,4,0,0,0,35,37]},{"features":[31,2,159187,15,10,2,2,0,4,1,0,0,25,37]},{"features":[67,4,167015,9,13,6,11,1,4,1,0,0,30,37]},{"features":[40,2,199668,11,9,0,11,3,4,0,0,0,25,37]},{"features":[35,2,37778,11,9,2,2,0,4,1,0,0,50,37]},{"features":[54,4,139023,15,10,2,11,0,4,1,0,0,40,37]},{"features":[45,3,188694,14,15,2,9,0,4,1,0,0,50,37]},{"features":[50,2,178251,12,14,2,0,5,4,0,0,0,40,37]},{"features":[51,2,81534,1,7,4,7,2,1,1,0,0,35,37]},{"features":[37,2,353550,12,14,2,3,0,4,1,15024,0,60,37]},{"features":[54,1,231482,11,9,2,2,0,4,1,0,0,40,30]},{"features":[22,2,228394,11,9,4,7,1,4,0,0,0,50,37]},{"features":[38,1,94529,11,9,2,5,5,4,0,3103,0,50,37]},{"features":[35,2,135289,8,11,0,2,1,4,1,0,0,50,37]},{"features":[37,0,32950,7,12,0,3,4,2,0,0,0,40,37]},{"features":[45,2,165346,15,10,0,3,4,4,0,0,0,64,37]},{"features":[57,1,62701,15,10,6,3,1,4,1,6849,0,40,37]},{"features":[30,2,49358,2,8,4,11,3,2,0,0,0,40,37]},{"features":[52,2,227832,9,13,2,9,0,4,1,0,0,50,37]},{"features":[67,2,188903,9,13,2,9,0,4,1,0,0,40,37]},{"features":[28,4,183151,11,9,2,2,0,4,1,0,0,40,37]},{"features":[42,5,116493,9,13,2,10,0,4,1,0,0,52,37]},{"features":[48,1,93449,14,15,2,9,0,1,1,99999,0,40,28]},{"features":[18,2,211683,2,8,4,5,3,4,1,0,0,20,37]},{"features":[47,2,155107,11,9,2,12,0,4,1,0,0,40,37]},{"features":[55,3,150917,15,10,2,3,0,4,1,0,1977,45,37]},{"features":[51,2,135388,2,8,6,6,1,4,1,0,1564,40,37]},{"features":[38,2,183683,0,6,3,7,1,4,1,0,0,45,37]},{"features":[47,4,185859,11,9,2,4,0,4,1,3103,0,60,37]},{"features":[44,4,22933,11,9,2,3,0,4,1,0,0,40,37]},{"features":[40,2,356934,14,15,2,3,0,4,1,0,0,50,37]},{"features":[52,2,94448,8,11,2,9,0,4,1,0,0,40,37]},{"features":[59,2,107318,5,4,2,2,0,4,1,5178,0,50,37]},{"features":[31,2,83413,11,9,4,11,3,4,1,0,0,40,37]},{"features":[34,2,162312,9,13,2,0,0,1,1,0,0,40,28]},{"features":[44,2,118212,0,6,2,6,0,4,1,0,0,40,37]},{"features":[35,1,132879,11,9,2,13,0,4,1,0,0,40,37]},{"features":[25,4,121285,9,13,4,11,1,4,0,0,0,40,37]},{"features":[22,2,341760,9,13,4,3,3,4,0,0,0,40,37]},{"features":[35,2,216473,11,9,0,2,4,4,1,0,0,40,37]},{"features":[25,2,179255,15,10,4,0,3,4,0,0,0,25,37]},{"features":[36,2,298635,9,13,2,7,0,3,1,0,0,40,18]},{"features":[20,2,204596,15,10,4,11,3,4,0,0,0,32,37]},{"features":[27,2,285897,11,9,2,13,0,4,1,0,1887,40,37]},{"features":[19,2,386492,15,10,4,5,3,4,1,0,0,16,37]},{"features":[29,2,178610,15,10,0,7,4,4,0,0,0,21,37]},{"features":[49,2,96854,11,9,0,7,4,4,1,0,0,40,37]},{"features":[45,2,293628,15,10,2,9,0,4,1,0,0,50,28]},{"features":[67,2,192995,11,9,6,0,4,4,0,6723,0,40,37]},{"features":[30,2,235847,9,13,4,7,3,4,0,0,0,24,37]}]}

Model Explainability Monitor

Similar to the other monitoring types, the standard procedure of creating a feature attribution drift monitor is first run a baselining job, and then schedule the monitor.

[16]:
model_explainability_monitor = sagemaker.model_monitor.ModelExplainabilityMonitor(
    role=role,
    sagemaker_session=sagemaker_session,
    max_runtime_in_seconds=3600,
)
INFO:sagemaker.image_uris:Defaulting to the only supported framework/algorithm version: 1.0.
INFO:sagemaker.image_uris:Ignoring unnecessary instance type: None.

Baselining job

A baselining job runs predictions on training dataset and suggests constraints. The suggest_baseline() method of ModelExplainabilityMonitor starts a SageMaker Clarify processing job to generate the constraints.

The step is not mandatory, but providing constraints file to the monitor can enable violations file generation.

Configurations

Information about the input data need to be provided to the processor.

DataConfig stores information about the dataset to be analyzed. For example, the dataset file and its format (like JSON Lines), where to store the analysis results. Some special things to note about this configuration for the JSON Lines dataset,

  • The parameter value "features" or "label" is NOT a header string. Instead, it is a JMESPath expression (refer to its specification) that is used to locate the features list or the ground truth label in the dataset (the ground truth label is not needed for the explainability analysis, the parameter is specified so that the job knows it should be excluded from the dataset). In this example notebook they happen to be the same as the keys in the dataset. But for example, if the dataset has records like below, then the features parameter should use value "data.features.values", and the label parameter should use value "data.label".

    {"data": {"features": {"values": [25, 2, 226802, 1, 7, 4, 6, 3, 2, 1, 0, 0, 40, 37]}, "label": 0}}
    
  • SageMaker Clarify processing job will load the JSON Lines dataset into tabular representation for further analysis, and the parameter headers is the list of column names. The label header shall be the last one in the headers list, and the order of feature headers shall be the same as the order of features in a record.

[17]:
features_jmespath = "instances[*].features"
ground_truth_label_jmespath = "instances[*].label"
data_config = sagemaker.clarify.DataConfig(
    s3_data_input_path=train_data_s3_uri,
    s3_output_path=baselining_output_s3_uri,
    features=features_jmespath,
    label=ground_truth_label_jmespath,
    headers=all_headers,
    dataset_type=dataset_type,
)

ModelConfig is configuration related to model to be used for inferencing. In order to compute SHAP values, the SageMaker Clarify explainer generates synthetic dataset and then get its predictions for the SageMaker model. To accomplish this, the processing job will use the model to create an ephemeral endpoint (also known as “shadow endpoint”). The processing job will delete the shadow endpoint after the computations are completed. One special thing to note about this configuration for the JSON Lines model input and output,

  • content_template is used by SageMaker Clarify processing job to convert the tabular data to the request payload acceptable to the shadow endpoint. To be more specific, the placeholder $features will be replaced by the features list from records. The request payload of a record from the testing dataset happens to be similar to the record itself, like {"features":[28,2,133937,9,13,2,0,0,4,1,15024,0,55,37]}, because both the dataset and the model input conform to the same format.

[18]:
model_config = sagemaker.clarify.ModelConfig(
    model_name=model_name,  # The name of the SageMaker model
    instance_type="ml.m5.xlarge",  # The instance type of the shadow endpoint
    instance_count=1,  # The instance count of the shadow endpoint
    content_type=dataset_type,  # The data format of the model input
    accept_type=dataset_type,  # The data format of the model output
    content_template='{"instances":$records}',
    record_template='{"features":$features}',
)

Currently, the SageMaker Clarify explainer offers a scalable and efficient implementation of SHAP, so the explainability config is SHAPConfig, including

  • baseline: A list of records (at least one) to be used as the baseline dataset in the Kernel SHAP algorithm, each record is JSON object that includes a list of features. It can also be a S3 object URI, the S3 file should be in the same format as dataset.

  • num_samples: Number of samples to be used in the Kernel SHAP algorithm. This number determines the size of the generated synthetic dataset to compute the SHAP values.

  • agg_method: Aggregation method for global SHAP values. Valid values are

    • “mean_abs” (mean of absolute SHAP values for all instances),

    • “median” (median of SHAP values for all instances) and

    • “mean_sq” (mean of squared SHAP values for all instances).

  • use_logit: Indicator of whether the logit function is to be applied to the model predictions. Default is False. If “use_logit” is true then the SHAP values will have log-odds units.

  • save_local_shap_values: Indicator of whether to save the local SHAP values in the output location. Default is True.

[19]:
# Here use the mean value of train dataset as SHAP baseline
dataset = []
with open(train_dataset_path) as f:
    instances = json.load(f)["instances"]
    for instance in instances:
        dataset.append(instance["features"])
mean_values = pd.DataFrame(dataset).mean().round().astype(int).to_list()
mean_record = {"features": mean_values}
shap_baseline = {"instances": [mean_record]}
print(f"SHAP baseline: {shap_baseline}")

shap_config = sagemaker.clarify.SHAPConfig(
    baseline=shap_baseline,
    num_samples=100,
    agg_method="mean_abs",
    save_local_shap_values=False,
)
SHAP baseline: {'instances': [{'features': [39, 2, 184870, 10, 10, 3, 6, 1, 4, 1, 1597, 61, 41, 37]}]}

Kick off baselining job

Call the suggest_baseline() method to start the baselining job. The model output has a key “score” pointing to a confidence score value between 0 and 1. So, the model_scores parameter is set to the JMESPath expression“score” which can locate the score in the model output.

[20]:
confidence_score_jmespath = "predictions[*].score"
model_explainability_monitor.suggest_baseline(
    explainability_config=shap_config,
    data_config=data_config,
    model_config=model_config,
    model_scores=confidence_score_jmespath,  # The JMESPath to locate the confidence score in model output
)
INFO:sagemaker.image_uris:Defaulting to the only supported framework/algorithm version: 1.0.
INFO:sagemaker.image_uris:Ignoring unnecessary instance type: None.
INFO:sagemaker.clarify:Analysis Config: {'dataset_type': 'application/json', 'features': 'instances[*].features', 'headers': ['Age', 'Workclass', 'fnlwgt', 'Education', 'Education-Num', 'Marital Status', 'Occupation', 'Relationship', 'Ethnic group', 'Sex', 'Capital Gain', 'Capital Loss', 'Hours per week', 'Country', 'Target'], 'label': 'instances[*].label', 'predictor': {'model_name': 'DEMO-xgb-churn-pred-model-monitor-1705692267-227f', 'instance_type': 'ml.m5.xlarge', 'initial_instance_count': 1, 'accept_type': 'application/json', 'content_type': 'application/json', 'content_template': '{"instances":$records}', 'record_template': '{"features":$features}', 'label': 'predictions[*].score'}, 'methods': {'report': {'name': 'report', 'title': 'Analysis Report'}, 'shap': {'use_logit': False, 'save_local_shap_values': False, 'baseline': {'instances': [{'features': [39, 2, 184870, 10, 10, 3, 6, 1, 4, 1, 1597, 61, 41, 37]}]}, 'num_samples': 100, 'agg_method': 'mean_abs'}}}
INFO:sagemaker:Creating processing-job with name baseline-suggestion-job-2024-01-19-19-30-16-810
[20]:
<sagemaker.processing.ProcessingJob at 0x7f67b07397f0>

NOTE: The following cell waits until the baselining job is completed (in about 10 minutes). It then inspects the suggested constraints. This step can be skipped, because the monitor to be scheduled will automatically pick up baselining job name and wait for it before monitoring execution.

[21]:
model_explainability_monitor.latest_baselining_job.wait(logs=False)
print()
model_explainability_constraints = model_explainability_monitor.suggested_constraints()
print(f"Suggested constraints: {model_explainability_constraints.file_s3_uri}")
print(
    sagemaker.s3.S3Downloader.read_file(
        s3_uri=model_explainability_constraints.file_s3_uri,
        sagemaker_session=sagemaker_session,
    )
)
.......................................................................................................!
Suggested constraints: s3://sagemaker-us-west-2-678264136642/sagemaker/DEMO-ClarifyModelMonitor-1705692266-3764/baselining-output/analysis.json
{
    "version": "1.0",
    "explanations": {
        "kernel_shap": {
            "0.7956783771514893": {
                "global_shap_values": {
                    "Age": 0.05967323398925334,
                    "Workclass": 0.009303977202541277,
                    "fnlwgt": 0.0011532925777696535,
                    "Education": 0.014668402906540028,
                    "Education-Num": 0.09897500295961109,
                    "Marital Status": 0.05465943541380248,
                    "Occupation": 0.002534111174299059,
                    "Relationship": 0.018197139997990445,
                    "Ethnic group": 0.005443095081746528,
                    "Sex": 0.03218866814815311,
                    "Capital Gain": 0.09933718978948747,
                    "Capital Loss": 0.013533278372092259,
                    "Hours per week": 0.03648060306227074,
                    "Country": 0.004879998492050835
                },
                "expected_value": 0.2506232261657715
            }
        }
    }
}

Monitoring Schedule

With above constraints collected, now call create_monitoring_schedule() method to schedule an hourly model explainability monitor.

If a baselining job has been submitted, then the monitor object will automatically pick up the analysis configuration from the baselining job. But if the baselining step is skipped, or if the capture dataset has different nature than the training dataset, then analysis configuration has to be provided.

ModelConfig is required by ExplainabilityAnalysisConfig for the same reason as it is required by the baselining job. Note that only features are required for computing feature attribution, so ground truth label should be excluded.

Highlights,

  • data_capture_s3_uri is the location of data captured by the batch transform job

  • features_attribute is the JMESPath expression to locate the features in model input, similar to the features parameter of DataConfig.

  • inference_attribute stores the JMESPath expression to locate the confidence score in model output, similar to the model_scores parameter of the suggest_baseline() method.

[22]:
schedule_expression = sagemaker.model_monitor.CronExpressionGenerator.hourly()
[23]:
# Remove label because only features are required for the analysis
headers_without_label_header = copy.deepcopy(all_headers)
headers_without_label_header.remove(label_header)
model_explainability_analysis_config = sagemaker.model_monitor.ExplainabilityAnalysisConfig(
    explainability_config=shap_config,
    model_config=model_config,
    headers=headers_without_label_header,
)
model_explainability_monitor.create_monitoring_schedule(
    analysis_config=model_explainability_analysis_config,
    batch_transform_input=sagemaker.model_monitor.BatchTransformInput(
        data_captured_destination_s3_uri=data_capture_s3_uri,
        destination="/opt/ml/processing/transform",
        dataset_format=sagemaker.model_monitor.MonitoringDatasetFormat.json(lines=False),
        features_attribute=features_jmespath,
        inference_attribute=confidence_score_jmespath,
    ),
    output_s3_uri=monitor_output_s3_uri,
    schedule_cron_expression=schedule_expression,
)
print(
    f"Model explainability monitoring schedule: {model_explainability_monitor.monitoring_schedule_name}"
)
INFO:sagemaker.model_monitor.clarify_model_monitoring:Uploading analysis config to {s3_uri}.
INFO:sagemaker.model_monitor.model_monitoring:Creating Monitoring Schedule with name: monitoring-schedule-2024-01-19-19-39-01-339
Model explainability monitoring schedule: monitoring-schedule-2024-01-19-19-39-01-339

Wait for the first execution

The schedule starts jobs at the previously specified intervals. Code below waits until time crosses the hour boundary (in UTC) to see executions kick off.

Note: Even for an hourly schedule, Amazon SageMaker has a buffer period of 20 minutes to schedule executions. The execution might start in anywhere from zero to ~20 minutes from the hour boundary. This is expected and done for load balancing in the backend.

[24]:
def wait_for_execution_to_start(model_monitor):
    print(
        "An hourly schedule was created above and it will kick off executions ON the hour (plus 0 - 20 min buffer)."
    )

    print("Waiting for the first execution to happen", end="")
    schedule_desc = model_monitor.describe_schedule()
    while "LastMonitoringExecutionSummary" not in schedule_desc:
        schedule_desc = model_monitor.describe_schedule()
        print(".", end="", flush=True)
        time.sleep(60)
    print()
    print("Done! Execution has been created")

    print("Now waiting for execution to start", end="")
    while schedule_desc["LastMonitoringExecutionSummary"]["MonitoringExecutionStatus"] in "Pending":
        schedule_desc = model_monitor.describe_schedule()
        print(".", end="", flush=True)
        time.sleep(10)

    print()
    print("Done! Execution has started")

NOTE: The following cell waits until the first monitoring execution is started. As explained above, the wait could take more than 60 minutes.

[25]:
wait_for_execution_to_start(model_explainability_monitor)
An hourly schedule was created above and it will kick off executions ON the hour (plus 0 - 20 min buffer).
Waiting for the first execution to happen...............................
Done! Execution has been created
Now waiting for execution to start..........
Done! Execution has started

In real world, a monitoring schedule is supposed to be active all the time. But in this example, it can be stopped to avoid incurring extra charges. A stopped schedule will not trigger further executions, but the ongoing execution will continue. And if needed, the schedule can be restarted by start_monitoring_schedule().

[26]:
model_explainability_monitor.stop_monitoring_schedule()
INFO:sagemaker:Stopping Monitoring Schedule with name: monitoring-schedule-2024-01-19-19-39-01-339

Wait for the execution to finish

In the previous cell, the first execution has started. This section waits for the execution to finish so that its analysis results are available. Here are the possible terminal states and what each of them mean:

  • Completed - This means the monitoring execution completed, and no issues were found in the violations report.

  • CompletedWithViolations - This means the execution completed, but constraint violations were detected.

  • Failed - The monitoring execution failed, maybe due to client error (perhaps incorrect role permissions) or infrastructure issues. Further examination of FailureReason and ExitMessage is necessary to identify what exactly happened.

  • Stopped - job exceeded max runtime or was manually stopped.

[27]:
# Waits for the schedule to have last execution in a terminal status.
def wait_for_execution_to_finish(model_monitor):
    schedule_desc = model_monitor.describe_schedule()
    execution_summary = schedule_desc.get("LastMonitoringExecutionSummary")
    if execution_summary is not None:
        print("Waiting for execution to finish", end="")
        while execution_summary["MonitoringExecutionStatus"] not in [
            "Completed",
            "CompletedWithViolations",
            "Failed",
            "Stopped",
        ]:
            print(".", end="", flush=True)
            time.sleep(60)
            schedule_desc = model_monitor.describe_schedule()
            execution_summary = schedule_desc["LastMonitoringExecutionSummary"]
        print()
        print(f"Done! Execution Status: {execution_summary['MonitoringExecutionStatus']}")
    else:
        print("Last execution not found")

NOTE: The following cell takes about 10 minutes.

[28]:
wait_for_execution_to_finish(model_explainability_monitor)
Waiting for execution to finish..........
Done! Execution Status: Completed

Inspect execution results

List the generated reports,

  • analysis.json includes the global SHAP values.

  • report.* files are static report files to visualize the SHAP values.

[29]:
schedule_desc = model_explainability_monitor.describe_schedule()
execution_summary = schedule_desc.get("LastMonitoringExecutionSummary")
if execution_summary and execution_summary["MonitoringExecutionStatus"] in [
    "Completed",
    "CompletedWithViolations",
]:
    last_model_explainability_monitor_execution = model_explainability_monitor.list_executions()[-1]
    last_model_explainability_monitor_execution_report_uri = (
        last_model_explainability_monitor_execution.output.destination
    )
    print(f"Report URI: {last_model_explainability_monitor_execution_report_uri}")
    last_model_explainability_monitor_execution_report_files = sorted(
        sagemaker.s3.S3Downloader.list(
            s3_uri=last_model_explainability_monitor_execution_report_uri,
            sagemaker_session=sagemaker_session,
        )
    )
    print("Found Report Files:")
    print("\n ".join(last_model_explainability_monitor_execution_report_files))
else:
    last_model_explainability_monitor_execution = None
    print(
        "====STOP==== \n No completed executions to inspect further. Please wait till an execution completes or investigate previously reported failures."
    )
    print(schedule_desc)
Report URI: s3://sagemaker-us-west-2-678264136642/sagemaker/DEMO-ClarifyModelMonitor-1705692266-3764/monitor-output/monitoring-schedule-2024-01-19-19-39-01-339/2024/01/19/20
Found Report Files:
s3://sagemaker-us-west-2-678264136642/sagemaker/DEMO-ClarifyModelMonitor-1705692266-3764/monitor-output/monitoring-schedule-2024-01-19-19-39-01-339/2024/01/19/20/analysis.json
 s3://sagemaker-us-west-2-678264136642/sagemaker/DEMO-ClarifyModelMonitor-1705692266-3764/monitor-output/monitoring-schedule-2024-01-19-19-39-01-339/2024/01/19/20/report.html
 s3://sagemaker-us-west-2-678264136642/sagemaker/DEMO-ClarifyModelMonitor-1705692266-3764/monitor-output/monitoring-schedule-2024-01-19-19-39-01-339/2024/01/19/20/report.ipynb
 s3://sagemaker-us-west-2-678264136642/sagemaker/DEMO-ClarifyModelMonitor-1705692266-3764/monitor-output/monitoring-schedule-2024-01-19-19-39-01-339/2024/01/19/20/report.pdf

If there are any violations compared to the baseline, they are listed here. See Feature Attribution Drift Violations for the schema of the file, and how violations are detected.

[30]:
violations = model_explainability_monitor.latest_monitoring_constraint_violations()
if violations is not None:
    pprint.PrettyPrinter(indent=4).pprint(violations.body_dict)

Could not retrieve constraints file at location 's3://sagemaker-us-west-2-678264136642/sagemaker/DEMO-ClarifyModelMonitor-1705692266-3764/monitor-output/monitoring-schedule-2024-01-19-19-39-01-339/2024/01/19/20/constraint_violations.json'. To manually retrieve ConstraintViolations object from a given uri, use 'my_model_monitor.constraints(my_s3_uri)' or 'ConstraintViolations.from_s3_uri(my_s3_uri)'

By default, the analysis results are also published to CloudWatch, see CloudWatch Metrics for Feature Attribution Drift Analysis.

Cleanup

If there is no plan to collect more data for feature attribution drift monitoring, then the monitor should be stopped (and deleted) to avoid incurring additional charges. Note that deleting the monitor does not delete the data in S3.

[31]:
model_explainability_monitor.stop_monitoring_schedule()
wait_for_execution_to_finish(model_explainability_monitor)
model_explainability_monitor.delete_monitoring_schedule()
INFO:sagemaker:Stopping Monitoring Schedule with name: monitoring-schedule-2024-01-19-19-39-01-339
INFO:sagemaker:Deleting Monitoring Schedule with name: monitoring-schedule-2024-01-19-19-39-01-339
Waiting for execution to finish
Done! Execution Status: Completed
INFO:sagemaker.model_monitor.clarify_model_monitoring:Deleting Model Explainability Job Definition with name: model-explainability-job-definition-2024-01-19-19-39-01-339
[32]:
sagemaker_session.delete_model(model_name)
INFO:sagemaker:Deleting model with name: DEMO-xgb-churn-pred-model-monitor-1705692267-227f

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.

This us-east-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This us-east-2 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This us-west-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This ca-central-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This sa-east-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This eu-west-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This eu-west-2 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This eu-west-3 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This eu-central-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This eu-north-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This ap-southeast-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This ap-southeast-2 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This ap-northeast-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This ap-northeast-2 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This ap-south-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable