Part 1: Distributed data parallel MNIST training with PyTorch and SageMaker distributed

This notebook’s CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

This us-west-2 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

Background

Amazon SageMaker’s distributed library can be used to train deep learning models faster and cheaper. The data parallel feature in this library is a distributed data parallel training framework for PyTorch, TensorFlow, and MXNet. This notebook demonstrates how to use the SageMaker distributed data library to train a PyTorch model using the MNIST dataset.

This notebook example shows how to use smdistributed.dataparallel with PyTorch in SageMaker using MNIST dataset.

For more information:

  1. SageMaker distributed data parallel PyTorch API Specification

  2. Getting started with SageMaker distributed data parallel

  3. PyTorch in SageMaker

Dataset

This example uses the MNIST dataset. MNIST is a widely used dataset for handwritten digit classification. It consists of 70,000 labeled 28x28 pixel grayscale images of hand-written digits. The dataset is split into 60,000 training images and 10,000 test images. There are 10 classes (one for each of the 10 digits).

NOTE: This example requires SageMaker Python SDK v2.**.

[1]:

!pip install sagemaker --upgrade

SageMaker role

The following code cell defines role which is the IAM role ARN used to create and run SageMaker training and hosting jobs. This is the same IAM role used to create this SageMaker Notebook instance.

role must have permission to create a SageMaker training job and launch an endpoint to host a model. For granular policies you can use to grant these permissions, see Amazon SageMaker Roles.

[2]:

import sagemaker

sagemaker_session = sagemaker.Session()
role = sagemaker.get_execution_role()
role_name = role.split(["/"][-1])
print(f"The Amazon Resource Name (ARN) of the role used for this demo is: {role}")
print(f"The name of the role used for this demo is: {role_name[-1]}")

To verify that the role above has required permissions:

  1. Go to the IAM console: https://console.aws.amazon.com/iam/home.

  2. Select Roles.

  3. Enter the role name in the search box to search for that role.

  4. Select the role.

  5. Use the Permissions tab to verify this role has required permissions attached.

Model training with SageMaker distributed data parallel

Training script

The MNIST dataset is downloaded using the torchvision.datasets PyTorch module; you can see how this is implemented in the train_pytorch_smdataparallel_mnist.py training script that is printed out in the next cell.

The training script provides the code you need for distributed data parallel (DDP) training using SageMaker’s distributed data parallel library (smdistributed.dataparallel). The training script is very similar to a PyTorch training script you might run outside SageMaker, but modified to run with the smdistributed.dataparallel library. This library’s PyTorch client provides an alternative to PyTorch’s native DDP.

For details about how to use smdistributed.dataparallel’s DDP in your native PyTorch script, see the Modify a PyTorch Training Script Using SMD Data Parallel.

[3]:

!pygmentize code/train_pytorch_smdataparallel_mnist.py

Estimator function options

In the following code block, you can update the estimator function to use a different instance type, instance count, and distribution strategy. You’re also passing in the training script you reviewed in the previous cell to this estimator.

Instance types

smdistributed.dataparallel supports model training on SageMaker with the following instance types only. For best performance, it is recommended you use an instance type that supports Amazon Elastic Fabric Adapter (ml.p3dn.24xlarge and ml.p4d.24xlarge).

  1. ml.p3.16xlarge

  2. ml.p3dn.24xlarge [Recommended]

  3. ml.p4d.24xlarge [Recommended]

If you want to use another instance type where SM DDP is not supported, you can change simply change the distribution parameter to use another PyTorch launcher (as detailed here), with no required changes to your training code.

Instance count

In order to take advantage of the smdistributed.dataparallel, you should use at least 2 instances.

Distribution strategy

Note that to use DDP mode, you update the distribution strategy, and set it to use smdistributed dataparallel.

[4]:

from sagemaker.pytorch import PyTorch

estimator = PyTorch(
    base_job_name="pytorch-smdataparallel-mnist",
    source_dir="code",
    entry_point="train_pytorch_smdataparallel_mnist.py",
    role=role,
    framework_version="2.0.1",
    py_version="py310",
    # For training with multinode distributed training, set this count. Example: 2
    instance_count=2,
    # For training with p3dn instance use - ml.p3dn.24xlarge, with p4dn instance use - ml.p4d.24xlarge
    instance_type="ml.p4d.24xlarge",
    # instance_type="ml.g5.48xlarge",
    sagemaker_session=sagemaker_session,
    # Training using SMDataParallel Distributed Training Framework
    distribution={"smdistributed": {"dataparallel": {"enabled": True}}},
    # Training with torchrun launcher, for instance types that do not support smddp
    # distribution={ "torch_distributed": { "enabled": True } },
    hyperparameters={"region": sagemaker_session.boto_region_name},
    debugger_hook_config=False,
)

[ ]:

estimator.fit()

INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.
INFO:sagemaker:Creating training-job with name: pytorch-smdataparallel-mnist-2024-05-31-00-58-51-930

2024-05-31 00:58:55 Starting - Starting the training job...
2024-05-31 00:58:57 Pending - Training job waiting for capacity............
2024-05-31 01:01:10 Pending - Preparing the instances for training........................
2024-05-31 01:05:38 Downloading - Downloading input data...
2024-05-31 01:06:07 Downloading - Downloading the training image...............
2024-05-31 01:08:58 Training - Training image download completed. Training in progress.......bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
2024-05-31 01:09:54,297 sagemaker-training-toolkit INFO     Imported framework sagemaker_pytorch_container.training
2024-05-31 01:09:54,391 sagemaker-training-toolkit INFO     No Neurons detected (normal if no neurons installed)
2024-05-31 01:09:54,400 sagemaker_pytorch_container.training INFO     Block until all host DNS lookups succeed.
2024-05-31 01:09:54,401 sagemaker_pytorch_container.training INFO     Invoking SMDataParallel
2024-05-31 01:09:54,401 sagemaker_pytorch_container.training INFO     Invoking user training script.
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
2024-05-31 01:09:52,911 sagemaker-training-toolkit INFO     Imported framework sagemaker_pytorch_container.training
2024-05-31 01:09:53,009 sagemaker-training-toolkit INFO     No Neurons detected (normal if no neurons installed)
2024-05-31 01:09:53,018 sagemaker_pytorch_container.training INFO     Block until all host DNS lookups succeed.
2024-05-31 01:09:53,020 sagemaker_pytorch_container.training INFO     Invoking SMDataParallel
2024-05-31 01:09:53,020 sagemaker_pytorch_container.training INFO     Invoking user training script.
2024-05-31 01:09:54,488 sagemaker-training-toolkit INFO     No Neurons detected (normal if no neurons installed)
2024-05-31 01:09:55,921 sagemaker-training-toolkit INFO     No Neurons detected (normal if no neurons installed)
2024-05-31 01:09:57,392 sagemaker-training-toolkit INFO     No Neurons detected (normal if no neurons installed)
2024-05-31 01:09:57,401 sagemaker-training-toolkit INFO     Starting MPI run as worker node.
2024-05-31 01:09:57,401 sagemaker-training-toolkit INFO     Creating SSH daemon.
2024-05-31 01:09:57,402 sagemaker-training-toolkit INFO     Waiting for MPI workers to establish their SSH connections
2024-05-31 01:09:57,429 sagemaker-training-toolkit INFO     Cannot connect to host algo-1 at port 22. Retrying...
2024-05-31 01:09:57,429 sagemaker-training-toolkit INFO     Connection closed
2024-05-31 01:09:58,754 sagemaker-training-toolkit INFO     No Neurons detected (normal if no neurons installed)
2024-05-31 01:09:58,763 sagemaker-training-toolkit INFO     Starting MPI run as worker node.
2024-05-31 01:09:58,763 sagemaker-training-toolkit INFO     Waiting for MPI Master to create SSH daemon.
2024-05-31 01:09:58,787 paramiko.transport INFO     Connected (version 2.0, client OpenSSH_8.2p1)
2024-05-31 01:09:58,916 paramiko.transport INFO     Authentication (publickey) successful!
2024-05-31 01:09:58,916 sagemaker-training-toolkit INFO     Can connect to host algo-2
2024-05-31 01:09:58,916 sagemaker-training-toolkit INFO     MPI Master online, creating SSH daemon.
2024-05-31 01:09:58,916 sagemaker-training-toolkit INFO     Writing environment variables to /etc/environment for the MPI process.
2024-05-31 01:09:58,918 sagemaker-training-toolkit INFO     Waiting for MPI process to finish.
2024-05-31 01:09:58,461 sagemaker-training-toolkit INFO     Cannot connect to host algo-1 at port 22. Retrying...
2024-05-31 01:09:58,461 sagemaker-training-toolkit INFO     Connection closed
2024-05-31 01:09:59,472 paramiko.transport INFO     Connected (version 2.0, client OpenSSH_8.2p1)
2024-05-31 01:09:59,599 paramiko.transport INFO     Authentication (publickey) successful!
2024-05-31 01:09:59,599 sagemaker-training-toolkit INFO     Can connect to host algo-1 at port 22
2024-05-31 01:09:59,599 sagemaker-training-toolkit INFO     Connection closed
2024-05-31 01:09:59,600 sagemaker-training-toolkit INFO     Worker algo-1 available for communication
2024-05-31 01:09:59,600 sagemaker-training-toolkit INFO     Network interface name: eth0
2024-05-31 01:09:59,600 sagemaker-training-toolkit INFO     Host: ['algo-2', 'algo-1']
2024-05-31 01:09:59,718 sagemaker-training-toolkit INFO     No Neurons detected (normal if no neurons installed)
2024-05-31 01:09:59,727 sagemaker-training-toolkit INFO     instance type: ml.p4d.24xlarge
2024-05-31 01:09:59,727 sagemaker-training-toolkit INFO     Env Hosts: ['algo-2', 'algo-1'] Hosts: ['algo-2:8', 'algo-1:8'] process_per_hosts: 8 num_processes: 16
2024-05-31 01:09:59,820 sagemaker-training-toolkit INFO     No Neurons detected (normal if no neurons installed)
2024-05-31 01:09:59,829 sagemaker-training-toolkit INFO     Invoking user script
Training Env:
{
    "additional_framework_parameters": {
        "sagemaker_distributed_dataparallel_custom_mpi_options": "",
        "sagemaker_distributed_dataparallel_enabled": true,
        "sagemaker_instance_type": "ml.p4d.24xlarge"
    },
    "channel_input_dirs": {},
    "current_host": "algo-2",
    "current_instance_group": "homogeneousCluster",
    "current_instance_group_hosts": [
        "algo-2",
        "algo-1"
    ],
    "current_instance_type": "ml.p4d.24xlarge",
    "distribution_hosts": [
        "algo-2",
        "algo-1"
    ],
    "distribution_instance_groups": [
        "homogeneousCluster"
    ],
    "framework_module": "sagemaker_pytorch_container.training:main",
    "hosts": [
        "algo-1",
        "algo-2"
    ],
    "hyperparameters": {
        "region": "us-west-2"
    },
    "input_config_dir": "/opt/ml/input/config",
    "input_data_config": {},
    "input_dir": "/opt/ml/input",
    "instance_groups": [
        "homogeneousCluster"
    ],
    "instance_groups_dict": {
        "homogeneousCluster": {
            "instance_group_name": "homogeneousCluster",
            "instance_type": "ml.p4d.24xlarge",
            "hosts": [
                "algo-2",
                "algo-1"
            ]
        }
    },
    "is_hetero": false,
    "is_master": true,
    "is_modelparallel_enabled": null,
    "is_smddpmprun_installed": false,
    "is_smddprun_installed": true,
    "job_name": "pytorch-smdataparallel-mnist-2024-05-31-00-58-51-930",
    "log_level": 20,
    "master_hostname": "algo-2",
    "model_dir": "/opt/ml/model",
    "module_dir": "s3://sagemaker-us-west-2-570106654206/pytorch-smdataparallel-mnist-2024-05-31-00-58-51-930/source/sourcedir.tar.gz",
    "module_name": "train_pytorch_smdataparallel_mnist",
    "network_interface_name": "eth0",
    "num_cpus": 96,
    "num_gpus": 8,
    "num_neurons": 0,
    "output_data_dir": "/opt/ml/output/data",
    "output_dir": "/opt/ml/output",
    "output_intermediate_dir": "/opt/ml/output/intermediate",
    "resource_config": {
        "current_host": "algo-2",
        "current_instance_type": "ml.p4d.24xlarge",
        "current_group_name": "homogeneousCluster",
        "hosts": [
            "algo-1",
            "algo-2"
        ],
        "instance_groups": [
            {
                "instance_group_name": "homogeneousCluster",
                "instance_type": "ml.p4d.24xlarge",
                "hosts": [
                    "algo-2",
                    "algo-1"
                ]
            }
        ],
        "network_interface_name": "eth0"
    },
    "user_entry_point": "train_pytorch_smdataparallel_mnist.py"
}
Environment variables:
SM_HOSTS=["algo-1","algo-2"]
SM_NETWORK_INTERFACE_NAME=eth0
SM_HPS={"region":"us-west-2"}
SM_USER_ENTRY_POINT=train_pytorch_smdataparallel_mnist.py
SM_FRAMEWORK_PARAMS={"sagemaker_distributed_dataparallel_custom_mpi_options":"","sagemaker_distributed_dataparallel_enabled":true,"sagemaker_instance_type":"ml.p4d.24xlarge"}
SM_RESOURCE_CONFIG={"current_group_name":"homogeneousCluster","current_host":"algo-2","current_instance_type":"ml.p4d.24xlarge","hosts":["algo-1","algo-2"],"instance_groups":[{"hosts":["algo-2","algo-1"],"instance_group_name":"homogeneousCluster","instance_type":"ml.p4d.24xlarge"}],"network_interface_name":"eth0"}
SM_INPUT_DATA_CONFIG={}
SM_OUTPUT_DATA_DIR=/opt/ml/output/data
SM_CHANNELS=[]
SM_CURRENT_HOST=algo-2
SM_CURRENT_INSTANCE_TYPE=ml.p4d.24xlarge
SM_CURRENT_INSTANCE_GROUP=homogeneousCluster
SM_CURRENT_INSTANCE_GROUP_HOSTS=["algo-2","algo-1"]
SM_INSTANCE_GROUPS=["homogeneousCluster"]
SM_INSTANCE_GROUPS_DICT={"homogeneousCluster":{"hosts":["algo-2","algo-1"],"instance_group_name":"homogeneousCluster","instance_type":"ml.p4d.24xlarge"}}
SM_DISTRIBUTION_INSTANCE_GROUPS=["homogeneousCluster"]
SM_IS_HETERO=false
SM_MODULE_NAME=train_pytorch_smdataparallel_mnist
SM_LOG_LEVEL=20
SM_FRAMEWORK_MODULE=sagemaker_pytorch_container.training:main
SM_INPUT_DIR=/opt/ml/input
SM_INPUT_CONFIG_DIR=/opt/ml/input/config
SM_OUTPUT_DIR=/opt/ml/output
SM_NUM_CPUS=96
SM_NUM_GPUS=8
SM_NUM_NEURONS=0
SM_MODEL_DIR=/opt/ml/model
SM_MODULE_DIR=s3://sagemaker-us-west-2-570106654206/pytorch-smdataparallel-mnist-2024-05-31-00-58-51-930/source/sourcedir.tar.gz
SM_TRAINING_ENV={"additional_framework_parameters":{"sagemaker_distributed_dataparallel_custom_mpi_options":"","sagemaker_distributed_dataparallel_enabled":true,"sagemaker_instance_type":"ml.p4d.24xlarge"},"channel_input_dirs":{},"current_host":"algo-2","current_instance_group":"homogeneousCluster","current_instance_group_hosts":["algo-2","algo-1"],"current_instance_type":"ml.p4d.24xlarge","distribution_hosts":["algo-2","algo-1"],"distribution_instance_groups":["homogeneousCluster"],"framework_module":"sagemaker_pytorch_container.training:main","hosts":["algo-1","algo-2"],"hyperparameters":{"region":"us-west-2"},"input_config_dir":"/opt/ml/input/config","input_data_config":{},"input_dir":"/opt/ml/input","instance_groups":["homogeneousCluster"],"instance_groups_dict":{"homogeneousCluster":{"hosts":["algo-2","algo-1"],"instance_group_name":"homogeneousCluster","instance_type":"ml.p4d.24xlarge"}},"is_hetero":false,"is_master":true,"is_modelparallel_enabled":null,"is_smddpmprun_installed":false,"is_smddprun_installed":true,"job_name":"pytorch-smdataparallel-mnist-2024-05-31-00-58-51-930","log_level":20,"master_hostname":"algo-2","model_dir":"/opt/ml/model","module_dir":"s3://sagemaker-us-west-2-570106654206/pytorch-smdataparallel-mnist-2024-05-31-00-58-51-930/source/sourcedir.tar.gz","module_name":"train_pytorch_smdataparallel_mnist","network_interface_name":"eth0","num_cpus":96,"num_gpus":8,"num_neurons":0,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_group_name":"homogeneousCluster","current_host":"algo-2","current_instance_type":"ml.p4d.24xlarge","hosts":["algo-1","algo-2"],"instance_groups":[{"hosts":["algo-2","algo-1"],"instance_group_name":"homogeneousCluster","instance_type":"ml.p4d.24xlarge"}],"network_interface_name":"eth0"},"user_entry_point":"train_pytorch_smdataparallel_mnist.py"}
SM_USER_ARGS=["--region","us-west-2"]
SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
SM_HP_REGION=us-west-2
PYTHONPATH=/opt/ml/code:/opt/conda/bin:/opt/conda/lib/python310.zip:/opt/conda/lib/python3.10:/opt/conda/lib/python3.10/lib-dynload:/opt/conda/lib/python3.10/site-packages
Invoking script with the following command:
mpirun --host algo-2:8,algo-1:8 -np 16 --allow-run-as-root --tag-output --oversubscribe -mca btl_tcp_if_include eth0 -mca oob_tcp_if_include eth0 -mca plm_rsh_no_tree_spawn 1 -mca pml ob1 -mca btl ^openib -mca orte_abort_on_non_zero_status 1 -mca btl_vader_single_copy_mechanism none -mca plm_rsh_num_concurrent 2 -x NCCL_SOCKET_IFNAME=eth0 -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH -x SMDATAPARALLEL_USE_HOMOGENEOUS=1 -x FI_PROVIDER=efa -x RDMAV_FORK_SAFE=1 -x LD_PRELOAD=/opt/conda/lib/python3.10/site-packages/gethostname.cpython-310-x86_64-linux-gnu.so -x NCCL_PROTO=simple -x FI_EFA_USE_DEVICE_RDMA=1 -x SMDATAPARALLEL_SERVER_ADDR=algo-2 -x SMDATAPARALLEL_SERVER_PORT=7592 -x SAGEMAKER_INSTANCE_TYPE=ml.p4d.24xlarge smddprun /opt/conda/bin/python3.10 -m mpi4py train_pytorch_smdataparallel_mnist.py --region us-west-2
2024-05-31 01:09:59,851 sagemaker-training-toolkit INFO     smdistributed.dataparallel not found or using an older version without custom exceptions.SM training toolkit will track user script error only
Warning: Permanently added 'algo-1,10.0.170.27' (ECDSA) to the list of known hosts.
2024-05-31 01:10:00,923 sagemaker-training-toolkit INFO     Process[es]: [psutil.Process(pid=67, name='orted', status='sleeping', started='01:10:00')]
2024-05-31 01:10:00,923 sagemaker-training-toolkit INFO     Orted process found [psutil.Process(pid=67, name='orted', status='sleeping', started='01:10:00')]
2024-05-31 01:10:00,923 sagemaker-training-toolkit INFO     Waiting for orted process [psutil.Process(pid=67, name='orted', status='sleeping', started='01:10:00')]
[1,mpirank:0,algo-1]<stderr>:curl: /opt/conda/lib/libcurl.so.4: no version information available (required by curl)
[1,mpirank:8,algo-2]<stderr>:curl: /opt/conda/lib/libcurl.so.4: no version information available (required by curl)
[1,mpirank:0,algo-1]<stderr>:curl: /opt/conda/lib/libcurl.so.4: no version information available (required by curl)
[1,mpirank:8,algo-2]<stderr>:curl: /opt/conda/lib/libcurl.so.4: no version information available (required by curl)
[1,mpirank:0,algo-1]<stderr>:curl: /opt/conda/lib/libcurl.so.4: no version information available (required by curl)
[1,mpirank:8,algo-2]<stderr>:curl: /opt/conda/lib/libcurl.so.4: no version information available (required by curl)
[1,mpirank:1,algo-1]<stdout>:DDP Mode
[1,mpirank:1,algo-1]<stdout>:Using smddp as backend
[1,mpirank:5,algo-1]<stdout>:DDP Mode
[1,mpirank:5,algo-1]<stdout>:Using smddp as backend
[1,mpirank:3,algo-1]<stdout>:DDP Mode
[1,mpirank:3,algo-1]<stdout>:Using smddp as backend
[1,mpirank:0,algo-1]<stdout>:DDP Mode
[1,mpirank:0,algo-1]<stdout>:Using smddp as backend
[1,mpirank:6,algo-1]<stdout>:DDP Mode
[1,mpirank:6,algo-1]<stdout>:Using smddp as backend
[1,mpirank:4,algo-1]<stdout>:DDP Mode
[1,mpirank:4,algo-1]<stdout>:Using smddp as backend
[1,mpirank:7,algo-1]<stdout>:DDP Mode
[1,mpirank:7,algo-1]<stdout>:Using smddp as backend
[1,mpirank:2,algo-1]<stdout>:DDP Mode
[1,mpirank:2,algo-1]<stdout>:Using smddp as backend
[1,mpirank:11,algo-2]<stdout>:DDP Mode
[1,mpirank:11,algo-2]<stdout>:Using smddp as backend
[1,mpirank:9,algo-2]<stdout>:DDP Mode
[1,mpirank:9,algo-2]<stdout>:Using smddp as backend
[1,mpirank:13,algo-2]<stdout>:DDP Mode
[1,mpirank:13,algo-2]<stdout>:Using smddp as backend
[1,mpirank:8,algo-2]<stdout>:DDP Mode
[1,mpirank:8,algo-2]<stdout>:Using smddp as backend
[1,mpirank:15,algo-2]<stdout>:DDP Mode
[1,mpirank:15,algo-2]<stdout>:Using smddp as backend
[1,mpirank:10,algo-2]<stdout>:DDP Mode
[1,mpirank:10,algo-2]<stdout>:Using smddp as backend
[1,mpirank:12,algo-2]<stdout>:DDP Mode
[1,mpirank:12,algo-2]<stdout>:Using smddp as backend
[1,mpirank:14,algo-2]<stdout>:DDP Mode
[1,mpirank:14,algo-2]<stdout>:Using smddp as backend
[1,mpirank:0,algo-1]<stdout>:Running smdistributed.dataparallel v2.0.1
[1,mpirank:0,algo-1]<stdout>:SMDDP: Multi node EFA mode
[1,mpirank:1,algo-1]<stdout>:Successfully opened device rdmap16s27
[1,mpirank:0,algo-1]<stdout>:Successfully opened device rdmap16s27
[1,mpirank:7,algo-1]<stdout>:Successfully opened device rdmap160s27
[1,mpirank:5,algo-1]<stdout>:Successfully opened device rdmap144s27
[1,mpirank:4,algo-1]<stdout>:Successfully opened device rdmap144s27
[1,mpirank:3,algo-1]<stdout>:Successfully opened device rdmap32s27
[1,mpirank:2,algo-1]<stdout>:Successfully opened device rdmap32s27
[1,mpirank:6,algo-1]<stdout>:Successfully opened device rdmap160s27
[1,mpirank:13,algo-2]<stdout>:Successfully opened device rdmap144s27
[1,mpirank:11,algo-2]<stdout>:Successfully opened device rdmap32s27
[1,mpirank:8,algo-2]<stdout>:Successfully opened device rdmap16s27
[1,mpirank:10,algo-2]<stdout>:Successfully opened device rdmap32s27
[1,mpirank:15,algo-2]<stdout>:Successfully opened device rdmap160s27
[1,mpirank:12,algo-2]<stdout>:Successfully opened device rdmap144s27
[1,mpirank:14,algo-2]<stdout>:Successfully opened device rdmap160s27
[1,mpirank:9,algo-2]<stdout>:Successfully opened device rdmap16s27
[1,mpirank:0,algo-1]<stdout>:Downloading https://sagemaker-example-files-prod-us-west-2.s3.amazonaws.com/datasets/image/MNIST/train-images-idx3-ubyte.gz
[1,mpirank:8,algo-2]<stdout>:Downloading https://sagemaker-example-files-prod-us-west-2.s3.amazonaws.com/datasets/image/MNIST/train-images-idx3-ubyte.gz
[1,mpirank:0,algo-1]<stdout>:Downloading https://sagemaker-example-files-prod-us-west-2.s3.amazonaws.com/datasets/image/MNIST/train-images-idx3-ubyte.gz to /tmp/data/MNIST/raw/train-images-idx3-ubyte.gz
[1,mpirank:8,algo-2]<stdout>:Downloading https://sagemaker-example-files-prod-us-west-2.s3.amazonaws.com/datasets/image/MNIST/train-images-idx3-ubyte.gz to /tmp/data/MNIST/raw/train-images-idx3-ubyte.gz
[1,mpirank:8,algo-2]<stderr>:#015  0%|          | 0/9912422 [00:00<?, ?it/s]
[1,mpirank:0,algo-1]<stderr>:#015  0%|          | 0/9912422 [00:00<?, ?it/s]
[1,mpirank:0,algo-1]<stderr>:#015 67%|██████▋   | 6619136/9912422 [00:00<00:00, 56403769.51it/s]
[1,mpirank:8,algo-2]<stderr>:#015 67%|██████▋   | 6619136/9912422 [00:00<00:00, 52056890.69it/s]
[1,mpirank:0,algo-1]<stderr>:#015100%|██████████| 9912422/9912422 [00:00<00:00, 57476299.63it/s]
[1,mpirank:8,algo-2]<stderr>:#015100%|██████████| 9912422/9912422 [00:00<00:00, 53835928.82it/s]
[1,mpirank:0,algo-1]<stdout>:Extracting /tmp/data/MNIST/raw/train-images-idx3-ubyte.gz to /tmp/data/MNIST/raw
[1,mpirank:8,algo-2]<stdout>:Extracting /tmp/data/MNIST/raw/train-images-idx3-ubyte.gz to /tmp/data/MNIST/raw
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Downloading https://sagemaker-example-files-prod-us-west-2.s3.amazonaws.com/datasets/image/MNIST/train-labels-idx1-ubyte.gz
[1,mpirank:8,algo-2]<stdout>:
[1,mpirank:8,algo-2]<stdout>:Downloading https://sagemaker-example-files-prod-us-west-2.s3.amazonaws.com/datasets/image/MNIST/train-labels-idx1-ubyte.gz
[1,mpirank:0,algo-1]<stdout>:Downloading https://sagemaker-example-files-prod-us-west-2.s3.amazonaws.com/datasets/image/MNIST/train-labels-idx1-ubyte.gz to /tmp/data/MNIST/raw/train-labels-idx1-ubyte.gz
[1,mpirank:8,algo-2]<stdout>:Downloading https://sagemaker-example-files-prod-us-west-2.s3.amazonaws.com/datasets/image/MNIST/train-labels-idx1-ubyte.gz to /tmp/data/MNIST/raw/train-labels-idx1-ubyte.gz
[1,mpirank:8,algo-2]<stderr>:#015  0%|          | 0/28881 [00:00<?, ?it/s]
[1,mpirank:8,algo-2]<stderr>:#015100%|██████████| 28881/28881 [00:00<00:00, 25363419.98it/s]
[1,mpirank:8,algo-2]<stdout>:Extracting /tmp/data/MNIST/raw/train-labels-idx1-ubyte.gz to /tmp/data/MNIST/raw
[1,mpirank:8,algo-2]<stdout>:
[1,mpirank:8,algo-2]<stdout>:Downloading https://sagemaker-example-files-prod-us-west-2.s3.amazonaws.com/datasets/image/MNIST/t10k-images-idx3-ubyte.gz
[1,mpirank:0,algo-1]<stderr>:#015  0%|          | 0/28881 [00:00<?, ?it/s]
[1,mpirank:0,algo-1]<stderr>:#015100%|██████████| 28881/28881 [00:00<00:00, 14992041.31it/s]
[1,mpirank:0,algo-1]<stdout>:Extracting /tmp/data/MNIST/raw/train-labels-idx1-ubyte.gz to /tmp/data/MNIST/raw
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Downloading https://sagemaker-example-files-prod-us-west-2.s3.amazonaws.com/datasets/image/MNIST/t10k-images-idx3-ubyte.gz
[1,mpirank:0,algo-1]<stdout>:Downloading https://sagemaker-example-files-prod-us-west-2.s3.amazonaws.com/datasets/image/MNIST/t10k-images-idx3-ubyte.gz to /tmp/data/MNIST/raw/t10k-images-idx3-ubyte.gz
[1,mpirank:8,algo-2]<stdout>:Downloading https://sagemaker-example-files-prod-us-west-2.s3.amazonaws.com/datasets/image/MNIST/t10k-images-idx3-ubyte.gz to /tmp/data/MNIST/raw/t10k-images-idx3-ubyte.gz
[1,mpirank:0,algo-1]<stderr>:#015  0%|          | 0/1648877 [00:00<?, ?it/s]
[1,mpirank:8,algo-2]<stderr>:#015  0%|          | 0/1648877 [00:00<?, ?it/s]
[1,mpirank:0,algo-1]<stderr>:#015100%|██████████| 1648877/1648877 [00:00<00:00, 179572907.76it/s]
[1,mpirank:8,algo-2]<stderr>:#015100%|██████████| 1648877/1648877 [00:00<00:00, 217995000.68it/s]
[1,mpirank:0,algo-1]<stdout>:Extracting /tmp/data/MNIST/raw/t10k-images-idx3-ubyte.gz to /tmp/data/MNIST/raw
[1,mpirank:8,algo-2]<stdout>:Extracting /tmp/data/MNIST/raw/t10k-images-idx3-ubyte.gz to /tmp/data/MNIST/raw
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Downloading https://sagemaker-example-files-prod-us-west-2.s3.amazonaws.com/datasets/image/MNIST/t10k-labels-idx1-ubyte.gz
[1,mpirank:8,algo-2]<stdout>:
[1,mpirank:8,algo-2]<stdout>:Downloading https://sagemaker-example-files-prod-us-west-2.s3.amazonaws.com/datasets/image/MNIST/t10k-labels-idx1-ubyte.gz
[1,mpirank:8,algo-2]<stdout>:Downloading https://sagemaker-example-files-prod-us-west-2.s3.amazonaws.com/datasets/image/MNIST/t10k-labels-idx1-ubyte.gz to /tmp/data/MNIST/raw/t10k-labels-idx1-ubyte.gz
[1,mpirank:0,algo-1]<stdout>:Downloading https://sagemaker-example-files-prod-us-west-2.s3.amazonaws.com/datasets/image/MNIST/t10k-labels-idx1-ubyte.gz to /tmp/data/MNIST/raw/t10k-labels-idx1-ubyte.gz
[1,mpirank:8,algo-2]<stderr>:#015  0%|          | 0/4542 [00:00<?, ?it/s]
[1,mpirank:8,algo-2]<stderr>:#015100%|██████████| 4542/4542 [00:00<00:00, 7559733.64it/s]
[1,mpirank:8,algo-2]<stdout>:Extracting /tmp/data/MNIST/raw/t10k-labels-idx1-ubyte.gz to /tmp/data/MNIST/raw
[1,mpirank:8,algo-2]<stdout>:
[1,mpirank:0,algo-1]<stderr>:#015  0%|          | 0/4542 [00:00<?, ?it/s]
[1,mpirank:0,algo-1]<stderr>:#015100%|██████████| 4542/4542 [00:00<00:00, 8542838.01it/s]
[1,mpirank:0,algo-1]<stdout>:Extracting /tmp/data/MNIST/raw/t10k-labels-idx1-ubyte.gz to /tmp/data/MNIST/raw
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Bootstrap : Using eth0:10.0.155.21<0>
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO cudaDriverVersion 12020
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO cudaDriverVersion 12020
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO cudaDriverVersion 12020
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO cudaDriverVersion 12020
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO cudaDriverVersion 12020
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO cudaDriverVersion 12020
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO cudaDriverVersion 12020
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Bootstrap : Using eth0:10.0.155.21<0>
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Bootstrap : Using eth0:10.0.155.21<0>
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Bootstrap : Using eth0:10.0.155.21<0>
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Bootstrap : Using eth0:10.0.155.21<0>
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Bootstrap : Using eth0:10.0.155.21<0>
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Bootstrap : Using eth0:10.0.155.21<0>
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Bootstrap : Using eth0:10.0.155.21<0>
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO cudaDriverVersion 12020
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO cudaDriverVersion 12020
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO cudaDriverVersion 12020
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO cudaDriverVersion 12020
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Bootstrap : Using eth0:10.0.170.27<0>
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Bootstrap : Using eth0:10.0.170.27<0>
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Bootstrap : Using eth0:10.0.170.27<0>
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Bootstrap : Using eth0:10.0.170.27<0>
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO cudaDriverVersion 12020
[1,mpirank:0,algo-1]<stdout>:NCCL version 2.17.1+cuda11.8
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO cudaDriverVersion 12020
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO cudaDriverVersion 12020
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO cudaDriverVersion 12020
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO cudaDriverVersion 12020
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Bootstrap : Using eth0:10.0.170.27<0>
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Bootstrap : Using eth0:10.0.170.27<0>
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Bootstrap : Using eth0:10.0.170.27<0>
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Bootstrap : Using eth0:10.0.170.27<0>
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.7.3-aws
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO NET/OFI Using CUDA runtime version 11080
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO NET/OFI Configuring AWS-specific options
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1
[1,mpirank:3,algo-1]<stdout>:
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] configure_nvls_option:287 NCCL WARN NET/OFI Could not find ncclGetVersion symbol
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO NET/OFI Disabling NVLS support due to NCCL version 0
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO NET/OFI Internode latency set at 75.0 us
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.7.3-aws
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO NET/OFI Using CUDA runtime version 11080
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO NET/OFI Configuring AWS-specific options
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.7.3-aws
[1,mpirank:1,algo-1]<stdout>:
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] configure_nvls_option:287 NCCL WARN NET/OFI Could not find ncclGetVersion symbol
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO NET/OFI Disabling NVLS support due to NCCL version 0
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO NET/OFI Internode latency set at 75.0 us
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO NET/OFI Using CUDA runtime version 11080
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO NET/OFI Configuring AWS-specific options
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1
[1,mpirank:2,algo-1]<stdout>:
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] configure_nvls_option:287 NCCL WARN NET/OFI Could not find ncclGetVersion symbol
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO NET/OFI Disabling NVLS support due to NCCL version 0
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO NET/OFI Internode latency set at 75.0 us
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.7.3-aws
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.7.3-aws
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO NET/OFI Using CUDA runtime version 11080
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO NET/OFI Configuring AWS-specific options
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO NET/OFI Using CUDA runtime version 11080
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO NET/OFI Configuring AWS-specific options
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1
[1,mpirank:6,algo-1]<stdout>:
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] configure_nvls_option:287 NCCL WARN NET/OFI Could not find ncclGetVersion symbol
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO NET/OFI Disabling NVLS support due to NCCL version 0
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:4,algo-1]<stdout>:
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] configure_nvls_option:287 NCCL WARN NET/OFI Could not find ncclGetVersion symbol
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO NET/OFI Disabling NVLS support due to NCCL version 0
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO NET/OFI Internode latency set at 75.0 us
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO NET/OFI Internode latency set at 75.0 us
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.7.3-aws
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO NET/OFI Using CUDA runtime version 11080
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO NET/OFI Configuring AWS-specific options
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] configure_nvls_option:287 NCCL WARN NET/OFI Could not find ncclGetVersion symbol
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO NET/OFI Disabling NVLS support due to NCCL version 0
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO NET/OFI Internode latency set at 75.0 us
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.7.3-aws
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.7.3-aws
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO NET/OFI Using CUDA runtime version 11080
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO NET/OFI Configuring AWS-specific options
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1
[1,mpirank:13,algo-2]<stdout>:
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] configure_nvls_option:287 NCCL WARN NET/OFI Could not find ncclGetVersion symbol
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO NET/OFI Disabling NVLS support due to NCCL version 0
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO NET/OFI Internode latency set at 75.0 us
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO NET/OFI Using CUDA runtime version 11080
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO NET/OFI Configuring AWS-specific options
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1
[1,mpirank:9,algo-2]<stdout>:
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] configure_nvls_option:287 NCCL WARN NET/OFI Could not find ncclGetVersion symbol
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO NET/OFI Disabling NVLS support due to NCCL version 0
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO NET/OFI Internode latency set at 75.0 us
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.7.3-aws
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO NET/OFI Using CUDA runtime version 11080
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO NET/OFI Configuring AWS-specific options
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1
[1,mpirank:8,algo-2]<stdout>:
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] configure_nvls_option:287 NCCL WARN NET/OFI Could not find ncclGetVersion symbol
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO NET/OFI Disabling NVLS support due to NCCL version 0
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO NET/OFI Internode latency set at 75.0 us
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.7.3-aws
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO NET/OFI Using CUDA runtime version 11080
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO NET/OFI Configuring AWS-specific options
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1
[1,mpirank:12,algo-2]<stdout>:
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] configure_nvls_option:287 NCCL WARN NET/OFI Could not find ncclGetVersion symbol
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO NET/OFI Disabling NVLS support due to NCCL version 0
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO NET/OFI Internode latency set at 75.0 us
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.7.3-aws
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO NET/OFI Using CUDA runtime version 11080
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO NET/OFI Configuring AWS-specific options
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1
[1,mpirank:15,algo-2]<stdout>:
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] configure_nvls_option:287 NCCL WARN NET/OFI Could not find ncclGetVersion symbol
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO NET/OFI Disabling NVLS support due to NCCL version 0
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO NET/OFI Internode latency set at 75.0 us
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.7.3-aws
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO NET/OFI Using CUDA runtime version 11080
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO NET/OFI Configuring AWS-specific options
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1
[1,mpirank:11,algo-2]<stdout>:
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] configure_nvls_option:287 NCCL WARN NET/OFI Could not find ncclGetVersion symbol
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO NET/OFI Disabling NVLS support due to NCCL version 0
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO NET/OFI Internode latency set at 75.0 us
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO NET/OFI Selected Provider is efa (found 4 nics)
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO NET/OFI Using transport protocol SENDRECV
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO NET/OFI Selected Provider is efa (found 4 nics)
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO NET/OFI Using transport protocol SENDRECV
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO NET/OFI Selected Provider is efa (found 4 nics)
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO NET/OFI Using transport protocol SENDRECV
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO NET/OFI Selected Provider is efa (found 4 nics)
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO NET/OFI Using transport protocol SENDRECV
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO NET/OFI Selected Provider is efa (found 4 nics)
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO NET/OFI Using transport protocol SENDRECV
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO NET/OFI Selected Provider is efa (found 4 nics)
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO NET/OFI Using transport protocol SENDRECV
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.7.3-aws
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO NET/OFI Using CUDA runtime version 11080
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO NET/OFI Configuring AWS-specific options
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1
[1,mpirank:5,algo-1]<stdout>:
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] configure_nvls_option:287 NCCL WARN NET/OFI Could not find ncclGetVersion symbol
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO NET/OFI Disabling NVLS support due to NCCL version 0
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO NET/OFI Internode latency set at 75.0 us
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.7.3-aws
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO NET/OFI Using CUDA runtime version 11080
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO NET/OFI Configuring AWS-specific options
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1
[1,mpirank:7,algo-1]<stdout>:
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] configure_nvls_option:287 NCCL WARN NET/OFI Could not find ncclGetVersion symbol
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO NET/OFI Disabling NVLS support due to NCCL version 0
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO NET/OFI Internode latency set at 75.0 us
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.7.3-aws
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO NET/OFI Using CUDA runtime version 11080
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO NET/OFI Configuring AWS-specific options
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1
[1,mpirank:14,algo-2]<stdout>:
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] configure_nvls_option:287 NCCL WARN NET/OFI Could not find ncclGetVersion symbol
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO NET/OFI Disabling NVLS support due to NCCL version 0
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO NET/OFI Internode latency set at 75.0 us
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO NET/OFI Selected Provider is efa (found 4 nics)
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO NET/OFI Using transport protocol SENDRECV
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO NET/OFI Selected Provider is efa (found 4 nics)
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO NET/OFI Using transport protocol SENDRECV
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO NET/OFI Selected Provider is efa (found 4 nics)
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO NET/OFI Using transport protocol SENDRECV
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO NET/OFI Selected Provider is efa (found 4 nics)
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO NET/OFI Using transport protocol SENDRECV
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO NET/OFI Selected Provider is efa (found 4 nics)
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO NET/OFI Using transport protocol SENDRECV
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO NET/OFI Selected Provider is efa (found 4 nics)
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO NET/OFI Using transport protocol SENDRECV
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO NET/OFI Selected Provider is efa (found 4 nics)
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO NET/OFI Using transport protocol SENDRECV
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO NET/OFI Selected Provider is efa (found 4 nics)
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO NET/OFI Using transport protocol SENDRECV
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO NET/OFI Selected Provider is efa (found 4 nics)
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO NET/OFI Using transport protocol SENDRECV
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO NET/OFI Setting FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES not supported.
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO NET/OFI Setting FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES not supported.
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO NET/OFI Setting FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES not supported.
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO NET/OFI Setting FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES not supported.
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO NET/OFI Setting FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES not supported.
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO NET/OFI Setting FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES not supported.
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO NET/OFI Setting FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES not supported.
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO NET/OFI Setting FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES not supported.
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO NET/OFI Setting FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES not supported.
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO NET/OFI Setting FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES not supported.
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Using network AWS Libfabric
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO DMA-BUF is available on GPU device 3
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO NET/OFI Setting FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES not supported.
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO NET/OFI Setting FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES not supported.
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO NET/OFI Setting FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES not supported.
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO NET/OFI Setting FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES not supported.
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Using network AWS Libfabric
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO DMA-BUF is available on GPU device 1
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Using network AWS Libfabric
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO DMA-BUF is available on GPU device 2
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Using network AWS Libfabric
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO NET/OFI Setting FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES not supported.
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Using network AWS Libfabric
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO DMA-BUF is available on GPU device 4
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Using network AWS Libfabric
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO DMA-BUF is available on GPU device 6
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO DMA-BUF is available on GPU device 0
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Using network AWS Libfabric
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO DMA-BUF is available on GPU device 5
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Using network AWS Libfabric
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO DMA-BUF is available on GPU device 7
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Using network AWS Libfabric
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO DMA-BUF is available on GPU device 7
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Using network AWS Libfabric
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO DMA-BUF is available on GPU device 4
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Using network AWS Libfabric
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO DMA-BUF is available on GPU device 1
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Using network AWS Libfabric
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Using network AWS Libfabric
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Using network AWS Libfabric
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO DMA-BUF is available on GPU device 6
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO DMA-BUF is available on GPU device 0
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO DMA-BUF is available on GPU device 3
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Using network AWS Libfabric
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO DMA-BUF is available on GPU device 5
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO NET/OFI Initializing aws-ofi-nccl 1.7.3-aws
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO NET/OFI Using CUDA runtime version 11080
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO NET/OFI Configuring AWS-specific options
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1
[1,mpirank:10,algo-2]<stdout>:
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] configure_nvls_option:287 NCCL WARN NET/OFI Could not find ncclGetVersion symbol
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO NET/OFI Disabling NVLS support due to NCCL version 0
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO NET/OFI Running on p4d.24xlarge platform, Setting NCCL_TOPO_FILE environment variable to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO NET/OFI Internode latency set at 75.0 us
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO NET/OFI Selected Provider is efa (found 4 nics)
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO NET/OFI Using transport protocol SENDRECV
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO NET/OFI Setting FI_OPT_EFA_SENDRECV_IN_ORDER_ALIGNED_128_BYTES not supported.
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Using network AWS Libfabric
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO DMA-BUF is available on GPU device 2
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Setting affinity for GPU 7 to fc000000,0000fc00,00000000
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Setting affinity for GPU 6 to 03f00000,000003f0,00000000
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Setting affinity for GPU 0 to 3f0000,0000003f
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Setting affinity for GPU 1 to 0fc00000,00000fc0
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Setting affinity for GPU 5 to 0fc000,0000000f,c0000000
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Setting affinity for GPU 2 to 03,f0000000,0003f000
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Setting affinity for GPU 3 to fc,00000000,00fc0000
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Setting affinity for GPU 4 to 3f00,00000000,3f000000
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Setting affinity for GPU 5 to 0fc000,0000000f,c0000000
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Setting affinity for GPU 7 to fc000000,0000fc00,00000000
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Setting affinity for GPU 4 to 3f00,00000000,3f000000
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Setting affinity for GPU 2 to 03,f0000000,0003f000
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Setting affinity for GPU 6 to 03f00000,000003f0,00000000
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Setting affinity for GPU 3 to fc,00000000,00fc0000
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Setting affinity for GPU 1 to 0fc00000,00000fc0
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Setting affinity for GPU 0 to 3f0000,0000003f
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] 0/-1/-1->7->6 [2] 0/-1/-1->7->6 [3] 0/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] 0/-1/-1->7->6 [6] 0/-1/-1->7->6 [7] 0/-1/-1->7->6
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO P2P Chunksize set to 131072
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/14/-1->6->-1 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->14
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO P2P Chunksize set to 131072
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] -1/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] -1/-1/-1->5->4
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO P2P Chunksize set to 131072
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/12/-1->4->-1 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->12 [7] 5/-1/-1->4->3
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO P2P Chunksize set to 131072
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] -1/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] -1/-1/-1->3->2 [7] 4/-1/-1->3->2
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO P2P Chunksize set to 131072
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO P2P Chunksize set to 131072
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 00/08 :    0   7   6   5   4   3   2   1   8  15  14  13  12  11  10   9
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 01/08 :    0   3  10  15  14  13  12   9   8  11   2   7   6   5   4   1
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 02/08 :    0   7   6   5  12  11  10   9   8  15  14  13   4   3   2   1
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 03/08 :    0   5   4   7  14  11  10   9   8  13  12  15   6   3   2   1
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 04/08 :    0   7   6   5   4   3   2   1   8  15  14  13  12  11  10   9
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 05/08 :    0   3  10  15  14  13  12   9   8  11   2   7   6   5   4   1
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 06/08 :    0   7   6   5  12  11  10   9   8  15  14  13   4   3   2   1
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/10/-1->2->-1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->10 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO P2P Chunksize set to 131072
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Trees [0] 10/-1/-1->9->8 [1] -1/-1/-1->9->8 [2] 10/-1/-1->9->8 [3] 10/-1/-1->9->8 [4] 10/-1/-1->9->8 [5] -1/-1/-1->9->8 [6] 10/-1/-1->9->8 [7] 10/-1/-1->9->8
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO P2P Chunksize set to 131072
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 07/08 :    0   5   4   7  14  11  10   9   8  13  12  15   6   3   2   1
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Trees [0] 1/8/-1->0->-1 [1] 1/-1/-1->0->7 [2] 1/-1/-1->0->7 [3] 1/-1/-1->0->7 [4] 1/-1/-1->0->8 [5] 1/-1/-1->0->7 [6] 1/-1/-1->0->7 [7] 1/-1/-1->0->7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO P2P Chunksize set to 131072
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Trees [0] 9/-1/-1->8->0 [1] 9/-1/-1->8->15 [2] 9/-1/-1->8->15 [3] 9/-1/-1->8->15 [4] 9/0/-1->8->-1 [5] 9/-1/-1->8->15 [6] 9/-1/-1->8->15 [7] 9/-1/-1->8->15
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO P2P Chunksize set to 131072
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Trees [0] 11/-1/-1->10->9 [1] 11/-1/-1->10->2 [2] 11/-1/-1->10->9 [3] 11/-1/-1->10->9 [4] 11/-1/-1->10->9 [5] 11/2/-1->10->-1 [6] 11/-1/-1->10->9 [7] 11/-1/-1->10->9
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO P2P Chunksize set to 131072
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Trees [0] 15/-1/-1->14->13 [1] 15/-1/-1->14->13 [2] 15/-1/-1->14->13 [3] 15/-1/-1->14->6 [4] 15/-1/-1->14->13 [5] 15/-1/-1->14->13 [6] 15/-1/-1->14->13 [7] 15/6/-1->14->-1
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO P2P Chunksize set to 131072
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Trees [0] 12/-1/-1->11->10 [1] 12/-1/-1->11->10 [2] -1/-1/-1->11->10 [3] 12/-1/-1->11->10 [4] 12/-1/-1->11->10 [5] 12/-1/-1->11->10 [6] -1/-1/-1->11->10 [7] 12/-1/-1->11->10
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO P2P Chunksize set to 131072
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Trees [0] 14/-1/-1->13->12 [1] 14/-1/-1->13->12 [2] 14/-1/-1->13->12 [3] -1/-1/-1->13->12 [4] 14/-1/-1->13->12 [5] 14/-1/-1->13->12 [6] 14/-1/-1->13->12 [7] -1/-1/-1->13->12
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO P2P Chunksize set to 131072
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Trees [0] 13/-1/-1->12->11 [1] 13/-1/-1->12->11 [2] 13/-1/-1->12->4 [3] 13/-1/-1->12->11 [4] 13/-1/-1->12->11 [5] 13/-1/-1->12->11 [6] 13/4/-1->12->-1 [7] 13/-1/-1->12->11
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO P2P Chunksize set to 131072
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Trees [0] -1/-1/-1->15->14 [1] 8/-1/-1->15->14 [2] 8/-1/-1->15->14 [3] 8/-1/-1->15->14 [4] -1/-1/-1->15->14 [5] 8/-1/-1->15->14 [6] 8/-1/-1->15->14 [7] 8/-1/-1->15->14
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO P2P Chunksize set to 131072
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 01/0 : 8[101c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 03/0 : 4[901c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 01/0 : 10[201c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 01/0 : 2[201c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 03/0 : 12[901c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 05/0 : 10[201c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 05/0 : 2[201c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 05/0 : 8[101c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 00/0 : 9[101d0] -> 0[101c0] [send] via NET/AWS Libfabric/0/GDRDMA
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 07/0 : 12[901c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 00/0 : 1[101d0] -> 8[101c0] [send] via NET/AWS Libfabric/0/GDRDMA
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 07/0 : 4[901c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 03/0 : 7[a01d0] -> 14[a01c0] [receive] via NET/AWS Libfabric/3/GDRDMA
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 03/0 : 15[a01d0] -> 6[a01c0] [receive] via NET/AWS Libfabric/3/GDRDMA
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 03/0 : 8[101c0] -> 13[901d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 04/0 : 9[101d0] -> 0[101c0] [send] via NET/AWS Libfabric/0/GDRDMA
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 04/0 : 1[101d0] -> 8[101c0] [send] via NET/AWS Libfabric/0/GDRDMA
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 07/0 : 8[101c0] -> 13[901d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 07/0 : 7[a01d0] -> 14[a01c0] [receive] via NET/AWS Libfabric/3/GDRDMA
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 07/0 : 15[a01d0] -> 6[a01c0] [receive] via NET/AWS Libfabric/3/GDRDMA
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 01/0 : 3[201d0] -> 10[201c0] [send] via NET/AWS Libfabric/1/GDRDMA
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 01/0 : 11[201d0] -> 2[201c0] [send] via NET/AWS Libfabric/1/GDRDMA
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 02/0 : 13[901d0] -> 4[901c0] [receive] via NET/AWS Libfabric/2/GDRDMA
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 02/0 : 5[901d0] -> 12[901c0] [receive] via NET/AWS Libfabric/2/GDRDMA
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 02/0 : 5[901d0] -> 12[901c0] [send] via NET/AWS Libfabric/2/GDRDMA
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 02/0 : 13[901d0] -> 4[901c0] [send] via NET/AWS Libfabric/2/GDRDMA
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 01/0 : 11[201d0] -> 2[201c0] [receive] via NET/AWS Libfabric/1/GDRDMA
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 01/0 : 3[201d0] -> 10[201c0] [receive] via NET/AWS Libfabric/1/GDRDMA
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 05/0 : 3[201d0] -> 10[201c0] [send] via NET/AWS Libfabric/1/GDRDMA
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 05/0 : 11[201d0] -> 2[201c0] [send] via NET/AWS Libfabric/1/GDRDMA
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 06/0 : 13[901d0] -> 4[901c0] [receive] via NET/AWS Libfabric/2/GDRDMA
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 06/0 : 5[901d0] -> 12[901c0] [receive] via NET/AWS Libfabric/2/GDRDMA
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 00/0 : 9[101d0] -> 0[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 00/0 : 1[101d0] -> 8[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 06/0 : 5[901d0] -> 12[901c0] [send] via NET/AWS Libfabric/2/GDRDMA
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 01/0 : 12[901c0] -> 9[101d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 06/0 : 13[901d0] -> 4[901c0] [send] via NET/AWS Libfabric/2/GDRDMA
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 01/0 : 4[901c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 05/0 : 11[201d0] -> 2[201c0] [receive] via NET/AWS Libfabric/1/GDRDMA
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 05/0 : 3[201d0] -> 10[201c0] [receive] via NET/AWS Libfabric/1/GDRDMA
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 05/0 : 12[901c0] -> 9[101d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 05/0 : 4[901c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 04/0 : 9[101d0] -> 0[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 04/0 : 1[101d0] -> 8[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 00/0 : 8[101c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 02/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 02/0 : 8[101c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 03/0 : 7[a01d0] -> 14[a01c0] [send] via NET/AWS Libfabric/3/GDRDMA
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 03/0 : 15[a01d0] -> 6[a01c0] [send] via NET/AWS Libfabric/3/GDRDMA
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 04/0 : 8[101c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 06/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 07/0 : 15[a01d0] -> 6[a01c0] [send] via NET/AWS Libfabric/3/GDRDMA
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 03/0 : 6[a01c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 06/0 : 8[101c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 07/0 : 7[a01d0] -> 14[a01c0] [send] via NET/AWS Libfabric/3/GDRDMA
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 03/0 : 14[a01c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 00/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 07/0 : 6[a01c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 00/0 : 13[901d0] -> 12[901c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 07/0 : 14[a01c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 00/0 : 15[a01d0] -> 14[a01c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 00/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 00/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 01/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 00/0 : 10[201c0] -> 9[101d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 01/0 : 13[901d0] -> 12[901c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 01/0 : 15[a01d0] -> 14[a01c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 01/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 02/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 02/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 03/0 : 13[901d0] -> 12[901c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 02/0 : 10[201c0] -> 9[101d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 02/0 : 15[a01d0] -> 14[a01c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 03/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 03/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 04/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 03/0 : 10[201c0] -> 9[101d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 04/0 : 13[901d0] -> 12[901c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 04/0 : 15[a01d0] -> 14[a01c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 04/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 05/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 04/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 05/0 : 13[901d0] -> 12[901c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 04/0 : 10[201c0] -> 9[101d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 05/0 : 15[a01d0] -> 14[a01c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 05/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 06/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 06/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 00/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 06/0 : 10[201c0] -> 9[101d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 07/0 : 13[901d0] -> 12[901c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 00/0 : 11[201d0] -> 10[201c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 06/0 : 15[a01d0] -> 14[a01c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 07/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 07/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 02/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 02/0 : 11[201d0] -> 10[201c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 07/0 : 10[201c0] -> 9[101d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 00/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 00/0 : 14[a01c0] -> 13[901d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 03/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 03/0 : 11[201d0] -> 10[201c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 00/0 : 12[901c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 01/0 : 9[101d0] -> 8[101c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 01/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 00/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 01/0 : 14[a01c0] -> 13[901d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 04/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 01/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 04/0 : 11[201d0] -> 10[201c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 02/0 : 12[901c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 02/0 : 9[101d0] -> 8[101c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 02/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 02/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 02/0 : 14[a01c0] -> 13[901d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 06/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 02/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 06/0 : 11[201d0] -> 10[201c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 04/0 : 12[901c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 03/0 : 9[101d0] -> 8[101c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 04/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 04/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 04/0 : 14[a01c0] -> 13[901d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 07/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 03/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 07/0 : 11[201d0] -> 10[201c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 06/0 : 12[901c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 05/0 : 9[101d0] -> 8[101c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 06/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 05/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 05/0 : 14[a01c0] -> 13[901d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 05/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 06/0 : 9[101d0] -> 8[101c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 06/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 06/0 : 14[a01c0] -> 13[901d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 06/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 07/0 : 9[101d0] -> 8[101c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 07/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Connected all rings
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Connected all rings
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Connected all rings
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Connected all rings
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Connected all rings
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Connected all rings
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Connected all rings
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Connected all rings
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Connected all rings
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Connected all rings
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Connected all rings
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Connected all rings
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Connected all rings
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Connected all rings
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Connected all rings
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Connected all rings
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 00/0 : 8[101c0] -> 9[101d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 01/0 : 8[101c0] -> 9[101d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 02/0 : 8[101c0] -> 9[101d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 02/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 03/0 : 8[101c0] -> 9[101d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 04/0 : 8[101c0] -> 9[101d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 00/0 : 12[901c0] -> 13[901d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 05/0 : 8[101c0] -> 9[101d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 00/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 00/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 00/0 : 10[201c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 01/0 : 12[901c0] -> 13[901d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 00/0 : 14[a01c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 00/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 06/0 : 8[101c0] -> 9[101d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 06/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 01/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 01/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 01/0 : 10[201c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 02/0 : 12[901c0] -> 13[901d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 01/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 00/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 00/0 : 13[901d0] -> 14[a01c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 01/0 : 14[a01c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 07/0 : 8[101c0] -> 9[101d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 02/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 00/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 02/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 02/0 : 10[201c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 00/0 : 11[201d0] -> 12[901c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 03/0 : 12[901c0] -> 13[901d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 02/0 : 14[a01c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 01/0 : 13[901d0] -> 14[a01c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 02/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 01/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 00/0 : 9[101d0] -> 10[201c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 00/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 03/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 03/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 01/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 01/0 : 11[201d0] -> 12[901c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 03/0 : 10[201c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 04/0 : 12[901c0] -> 13[901d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 02/0 : 13[901d0] -> 14[a01c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 02/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 03/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 03/0 : 14[a01c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 02/0 : 9[101d0] -> 10[201c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 03/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 04/0 : 10[201c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 02/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 04/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 04/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 03/0 : 11[201d0] -> 12[901c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 05/0 : 12[901c0] -> 13[901d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 04/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 04/0 : 13[901d0] -> 14[a01c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 04/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 04/0 : 14[a01c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 03/0 : 9[101d0] -> 10[201c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 04/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 05/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 05/0 : 10[201c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 03/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 05/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 04/0 : 11[201d0] -> 12[901c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 06/0 : 12[901c0] -> 13[901d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 05/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 05/0 : 13[901d0] -> 14[a01c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 05/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 05/0 : 14[a01c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 04/0 : 9[101d0] -> 10[201c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 05/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 06/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 06/0 : 10[201c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 06/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 04/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 07/0 : 12[901c0] -> 13[901d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 05/0 : 11[201d0] -> 12[901c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 06/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 06/0 : 14[a01c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 06/0 : 13[901d0] -> 14[a01c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 06/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 06/0 : 9[101d0] -> 10[201c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 07/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 07/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 07/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 07/0 : 10[201c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 06/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 07/0 : 11[201d0] -> 12[901c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 07/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 07/0 : 14[a01c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 07/0 : 9[101d0] -> 10[201c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 02/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 07/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 02/0 : 13[901d0] -> 12[901c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 00/0 : 9[101d0] -> 8[101c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 01/0 : 8[101c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 06/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 00/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 02/0 : 4[901c0] -> 12[901c0] [receive] via NET/AWS Libfabric/2/GDRDMA
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 04/0 : 9[101d0] -> 8[101c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 06/0 : 13[901d0] -> 12[901c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 03/0 : 8[101c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 03/0 : 14[a01c0] -> 6[a01c0] [receive] via NET/AWS Libfabric/3/GDRDMA
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 02/0 : 12[901c0] -> 4[901c0] [receive] via NET/AWS Libfabric/2/GDRDMA
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 04/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 03/0 : 6[a01c0] -> 14[a01c0] [receive] via NET/AWS Libfabric/3/GDRDMA
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 01/0 : 2[201c0] -> 10[201c0] [receive] via NET/AWS Libfabric/1/GDRDMA
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 05/0 : 8[101c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 01/0 : 10[201c0] -> 2[201c0] [receive] via NET/AWS Libfabric/1/GDRDMA
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 06/0 : 4[901c0] -> 12[901c0] [receive] via NET/AWS Libfabric/2/GDRDMA
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 07/0 : 8[101c0] -> 15[a01d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 06/0 : 12[901c0] -> 4[901c0] [receive] via NET/AWS Libfabric/2/GDRDMA
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 07/0 : 14[a01c0] -> 6[a01c0] [receive] via NET/AWS Libfabric/3/GDRDMA
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 01/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 01/0 : 11[201d0] -> 10[201c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 05/0 : 2[201c0] -> 10[201c0] [receive] via NET/AWS Libfabric/1/GDRDMA
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 01/0 : 15[a01d0] -> 8[101c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 07/0 : 6[a01c0] -> 14[a01c0] [receive] via NET/AWS Libfabric/3/GDRDMA
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 05/0 : 10[201c0] -> 2[201c0] [receive] via NET/AWS Libfabric/1/GDRDMA
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 05/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 01/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 02/0 : 12[901c0] -> 4[901c0] [send] via NET/AWS Libfabric/2/GDRDMA
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 05/0 : 11[201d0] -> 10[201c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 02/0 : 15[a01d0] -> 8[101c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 03/0 : 6[a01c0] -> 14[a01c0] [send] via NET/AWS Libfabric/3/GDRDMA
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 02/0 : 4[901c0] -> 12[901c0] [send] via NET/AWS Libfabric/2/GDRDMA
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 02/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 03/0 : 15[a01d0] -> 8[101c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 03/0 : 14[a01c0] -> 6[a01c0] [send] via NET/AWS Libfabric/3/GDRDMA
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 01/0 : 10[201c0] -> 2[201c0] [send] via NET/AWS Libfabric/1/GDRDMA
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 8[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 01/0 : 2[201c0] -> 10[201c0] [send] via NET/AWS Libfabric/1/GDRDMA
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 00/0 : 8[101c0] -> 0[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 03/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 06/0 : 12[901c0] -> 4[901c0] [send] via NET/AWS Libfabric/2/GDRDMA
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 05/0 : 15[a01d0] -> 8[101c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 06/0 : 4[901c0] -> 12[901c0] [send] via NET/AWS Libfabric/2/GDRDMA
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 07/0 : 6[a01c0] -> 14[a01c0] [send] via NET/AWS Libfabric/3/GDRDMA
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 05/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 07/0 : 14[a01c0] -> 6[a01c0] [send] via NET/AWS Libfabric/3/GDRDMA
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 06/0 : 15[a01d0] -> 8[101c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 8[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 05/0 : 10[201c0] -> 2[201c0] [send] via NET/AWS Libfabric/1/GDRDMA
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 04/0 : 8[101c0] -> 0[101c0] [receive] via NET/AWS Libfabric/0/GDRDMA
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 05/0 : 2[201c0] -> 10[201c0] [send] via NET/AWS Libfabric/1/GDRDMA
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 06/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 07/0 : 15[a01d0] -> 8[101c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 01/0 : 12[901c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 01/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 07/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 00/0 : 8[101c0] -> 0[101c0] [send] via NET/AWS Libfabric/0/GDRDMA
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 03/0 : 12[901c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 8[101c0] [send] via NET/AWS Libfabric/0/GDRDMA
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 03/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Connected all trees
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO 8 coll channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 05/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 05/0 : 12[901c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Connected all trees
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO 8 coll channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 04/0 : 8[101c0] -> 0[101c0] [send] via NET/AWS Libfabric/0/GDRDMA
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 8[101c0] [send] via NET/AWS Libfabric/0/GDRDMA
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 07/0 : 12[901c0] -> 11[201d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 07/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Connected all trees
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO 8 coll channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Connected all trees
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO 8 coll channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Connected all trees
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO 8 coll channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Connected all trees
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO 8 coll channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Connected all trees
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO 8 coll channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Connected all trees
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO 8 coll channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 03/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 03/0 : 15[a01d0] -> 14[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 07/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 07/0 : 15[a01d0] -> 14[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Connected all trees
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO 8 coll channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Connected all trees
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO 8 coll channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Connected all trees
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO 8 coll channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Connected all trees
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO 8 coll channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Connected all trees
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO 8 coll channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Connected all trees
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Connected all trees
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO 8 coll channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO 8 coll channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Connected all trees
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO 8 coll channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO comm 0x7f73840051e0 rank 7 nranks 16 cudaDev 7 busId a01d0 commId 0xfe10a3dce91ee731 - Init COMPLETE
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO comm 0x7fb3640051e0 rank 3 nranks 16 cudaDev 3 busId 201d0 commId 0xfe10a3dce91ee731 - Init COMPLETE
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO comm 0x7fabcc00ea60 rank 0 nranks 16 cudaDev 0 busId 101c0 commId 0xfe10a3dce91ee731 - Init COMPLETE
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO comm 0x7f0824005200 rank 4 nranks 16 cudaDev 4 busId 901c0 commId 0xfe10a3dce91ee731 - Init COMPLETE
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO comm 0x7efe44005130 rank 6 nranks 16 cudaDev 6 busId a01c0 commId 0xfe10a3dce91ee731 - Init COMPLETE
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO comm 0x7fdefc005130 rank 2 nranks 16 cudaDev 2 busId 201c0 commId 0xfe10a3dce91ee731 - Init COMPLETE
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO comm 0x7f860c005130 rank 5 nranks 16 cudaDev 5 busId 901d0 commId 0xfe10a3dce91ee731 - Init COMPLETE
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO comm 0x7fb47c005130 rank 1 nranks 16 cudaDev 1 busId 101d0 commId 0xfe10a3dce91ee731 - Init COMPLETE
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO comm 0x7f8ab80051e0 rank 15 nranks 16 cudaDev 7 busId a01d0 commId 0xfe10a3dce91ee731 - Init COMPLETE
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO comm 0x7fab18005130 rank 9 nranks 16 cudaDev 1 busId 101d0 commId 0xfe10a3dce91ee731 - Init COMPLETE
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO comm 0x7ff65c005130 rank 13 nranks 16 cudaDev 5 busId 901d0 commId 0xfe10a3dce91ee731 - Init COMPLETE
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO comm 0x7f758c00df60 rank 12 nranks 16 cudaDev 4 busId 901c0 commId 0xfe10a3dce91ee731 - Init COMPLETE
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO comm 0x7f3904005130 rank 10 nranks 16 cudaDev 2 busId 201c0 commId 0xfe10a3dce91ee731 - Init COMPLETE
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO comm 0x7ff198005320 rank 8 nranks 16 cudaDev 0 busId 101c0 commId 0xfe10a3dce91ee731 - Init COMPLETE
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO comm 0x7f936c0051e0 rank 11 nranks 16 cudaDev 3 busId 201d0 commId 0xfe10a3dce91ee731 - Init COMPLETE
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO comm 0x7f090c005130 rank 14 nranks 16 cudaDev 6 busId a01c0 commId 0xfe10a3dce91ee731 - Init COMPLETE
[1,mpirank:3,algo-1]<stderr>:/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:1331: UserWarning: dropout2d: Received a 2-D input to dropout2d, which is deprecated and will result in an error in a future release. To retain the behavior and silence this warning, please use dropout instead. Note that dropout2d exists to provide channel-wise dropout on inputs with 2 spatial dimensions, a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).
[1,mpirank:3,algo-1]<stderr>:  warnings.warn(warn_msg)
[1,mpirank:11,algo-2]<stderr>:/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:1331: UserWarning: dropout2d: Received a 2-D input to dropout2d, which is deprecated and will result in an error in a future release. To retain the behavior and silence this warning, please use dropout instead. Note that dropout2d exists to provide channel-wise dropout on inputs with 2 spatial dimensions, a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).
[1,mpirank:11,algo-2]<stderr>:  warnings.warn(warn_msg)
[1,mpirank:9,algo-2]<stderr>:/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:1331: UserWarning: dropout2d: Received a 2-D input to dropout2d, which is deprecated and will result in an error in a future release. To retain the behavior and silence this warning, please use dropout instead. Note that dropout2d exists to provide channel-wise dropout on inputs with 2 spatial dimensions, a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).
[1,mpirank:9,algo-2]<stderr>:  warnings.warn(warn_msg)
[1,mpirank:0,algo-1]<stderr>:/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:1331: UserWarning: dropout2d: Received a 2-D input to dropout2d, which is deprecated and will result in an error in a future release. To retain the behavior and silence this warning, please use dropout instead. Note that dropout2d exists to provide channel-wise dropout on inputs with 2 spatial dimensions, a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).
[1,mpirank:0,algo-1]<stderr>:  warnings.warn(warn_msg)
[1,mpirank:2,algo-1]<stderr>:/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:1331: UserWarning: dropout2d: Received a 2-D input to dropout2d, which is deprecated and will result in an error in a future release. To retain the behavior and silence this warning, please use dropout instead. Note that dropout2d exists to provide channel-wise dropout on inputs with 2 spatial dimensions, a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).
[1,mpirank:2,algo-1]<stderr>:  warnings.warn(warn_msg)
[1,mpirank:8,algo-2]<stderr>:/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:1331: UserWarning: dropout2d: Received a 2-D input to dropout2d, which is deprecated and will result in an error in a future release. To retain the behavior and silence this warning, please use dropout instead. Note that dropout2d exists to provide channel-wise dropout on inputs with 2 spatial dimensions, a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).
[1,mpirank:8,algo-2]<stderr>:  warnings.warn(warn_msg)
[1,mpirank:15,algo-2]<stderr>:/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:1331: UserWarning: dropout2d: Received a 2-D input to dropout2d, which is deprecated and will result in an error in a future release. To retain the behavior and silence this warning, please use dropout instead. Note that dropout2d exists to provide channel-wise dropout on inputs with 2 spatial dimensions, a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).
[1,mpirank:15,algo-2]<stderr>:  warnings.warn(warn_msg)
[1,mpirank:5,algo-1]<stderr>:/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:1331: UserWarning: dropout2d: Received a 2-D input to dropout2d, which is deprecated and will result in an error in a future release. To retain the behavior and silence this warning, please use dropout instead. Note that dropout2d exists to provide channel-wise dropout on inputs with 2 spatial dimensions, a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).
[1,mpirank:5,algo-1]<stderr>:  warnings.warn(warn_msg)
[1,mpirank:6,algo-1]<stderr>:/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:1331: UserWarning: dropout2d: Received a 2-D input to dropout2d, which is deprecated and will result in an error in a future release. To retain the behavior and silence this warning, please use dropout instead. Note that dropout2d exists to provide channel-wise dropout on inputs with 2 spatial dimensions, a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).
[1,mpirank:6,algo-1]<stderr>:  warnings.warn(warn_msg)
[1,mpirank:13,algo-2]<stderr>:/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:1331: UserWarning: dropout2d: Received a 2-D input to dropout2d, which is deprecated and will result in an error in a future release. To retain the behavior and silence this warning, please use dropout instead. Note that dropout2d exists to provide channel-wise dropout on inputs with 2 spatial dimensions, a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).
[1,mpirank:13,algo-2]<stderr>:  warnings.warn(warn_msg)
[1,mpirank:4,algo-1]<stderr>:/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:1331: UserWarning: dropout2d: Received a 2-D input to dropout2d, which is deprecated and will result in an error in a future release. To retain the behavior and silence this warning, please use dropout instead. Note that dropout2d exists to provide channel-wise dropout on inputs with 2 spatial dimensions, a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).
[1,mpirank:4,algo-1]<stderr>:  warnings.warn(warn_msg)
[1,mpirank:12,algo-2]<stderr>:/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:1331: UserWarning: dropout2d: Received a 2-D input to dropout2d, which is deprecated and will result in an error in a future release. To retain the behavior and silence this warning, please use dropout instead. Note that dropout2d exists to provide channel-wise dropout on inputs with 2 spatial dimensions, a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).
[1,mpirank:12,algo-2]<stderr>:  warnings.warn(warn_msg)
[1,mpirank:1,algo-1]<stderr>:/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:1331: UserWarning: dropout2d: Received a 2-D input to dropout2d, which is deprecated and will result in an error in a future release. To retain the behavior and silence this warning, please use dropout instead. Note that dropout2d exists to provide channel-wise dropout on inputs with 2 spatial dimensions, a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).
[1,mpirank:1,algo-1]<stderr>:  warnings.warn(warn_msg)
[1,mpirank:10,algo-2]<stderr>:/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:1331: UserWarning: dropout2d: Received a 2-D input to dropout2d, which is deprecated and will result in an error in a future release. To retain the behavior and silence this warning, please use dropout instead. Note that dropout2d exists to provide channel-wise dropout on inputs with 2 spatial dimensions, a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).
[1,mpirank:10,algo-2]<stderr>:  warnings.warn(warn_msg)
[1,mpirank:14,algo-2]<stderr>:/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:1331: UserWarning: dropout2d: Received a 2-D input to dropout2d, which is deprecated and will result in an error in a future release. To retain the behavior and silence this warning, please use dropout instead. Note that dropout2d exists to provide channel-wise dropout on inputs with 2 spatial dimensions, a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).
[1,mpirank:14,algo-2]<stderr>:  warnings.warn(warn_msg)
[1,mpirank:7,algo-1]<stderr>:/opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:1331: UserWarning: dropout2d: Received a 2-D input to dropout2d, which is deprecated and will result in an error in a future release. To retain the behavior and silence this warning, please use dropout instead. Note that dropout2d exists to provide channel-wise dropout on inputs with 2 spatial dimensions, a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).
[1,mpirank:7,algo-1]<stderr>:  warnings.warn(warn_msg)
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Using network AWS Libfabric
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO DMA-BUF is available on GPU device 2
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Using network AWS Libfabric
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO DMA-BUF is available on GPU device 7
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Using network AWS Libfabric
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO DMA-BUF is available on GPU device 1
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Using network AWS Libfabric
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO DMA-BUF is available on GPU device 5
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Using network AWS Libfabric
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO DMA-BUF is available on GPU device 3
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Using network AWS Libfabric
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO DMA-BUF is available on GPU device 0
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Using network AWS Libfabric
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO DMA-BUF is available on GPU device 6
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Using network AWS Libfabric
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO DMA-BUF is available on GPU device 4
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:8,algo-2]<stdout>:NCCL version 2.17.1+cuda11.8
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Using network AWS Libfabric
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO DMA-BUF is available on GPU device 4
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Using network AWS Libfabric
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO DMA-BUF is available on GPU device 1
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Using network AWS Libfabric
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO DMA-BUF is available on GPU device 6
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Using network AWS Libfabric
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO DMA-BUF is available on GPU device 2
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Using network AWS Libfabric
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO DMA-BUF is available on GPU device 3
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Using network AWS Libfabric
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO DMA-BUF is available on GPU device 0
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Using network AWS Libfabric
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO DMA-BUF is available on GPU device 5
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Using network AWS Libfabric
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO DMA-BUF is available on GPU device 7
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/conda/share/aws-ofi-nccl/xml/p4d-24xl-topo.xml
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Setting affinity for GPU 1 to 0fc00000,00000fc0
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Setting affinity for GPU 3 to fc,00000000,00fc0000
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Setting affinity for GPU 0 to 3f0000,0000003f
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Setting affinity for GPU 6 to 03f00000,000003f0,00000000
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Setting affinity for GPU 2 to 03,f0000000,0003f000
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Setting affinity for GPU 7 to fc000000,0000fc00,00000000
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Setting affinity for GPU 5 to 0fc000,0000000f,c0000000
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Setting affinity for GPU 4 to 3f00,00000000,3f000000
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Setting affinity for GPU 6 to 03f00000,000003f0,00000000
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Setting affinity for GPU 7 to fc000000,0000fc00,00000000
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Setting affinity for GPU 5 to 0fc000,0000000f,c0000000
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Setting affinity for GPU 4 to 3f00,00000000,3f000000
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Setting affinity for GPU 1 to 0fc00000,00000fc0
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Setting affinity for GPU 2 to 03,f0000000,0003f000
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Setting affinity for GPU 0 to 3f0000,0000003f
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Setting affinity for GPU 3 to fc,00000000,00fc0000
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO P2P Chunksize set to 524288
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO P2P Chunksize set to 524288
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO P2P Chunksize set to 524288
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 00/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 01/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 02/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 03/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 04/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 05/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 06/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 07/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 08/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 09/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 10/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 11/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 12/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 13/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 14/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 15/24 :    0   1   2   3   4   5   6   7
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO P2P Chunksize set to 524288
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO P2P Chunksize set to 524288
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO P2P Chunksize set to 524288
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO P2P Chunksize set to 524288
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 16/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 17/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 18/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 19/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 20/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 21/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 22/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 23/24 :    0   1   2   3   4   5   6   7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO P2P Chunksize set to 524288
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO P2P Chunksize set to 524288
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 00/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 01/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 02/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 03/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 04/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 05/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 06/24 :    0   1   2   3   4   5   6   7
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO P2P Chunksize set to 524288
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO P2P Chunksize set to 524288
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO P2P Chunksize set to 524288
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO P2P Chunksize set to 524288
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO P2P Chunksize set to 524288
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO P2P Chunksize set to 524288
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 07/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 08/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 09/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 10/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 11/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 12/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 13/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 14/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 15/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 16/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 17/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 18/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 19/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 20/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 21/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 22/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 23/24 :    0   1   2   3   4   5   6   7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO P2P Chunksize set to 524288
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 00/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 00/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 00/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 00/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 00/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 00/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 00/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 01/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 01/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 01/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 01/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 01/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 00/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 01/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 01/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 00/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 00/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 00/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 00/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 02/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 00/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 00/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 02/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 00/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 02/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 02/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 02/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 02/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 01/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 02/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 02/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 01/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 01/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 01/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 01/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 03/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 01/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 01/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 03/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 01/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 03/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 03/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 03/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 02/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 03/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 03/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 02/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 02/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 02/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 02/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 04/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 02/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 02/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 04/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 02/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 04/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 04/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 04/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 04/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 03/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 03/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 03/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 03/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 04/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 03/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 05/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 03/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 03/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 05/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 05/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 03/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 05/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 05/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 05/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 04/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 04/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 04/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 04/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 05/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 04/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 06/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 04/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 04/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 06/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 04/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 06/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 06/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 06/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 06/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 06/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 05/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 05/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 06/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 05/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 05/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 05/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 07/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 05/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 05/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 07/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 07/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 07/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 05/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 07/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 07/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 06/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 06/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 06/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 07/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 06/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 06/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 06/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 08/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 06/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 08/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 08/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 08/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 08/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 06/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 08/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 08/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 07/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 07/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 07/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 08/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 07/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 07/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 07/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 09/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 07/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 09/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 09/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 09/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 09/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 07/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 09/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 09/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 08/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 08/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 08/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 08/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 09/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 08/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 08/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 10/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 08/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 10/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 10/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 10/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 10/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 08/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 10/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 10/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 09/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 10/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 09/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 09/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 09/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 11/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 09/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 09/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 11/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 11/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 09/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 11/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 11/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 09/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 11/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 11/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 10/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 11/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 10/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 10/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 10/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 12/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 10/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 10/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 12/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 12/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 10/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 12/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 12/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 10/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 12/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 12/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 11/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 12/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 13/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 11/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 11/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 11/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 11/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 11/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 13/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 13/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 11/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 13/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 13/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 13/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 13/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 11/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 12/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 13/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 14/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 12/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 12/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 12/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 12/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 12/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 14/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 14/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 14/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 14/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 12/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 14/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 13/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 12/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 14/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 14/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 15/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 13/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 13/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 13/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 15/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 13/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 13/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 15/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 15/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 15/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 13/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 15/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 15/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 13/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 14/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 15/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 16/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 14/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 14/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 14/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 16/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 14/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 14/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 16/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 16/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 16/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 14/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 16/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 16/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 15/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 14/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 17/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 16/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 17/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 15/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 15/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 15/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 17/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 15/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 15/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 17/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 17/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 15/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 17/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 17/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 15/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 16/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 18/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 17/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 18/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 16/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 16/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 16/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 16/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 18/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 16/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 18/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 18/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 16/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 18/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 18/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 18/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 19/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 17/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 16/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 19/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 17/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 17/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 17/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 17/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 19/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 17/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 19/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 19/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 17/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 20/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 19/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 19/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 19/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 17/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 18/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 20/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 18/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 18/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 18/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 18/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 18/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 20/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 20/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 20/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 18/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 20/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 21/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 20/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 20/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 19/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 18/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 21/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 19/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 19/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 19/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 19/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 19/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 21/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 21/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 21/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 21/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 19/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 21/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 22/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 21/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 19/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 20/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 22/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 20/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 22/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 20/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 20/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 20/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 20/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 22/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 22/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 22/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 22/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 22/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 23/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 20/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 20/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 21/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 23/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 21/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 23/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 23/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 21/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 21/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 21/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 21/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Channel 23/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 23/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 23/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 23/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 21/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 22/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 21/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 22/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 22/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 22/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 22/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 22/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 22/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 22/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 23/0 : 3[201d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 23/0 : 7[a01d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 23/0 : 2[201c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 23/0 : 1[101d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 23/0 : 6[a01c0] -> 7[a01d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 23/0 : 5[901d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 23/0 : 4[901c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Channel 23/0 : 0[101c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Connected all rings
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Connected all rings
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Connected all rings
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Connected all rings
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Connected all rings
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 00/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Connected all rings
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Connected all rings
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Connected all rings
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Connected all rings
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 01/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Connected all rings
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Connected all rings
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Connected all rings
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 02/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Connected all rings
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Connected all rings
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Connected all rings
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 00/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Connected all rings
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 03/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 01/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 04/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 02/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 05/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 03/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 06/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 04/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 07/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 05/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 08/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 06/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 09/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 07/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 10/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 08/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 11/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 09/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 10/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 12/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 11/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 13/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 12/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 14/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 15/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 13/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 16/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 14/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 15/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 17/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 16/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 18/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 17/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 19/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 18/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 20/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 19/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 21/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 20/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 22/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 21/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Channel 23/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 22/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 00/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 00/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 00/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 00/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Channel 23/0 : 7[a01d0] -> 6[a01c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 00/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 00/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 01/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 01/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 01/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 01/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 00/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 00/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 00/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 01/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 01/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 02/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 02/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 01/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 00/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 02/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 02/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 01/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 01/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 00/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 02/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 00/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 02/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 03/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 01/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 02/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 03/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 02/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 03/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 03/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 02/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 01/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 03/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 01/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 03/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 04/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 03/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 04/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 02/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 03/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 04/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 04/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 02/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 03/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 04/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 02/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 04/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 04/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 05/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 05/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 04/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 03/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 05/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 05/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 04/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 03/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 05/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 03/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 05/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 05/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 05/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 06/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 06/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 06/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 06/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 04/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 04/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 05/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 06/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 04/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 06/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 06/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 06/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 07/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 07/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 07/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 07/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 05/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 06/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 05/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 07/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 05/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 07/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 07/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 08/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 08/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 07/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 08/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 08/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 06/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 08/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 07/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 06/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 06/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 08/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 08/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 09/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 09/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 08/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 09/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 09/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 07/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 07/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 09/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 08/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 07/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 09/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 09/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 10/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 10/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 09/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 10/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 10/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 08/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 08/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 09/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 10/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 08/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 10/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 11/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 11/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 10/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 10/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 11/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 11/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 09/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 10/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 09/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 11/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 09/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 11/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 11/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 12/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 12/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 11/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 12/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 12/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 10/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 11/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 10/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 12/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 12/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 10/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 13/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 12/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 12/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 13/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 13/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 13/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 11/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 12/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 11/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 13/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 13/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 11/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 13/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 14/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 14/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 14/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 14/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 13/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 12/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 13/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 14/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 12/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 12/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 14/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 15/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 15/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 15/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 14/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 15/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 14/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 13/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 14/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 15/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 13/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 13/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 15/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 15/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 16/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 16/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 16/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 16/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 15/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 14/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 15/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 16/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 14/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 14/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 17/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 16/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 17/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 16/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 17/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 17/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 15/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 16/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 17/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 16/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 15/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 17/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 15/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 18/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 18/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 18/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 17/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 18/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 16/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 17/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 18/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 17/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 16/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 16/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 18/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 19/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 18/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 19/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 19/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 19/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 18/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 17/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 19/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 18/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 17/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 19/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 17/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 20/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 20/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 19/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 20/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 20/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 19/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 18/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 20/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 19/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 18/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 18/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 20/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 21/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 21/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 21/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 20/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 20/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 21/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 19/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 21/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 20/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 21/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 19/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 19/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 22/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 21/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 22/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 22/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 21/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 22/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 20/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 22/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 21/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 20/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Channel 23/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 22/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 20/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Channel 23/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 22/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Channel 23/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 22/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Channel 23/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 21/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Channel 23/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 22/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 21/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Channel 23/0 : 2[201c0] -> 1[101d0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 21/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Channel 23/0 : 4[901c0] -> 3[201d0] via P2P/IPC/read
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Channel 23/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 22/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO Connected all trees
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO 24 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO Connected all trees
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO 24 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Channel 23/0 : 6[a01c0] -> 5[901d0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 22/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 22/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Channel 23/0 : 5[901d0] -> 4[901c0] via P2P/IPC/read
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Channel 23/0 : 3[201d0] -> 2[201c0] via P2P/IPC/read
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Channel 23/0 : 1[101d0] -> 0[101c0] via P2P/IPC/read
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO Connected all trees
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO 24 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO Connected all trees
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO 24 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO Connected all trees
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO 24 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO Connected all trees
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO 24 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO Connected all trees
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO 24 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO Connected all trees
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO Connected all trees
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO 24 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO 24 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO Connected all trees
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO 24 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO Connected all trees
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO 24 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO Connected all trees
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO 24 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO Connected all trees
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO 24 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
[1,mpirank:3,algo-1]<stdout>:algo-2:183:360 [3] NCCL INFO comm 0x7fb26cb6f070 rank 3 nranks 8 cudaDev 3 busId 201d0 commId 0x9e611c13ebacc689 - Init COMPLETE
[1,mpirank:2,algo-1]<stdout>:algo-2:189:355 [2] NCCL INFO comm 0x7fddf8b65060 rank 2 nranks 8 cudaDev 2 busId 201c0 commId 0x9e611c13ebacc689 - Init COMPLETE
[1,mpirank:1,algo-1]<stdout>:algo-2:179:349 [1] NCCL INFO comm 0x7fb37cb63ee0 rank 1 nranks 8 cudaDev 1 busId 101d0 commId 0x9e611c13ebacc689 - Init COMPLETE
[1,mpirank:6,algo-1]<stdout>:algo-2:187:358 [6] NCCL INFO comm 0x7efd4cb63db0 rank 6 nranks 8 cudaDev 6 busId a01c0 commId 0x9e611c13ebacc689 - Init COMPLETE
[1,mpirank:0,algo-1]<stdout>:algo-2:203:352 [0] NCCL INFO comm 0x7faab0b7c120 rank 0 nranks 8 cudaDev 0 busId 101c0 commId 0x9e611c13ebacc689 - Init COMPLETE
[1,mpirank:5,algo-1]<stdout>:algo-2:185:354 [5] NCCL INFO comm 0x7f850cb64e70 rank 5 nranks 8 cudaDev 5 busId 901d0 commId 0x9e611c13ebacc689 - Init COMPLETE
[1,mpirank:4,algo-1]<stdout>:algo-2:186:359 [4] NCCL INFO comm 0x7f0724b64dd0 rank 4 nranks 8 cudaDev 4 busId 901c0 commId 0x9e611c13ebacc689 - Init COMPLETE
[1,mpirank:7,algo-1]<stdout>:algo-2:184:357 [7] NCCL INFO comm 0x7f728cb68490 rank 7 nranks 8 cudaDev 7 busId a01d0 commId 0x9e611c13ebacc689 - Init COMPLETE
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO Connected all trees
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO 24 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO Connected all trees
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO 24 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO Connected all trees
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO NCCL_PROTO set by environment to simple
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO 24 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer
[1,mpirank:12,algo-2]<stdout>:algo-1:184:353 [4] NCCL INFO comm 0x7f7488b79980 rank 4 nranks 8 cudaDev 4 busId 901c0 commId 0x6c7a1c18d57a7184 - Init COMPLETE
[1,mpirank:9,algo-2]<stdout>:algo-1:182:357 [1] NCCL INFO comm 0x7faa1cb63db0 rank 1 nranks 8 cudaDev 1 busId 101d0 commId 0x6c7a1c18d57a7184 - Init COMPLETE
[1,mpirank:11,algo-2]<stdout>:algo-1:183:356 [3] NCCL INFO comm 0x7f926cb64de0 rank 3 nranks 8 cudaDev 3 busId 201d0 commId 0x6c7a1c18d57a7184 - Init COMPLETE
[1,mpirank:8,algo-2]<stdout>:algo-1:203:348 [0] NCCL INFO comm 0x7ff08cb67830 rank 0 nranks 8 cudaDev 0 busId 101c0 commId 0x6c7a1c18d57a7184 - Init COMPLETE
[1,mpirank:15,algo-2]<stdout>:algo-1:186:358 [7] NCCL INFO comm 0x7f89bcb68360 rank 7 nranks 8 cudaDev 7 busId a01d0 commId 0x6c7a1c18d57a7184 - Init COMPLETE
[1,mpirank:13,algo-2]<stdout>:algo-1:188:354 [5] NCCL INFO comm 0x7ff558b64e70 rank 5 nranks 8 cudaDev 5 busId 901d0 commId 0x6c7a1c18d57a7184 - Init COMPLETE
[1,mpirank:10,algo-2]<stdout>:algo-1:187:352 [2] NCCL INFO comm 0x7f380cb64f30 rank 2 nranks 8 cudaDev 2 busId 201c0 commId 0x6c7a1c18d57a7184 - Init COMPLETE
[1,mpirank:14,algo-2]<stdout>:algo-1:185:355 [6] NCCL INFO comm 0x7f080cb63ee0 rank 6 nranks 8 cudaDev 6 busId a01c0 commId 0x6c7a1c18d57a7184 - Init COMPLETE
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 1 [0/60000 (0%)]#011Loss: 2.323947
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 1 [10240/60000 (17%)]#011Loss: 1.373116
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 1 [20480/60000 (34%)]#011Loss: 0.785794
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 1 [30720/60000 (51%)]#011Loss: 0.282431
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 1 [40960/60000 (68%)]#011Loss: 0.195794
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 1 [51200/60000 (85%)]#011Loss: 0.135728
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Test set: Average loss: 0.1108, Accuracy: 9650/10000 (96%)
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 2 [0/60000 (0%)]#011Loss: 0.215010
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 2 [10240/60000 (17%)]#011Loss: 0.258592
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 2 [20480/60000 (34%)]#011Loss: 0.292565
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 2 [30720/60000 (51%)]#011Loss: 0.116239
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 2 [40960/60000 (68%)]#011Loss: 0.081883
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 2 [51200/60000 (85%)]#011Loss: 0.038396
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Test set: Average loss: 0.0687, Accuracy: 9780/10000 (98%)
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 3 [0/60000 (0%)]#011Loss: 0.200773
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 3 [10240/60000 (17%)]#011Loss: 0.179987
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 3 [20480/60000 (34%)]#011Loss: 0.273319
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 3 [30720/60000 (51%)]#011Loss: 0.142453
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 3 [40960/60000 (68%)]#011Loss: 0.071805
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 3 [51200/60000 (85%)]#011Loss: 0.104552
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Test set: Average loss: 0.0545, Accuracy: 9822/10000 (98%)
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 4 [0/60000 (0%)]#011Loss: 0.186415
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 4 [10240/60000 (17%)]#011Loss: 0.144717
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 4 [20480/60000 (34%)]#011Loss: 0.283975
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 4 [30720/60000 (51%)]#011Loss: 0.124121
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 4 [40960/60000 (68%)]#011Loss: 0.051800
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 4 [51200/60000 (85%)]#011Loss: 0.084833
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Test set: Average loss: 0.0468, Accuracy: 9846/10000 (98%)
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 5 [0/60000 (0%)]#011Loss: 0.130977
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 5 [10240/60000 (17%)]#011Loss: 0.097565
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 5 [20480/60000 (34%)]#011Loss: 0.201473
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 5 [30720/60000 (51%)]#011Loss: 0.053192
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 5 [40960/60000 (68%)]#011Loss: 0.043612
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 5 [51200/60000 (85%)]#011Loss: 0.040844
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Test set: Average loss: 0.0445, Accuracy: 9851/10000 (99%)
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 6 [0/60000 (0%)]#011Loss: 0.117017
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 6 [10240/60000 (17%)]#011Loss: 0.131274
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 6 [20480/60000 (34%)]#011Loss: 0.125333
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 6 [30720/60000 (51%)]#011Loss: 0.052602
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 6 [40960/60000 (68%)]#011Loss: 0.070306
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 6 [51200/60000 (85%)]#011Loss: 0.028939
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Test set: Average loss: 0.0418, Accuracy: 9853/10000 (99%)
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 7 [0/60000 (0%)]#011Loss: 0.157929
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 7 [10240/60000 (17%)]#011Loss: 0.051118
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 7 [20480/60000 (34%)]#011Loss: 0.214014
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 7 [30720/60000 (51%)]#011Loss: 0.025940
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 7 [40960/60000 (68%)]#011Loss: 0.057824
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 7 [51200/60000 (85%)]#011Loss: 0.042735
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Test set: Average loss: 0.0404, Accuracy: 9864/10000 (99%)
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 8 [0/60000 (0%)]#011Loss: 0.142139
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 8 [10240/60000 (17%)]#011Loss: 0.021918
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 8 [20480/60000 (34%)]#011Loss: 0.132315
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 8 [30720/60000 (51%)]#011Loss: 0.025287
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 8 [40960/60000 (68%)]#011Loss: 0.024889
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 8 [51200/60000 (85%)]#011Loss: 0.043529
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Test set: Average loss: 0.0394, Accuracy: 9868/10000 (99%)
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 9 [0/60000 (0%)]#011Loss: 0.132236
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 9 [10240/60000 (17%)]#011Loss: 0.065302
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 9 [20480/60000 (34%)]#011Loss: 0.139778
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 9 [30720/60000 (51%)]#011Loss: 0.047606
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 9 [40960/60000 (68%)]#011Loss: 0.027578
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 9 [51200/60000 (85%)]#011Loss: 0.090857
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Test set: Average loss: 0.0392, Accuracy: 9865/10000 (99%)
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 10 [0/60000 (0%)]#011Loss: 0.085122
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 10 [10240/60000 (17%)]#011Loss: 0.089514
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 10 [20480/60000 (34%)]#011Loss: 0.169567
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 10 [30720/60000 (51%)]#011Loss: 0.034910
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 10 [40960/60000 (68%)]#011Loss: 0.042787
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 10 [51200/60000 (85%)]#011Loss: 0.016882
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Test set: Average loss: 0.0385, Accuracy: 9865/10000 (99%)
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 11 [0/60000 (0%)]#011Loss: 0.061399
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 11 [10240/60000 (17%)]#011Loss: 0.063447
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 11 [20480/60000 (34%)]#011Loss: 0.118218
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 11 [30720/60000 (51%)]#011Loss: 0.097643
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 11 [40960/60000 (68%)]#011Loss: 0.091272
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 11 [51200/60000 (85%)]#011Loss: 0.028417
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Test set: Average loss: 0.0381, Accuracy: 9867/10000 (99%)
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 12 [0/60000 (0%)]#011Loss: 0.136254
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 12 [10240/60000 (17%)]#011Loss: 0.080658
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 12 [20480/60000 (34%)]#011Loss: 0.120809
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 12 [30720/60000 (51%)]#011Loss: 0.049508
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 12 [40960/60000 (68%)]#011Loss: 0.047645
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 12 [51200/60000 (85%)]#011Loss: 0.051511
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Test set: Average loss: 0.0378, Accuracy: 9867/10000 (99%)
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 13 [0/60000 (0%)]#011Loss: 0.100337
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 13 [10240/60000 (17%)]#011Loss: 0.064384
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 13 [20480/60000 (34%)]#011Loss: 0.185607
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 13 [30720/60000 (51%)]#011Loss: 0.102012
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 13 [40960/60000 (68%)]#011Loss: 0.029130
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 13 [51200/60000 (85%)]#011Loss: 0.021421
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Test set: Average loss: 0.0379, Accuracy: 9868/10000 (99%)
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 14 [0/60000 (0%)]#011Loss: 0.136252
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 14 [10240/60000 (17%)]#011Loss: 0.096145
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 14 [20480/60000 (34%)]#011Loss: 0.244538
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 14 [30720/60000 (51%)]#011Loss: 0.063404
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 14 [40960/60000 (68%)]#011Loss: 0.026542
[1,mpirank:0,algo-1]<stdout>:Train Epoch: 14 [51200/60000 (85%)]#011Loss: 0.027133
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:0,algo-1]<stdout>:Test set: Average loss: 0.0378, Accuracy: 9868/10000 (99%)
[1,mpirank:0,algo-1]<stdout>:
[1,mpirank:6,algo-1]<stdout>:algo-2:187:512 [0] NCCL INFO [Service thread] Connection closed by localRank 5
[1,mpirank:4,algo-1]<stdout>:algo-2:186:518 [0] NCCL INFO [Service thread] Connection closed by localRank 5
[1,mpirank:0,algo-1]<stdout>:algo-2:203:513 [0] NCCL INFO [Service thread] Connection closed by localRank 5
[1,mpirank:2,algo-1]<stdout>:algo-2:189:516 [0] NCCL INFO [Service thread] Connection closed by localRank 5
[1,mpirank:4,algo-1]<stdout>:algo-2:186:518 [0] NCCL INFO [Service thread] Connection closed by localRank 1
[1,mpirank:6,algo-1]<stdout>:algo-2:187:512 [0] NCCL INFO [Service thread] Connection closed by localRank 1
[1,mpirank:2,algo-1]<stdout>:algo-2:189:516 [0] NCCL INFO [Service thread] Connection closed by localRank 1
[1,mpirank:0,algo-1]<stdout>:algo-2:203:513 [0] NCCL INFO [Service thread] Connection closed by localRank 1
[1,mpirank:6,algo-1]<stdout>:algo-2:187:512 [0] NCCL INFO [Service thread] Connection closed by localRank 3
[1,mpirank:4,algo-1]<stdout>:algo-2:186:518 [0] NCCL INFO [Service thread] Connection closed by localRank 3
[1,mpirank:0,algo-1]<stdout>:algo-2:203:513 [0] NCCL INFO [Service thread] Connection closed by localRank 3
[1,mpirank:2,algo-1]<stdout>:algo-2:189:516 [0] NCCL INFO [Service thread] Connection closed by localRank 3
[1,mpirank:10,algo-2]<stdout>:algo-1:187:512 [0] NCCL INFO [Service thread] Connection closed by localRank 5
[1,mpirank:12,algo-2]<stdout>:algo-1:184:511 [0] NCCL INFO [Service thread] Connection closed by localRank 5
[1,mpirank:8,algo-2]<stdout>:algo-1:203:516 [0] NCCL INFO [Service thread] Connection closed by localRank 5
[1,mpirank:14,algo-2]<stdout>:algo-1:185:513 [0] NCCL INFO [Service thread] Connection closed by localRank 5
[1,mpirank:12,algo-2]<stdout>:algo-1:184:511 [0] NCCL INFO [Service thread] Connection closed by localRank 7
[1,mpirank:14,algo-2]<stdout>:algo-1:185:513 [0] NCCL INFO [Service thread] Connection closed by localRank 7
[1,mpirank:8,algo-2]<stdout>:algo-1:203:516 [0] NCCL INFO [Service thread] Connection closed by localRank 7
[1,mpirank:10,algo-2]<stdout>:algo-1:187:512 [0] NCCL INFO [Service thread] Connection closed by localRank 7
[1,mpirank:14,algo-2]<stdout>:algo-1:185:513 [0] NCCL INFO [Service thread] Connection closed by localRank 3
[1,mpirank:12,algo-2]<stdout>:algo-1:184:511 [0] NCCL INFO [Service thread] Connection closed by localRank 3
[1,mpirank:10,algo-2]<stdout>:algo-1:187:512 [0] NCCL INFO [Service thread] Connection closed by localRank 3
[1,mpirank:8,algo-2]<stdout>:algo-1:203:516 [0] NCCL INFO [Service thread] Connection closed by localRank 3
[1,mpirank:12,algo-2]<stdout>:algo-1:184:511 [0] NCCL INFO [Service thread] Connection closed by localRank 1
[1,mpirank:10,algo-2]<stdout>:algo-1:187:512 [0] NCCL INFO [Service thread] Connection closed by localRank 1
[1,mpirank:14,algo-2]<stdout>:algo-1:185:513 [0] NCCL INFO [Service thread] Connection closed by localRank 1
[1,mpirank:8,algo-2]<stdout>:algo-1:203:516 [0] NCCL INFO [Service thread] Connection closed by localRank 1
[1,mpirank:4,algo-1]<stdout>:algo-2:186:518 [0] NCCL INFO [Service thread] Connection closed by localRank 6
[1,mpirank:2,algo-1]<stdout>:algo-2:189:516 [0] NCCL INFO [Service thread] Connection closed by localRank 6
[1,mpirank:0,algo-1]<stdout>:algo-2:203:513 [0] NCCL INFO [Service thread] Connection closed by localRank 6
[1,mpirank:4,algo-1]<stdout>:algo-2:186:518 [0] NCCL INFO [Service thread] Connection closed by localRank 7
[1,mpirank:2,algo-1]<stdout>:algo-2:189:516 [0] NCCL INFO [Service thread] Connection closed by localRank 7
[1,mpirank:0,algo-1]<stdout>:algo-2:203:513 [0] NCCL INFO [Service thread] Connection closed by localRank 7
[1,mpirank:4,algo-1]<stdout>:algo-2:186:518 [0] NCCL INFO [Service thread] Connection closed by localRank 2
[1,mpirank:0,algo-1]<stdout>:algo-2:203:513 [0] NCCL INFO [Service thread] Connection closed by localRank 2
[1,mpirank:14,algo-2]<stdout>:algo-1:185:513 [0] NCCL INFO [Service thread] Connection closed by localRank 4
[1,mpirank:8,algo-2]<stdout>:algo-1:203:516 [0] NCCL INFO [Service thread] Connection closed by localRank 4
[1,mpirank:10,algo-2]<stdout>:algo-1:187:512 [0] NCCL INFO [Service thread] Connection closed by localRank 4
[1,mpirank:8,algo-2]<stdout>:algo-1:203:516 [0] NCCL INFO [Service thread] Connection closed by localRank 6
[1,mpirank:10,algo-2]<stdout>:algo-1:187:512 [0] NCCL INFO [Service thread] Connection closed by localRank 6
[1,mpirank:10,algo-2]<stdout>:algo-1:187:512 [0] NCCL INFO [Service thread] Connection closed by localRank 0
[1,mpirank:0,algo-1]<stdout>:algo-2:203:513 [0] NCCL INFO [Service thread] Connection closed by localRank 4
2024-05-31 01:11:06,301 sagemaker-training-toolkit INFO     Invoked on_terminate from psutil.wait_for_procs
2024-05-31 01:11:06,302 sagemaker-training-toolkit INFO     process psutil.Process(pid=67, name='orted', status='terminated', started='01:10:00') terminated with exit code None
2024-05-31 01:11:06,302 sagemaker-training-toolkit INFO     Reporting status for ORTEd process. gone: [psutil.Process(pid=67, name='orted', status='terminated', started='01:10:00')] alive: []
2024-05-31 01:11:06,302 sagemaker-training-toolkit INFO     Orted process exited
2024-05-31 01:11:06,269 sagemaker-training-toolkit INFO     Waiting for the process to finish and give a return code.
2024-05-31 01:11:06,269 sagemaker-training-toolkit INFO     Done waiting for a return code. Received 0 from exiting process.
2024-05-31 01:11:06,270 sagemaker-training-toolkit INFO     Begin writing status file from leader node to worker nodes
2024-05-31 01:11:06,270 sagemaker-training-toolkit INFO     Start writing mpirun finished status to algo-1
2024-05-31 01:11:06,411 sagemaker-training-toolkit INFO     output from subprocess run CompletedProcess(args=['ssh', 'algo-1', 'touch', '/tmp/done.algo-2'], returncode=0, stdout='', stderr='')
2024-05-31 01:11:06,411 sagemaker-training-toolkit INFO     Finished writing status file

2024-05-31 01:11:39 Uploading - Uploading generated training model2024-05-31 01:11:36,332 sagemaker-training-toolkit INFO     Begin looking for status file on algo-1
2024-05-31 01:11:36,332 sagemaker-training-toolkit INFO     MPI training job status file found. Exit gracefully
2024-05-31 01:11:36,332 sagemaker-training-toolkit INFO     End looking for status file
2024-05-31 01:11:36,332 sagemaker-training-toolkit INFO     MPI process finished.
2024-05-31 01:11:36,332 sagemaker-training-toolkit INFO     Reporting training SUCCESS
2024-05-31 01:11:36,438 sagemaker-training-toolkit INFO     Finished writing status file from leader node to worker nodes
2024-05-31 01:11:36,439 sagemaker-training-toolkit INFO     Reporting training SUCCESS

2024-05-31 01:11:52 Completed - Training job completed

Next steps

Now that you have a trained model, you can deploy an endpoint to host the model. After you deploy the endpoint, you can then test it with inference requests. The following cell will store the model_data variable to be used with the inference notebook.

[ ]:

model_data = estimator.model_data
print("Storing {} as model_data".format(model_data))
%store model_data

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.

This us-east-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This us-east-2 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This us-west-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This ca-central-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This sa-east-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This eu-west-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This eu-west-2 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This eu-west-3 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This eu-central-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This eu-north-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This ap-southeast-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This ap-southeast-2 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This ap-northeast-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This ap-northeast-2 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable

This ap-south-1 badge failed to load. Check your device’s internet connectivity, otherwise the service is currently unavailable