Analyze collectives with the NcclAnalyser SDK

Analyze collectives with the NcclAnalyser SDK#

2026-07-22

4 min read time

In distributed deep learning, analyzing the performance of collective communication operations is crucial for diagnosing and optimizing performance at scale. NcclAnalyser is a Python SDK that parses and analyzes NVIDIA Collective Communications Library (NCCL) and ROCm Collective Communications Library (RCCL) kernel events from PyTorch trace files (JSON). It computes key metrics like communication latency, message sizes, algorithm bandwidth, bus bandwidth, and synchronization metrics (for example skew in start and end times), providing insights into communication patterns and potential bottlenecks in your distributed training or inference workflows.

Note

This topic covers the NcclAnalyser SDK and its API. To generate a multi-rank collective-communication report from the command line instead, see Generate a collective-communication report.

Key features#

  • Detailed NCCL event extraction: Parses JSON trace files to extract per-rank NCCL kernel events (those containing nccl in the kernel name).

  • Correlated events through collective ID: Each collective operation is assigned a unique collective_id, generated by combining the Process Group Name and an index_in_group. The index_in_group reflects the order in which that collective appears in the trace.

  • Latency computation for implicit-sync collectives: For collectives like allreduce, allgather, reducescatter, and alltoall, the tool computes communication latency by taking the minimum duration across ranks to eliminate waiting time.

  • Bandwidth analysis: Computes algorithm bandwidth based on input message size and communication latency, and finally bus bandwidth (link utilization).

  • Synchronization (wait-time) metrics: Tracks skew in start times (how late certain ranks arrive) and skew in end times, providing insights for diagnosing load imbalance or synchronization overheads.

  • Alltoallv support: Provides raw data for alltoallv events, which don’t enforce an implicit sync pattern. Includes per-instance metrics (wall time, throughput, size imbalance) and a summary aggregation grouped by process group and dtype. An optional heatmap mode (build_df_all2allv_heatmap) produces per rank-pair send volumes, useful for diagnosing Mixture-of-Experts (MoE) / expert-parallel routing imbalance.

  • Summary and detailed dataframes: Supports both high-level summaries for quick insights and detailed dataframes for advanced analysis, including per-rank metadata and performance metrics. Power users can build custom pipelines and analyses on top of the detailed dataframes.

  • PyTorch support: Currently built for PyTorch trace files, with the potential to extend support to other frameworks.

Before you begin#

  • TraceLens installed (see Install TraceLens).

  • Per-rank PyTorch profiler traces from a distributed run (one trace per rank).

Quick start#

NcclAnalyser can generate both a per-collective dataframe and a global summary. More features are demonstrated in the nccl_analyser_example.ipynb notebook.

Quick summary of implicit-sync collectives#

from TraceLens import NcclAnalyser
import os

# Define the root directory and create file paths for all ranks
root_dir = '/path/to/your/trace/files/directory'
world_size = 8
list_profile_filepaths = [os.path.join(root_dir, f'rank{i}_trace.json') for i in range(world_size)]

# Initialize the NCCL Analyser
my_nccl_analyser = NcclAnalyser(list_profile_filepaths, world_size)

# Generate the summarized dataframe for implicit-sync collectives
df_summary = my_nccl_analyser.build_df_summary_nccl_implicit_sync_cat(agg_metrics=['mean'])
print(df_summary.head())

# Save the summary to CSV
df_summary.to_csv('nccl_summary.csv', index=False)

The summarized dataframe:

  • Groups events by collective type, message size, and data type to compute aggregated metrics.

  • For communication performance, computes communication latency, algorithm bandwidth, and bus bandwidth.

  • For synchronization delay, computes metrics like skew in start times, which shows the wait time. This is based on the difference between the earliest and latest arrival across the ranks for a given collective.

Example summarized dataframe#

Collective name

In msg size (MB)

dtype

comm latency (µs)_mean

count

Total latency (ms)

algo bw (GB/s)_mean

bus bw (GB/s)_mean

skew in start time (µs)_mean

_allgather_base

204.00

BFloat16

6041.88

318

1921.32

33.00

28.88

11779.36

_reduce_scatter_base

3264.06

Float

11662.77

160

1866.04

273.43

239.25

60238.77

_reduce_scatter_base

8016.03

Float

22988.50

2

45.98

340.53

297.96

146.48

_allgather_base

501.00

BFloat16

11920.84

2

23.84

41.04

35.91

15405.14

allreduce

0.00

Float

18.58

6

0.11

0.00

0.00

936.63

Note that the last row’s message size is just rounded down to 0.

Modify the file paths for your profiles in the nccl_analyser_example.ipynb notebook to get analysis instantly.

Contents