Using rocpd output format#

To accommodate diverse analysis workflows, rocprofv3 provides comprehensive support for multiple output formats:

  • rocpd (SQLite3 database) - Default format providing structured data storage.

  • CSV (Comma-Separated Values) - Tabular format for spreadsheet applications and data analysis tools.

  • JSON (JavaScript Object Notation) - Structured format optimized for programmatic analysis and integration.

  • PFTrace (Perfetto protocol buffers) - Binary trace format for high-performance visualization using Perfetto.

  • OTF2 (Open Trace Format 2) - Standardized trace format for interoperability with third-party analysis tools.

Warning

Direct CSV, PFTrace, and OTF2 output from rocprofv3 is deprecated and might omit data for some tracing features, including hipFILE and rocSHMEM tracing. Collect in the default rocpd format, then use rocpd convert to create CSV, PFTrace, or OTF2 output.

The rocpd output format serves as the primary data repository for rocprofv3 profiling sessions. This format leverages SQLite3’s ACID-compliant database engine to provide robust, structured storage of comprehensive profiling datasets. The relational schema enables efficient querying and manipulation of profiling data through standard SQL interfaces, facilitating complex analytical operations and custom reporting workflows.

Features#

  • Comprehensive data model: Consolidates all profiling artifacts including execution traces, performance counters, hardware metrics, and contextual metadata within a single SQLite3 database file (.db extension).

  • Standards-compliant access: Supports querying through industry-standard SQL interfaces including command-line tools (sqlite3 CLI), programming language bindings (Python sqlite3 module, C/C++ SQLite API), and database management applications.

  • Advanced analytics integration: Facilitates sophisticated post-processing workflows through custom analytical scripts, automated reporting systems, and integration with third-party visualization and analysis frameworks that provide SQLite3 connectivity.

Generating rocpd output#

To generate profiling data in the default rocpd format, use:

rocprofv3 --hip-trace -- <application>

Or, explicitly specify the rocpd output format using the --output-format parameter:

rocprofv3 --hip-trace --output-format rocpd -- <application>

The profiling session generates output files following the naming convention %hostname%/%pid%_results.db, where:

  • %hostname%: The system hostname.

  • %pid%: The process identifier of the profiled application.

Converting rocpd to alternative formats#

The rocpd database format supports conversion to alternative output formats for specialized analysis and visualization workflows.

The rocpd conversion utility is distributed as part of the ROCm installation package, located in /opt/rocm-<version>/bin, and provides both executable and Python module interfaces for programmatic integration.

Installing the rocpd Python prerequisites#

Before running rocpd, install the Python packages it depends on, such as pandas, otf2, and jinja2. The ROCm installation ships the list of these packages in <ROCM_PATH>/share/rocprofiler-sdk/requirements.txt, where <ROCM_PATH> is the ROCm installation directory, for example /opt/rocm:

python3 -m pip install -r /opt/rocm/share/rocprofiler-sdk/requirements.txt

To keep the packages separate from the system Python, install them into a virtual environment:

python3 -m venv rocpd-venv
source rocpd-venv/bin/activate
python3 -m pip install -r /opt/rocm/share/rocprofiler-sdk/requirements.txt

Note

Install the packages with the same Python interpreter that runs rocpd. If you have several Python versions installed, see the Python interpreter compatibility notes below.

To transform database files into target formats, run the rocpd convert command with appropriate parameters.

  • CSV format conversion

    /opt/rocm/bin/rocpd convert -i <input-file>.db --output-format csv
    

    The CSV conversion process generates a consolidated trace output file rocpd-output-data/out_regions_trace.csv path relative to the current working directory.

    This consolidated approach replaces the previous API-specific CSV files (out_hip_api_trace.csv, out_hsa_api_trace.csv, out_marker_api_trace.csv, etc.) to provide comprehensive coverage of all traced regions, including MPI functions, pthread functions, and other regions captured by rocprofiler-systems beyond the core ROCm APIs.

  • OTF2 format conversion

    /opt/rocm/bin/rocpd convert -i <input-file>.db --output-format otf2
    
  • Perfetto trace format conversion

    /opt/rocm/bin/rocpd convert -i <input-file>.db --output-format pftrace
    

Python interpreter compatibility:

On encountering Python interpreter version conflicts, specify the appropriate Python executable explicitly:

python3.10 $(which rocpd) convert -f csv -i <input-file>.db

rocpd convert command-line options#

The command-line options as displayed using rocpd convert --help are listed here:

usage: rocpd convert [-h] -i INPUT [INPUT ...] -f {csv,pftrace,otf2} [{csv,pftrace,otf2} ...]
                     [-o OUTPUT_FILE] [-d OUTPUT_PATH] [--automerge-limit LIMIT]
                     [--kernel-rename]
                     [--agent-index-value {absolute,relative,type-relative}]
                     [--perfetto-backend {inprocess,system}]
                     [--perfetto-buffer-fill-policy {discard,ring_buffer}]
                     [--perfetto-buffer-size KB] [--perfetto-shmem-size-hint KB]
                     [--group-by-queue]
                     [--start START | --start-marker START_MARKER]
                     [--end END | --end-marker END_MARKER]
                     [--inclusive INCLUSIVE]

The following table provides a detailed listing of the rocpd convert command-line options:

Category Option Description
Required arguments -i INPUT [INPUT ...], --input INPUT [INPUT ...] Specifies input database file paths. Accepts multiple SQLite3 database files separated by whitespace for batch processing operations.
-f {csv,pftrace,otf2} [{csv,pftrace,otf2} ...], --output-format {csv,pftrace,otf2} [{csv,pftrace,otf2} ...] Defines target output formats. Supports concurrent conversion to multiple formats such as CSV, PFTrace, and OTF2.
I/O configuration -o OUTPUT_​FILE, --output-file OUTPUT_​FILE Configures the base filename for generated output files (default: out).
-d OUTPUT_​PATH, --output-path OUTPUT_​PATH Specifies the target directory for output file generation (default: ./rocpd-output-data).
--automerge-limit LIMIT Controls the database auto-merge limit. When the number of input databases exceeds this limit, they are automatically merged into a .rpdb package to stay within SQLite3's attach limit of 10. Default: 1, maximum: 8.
Kernel identification options --kernel-rename Substitutes kernel function names with the corresponding ROCTx marker annotations for enhanced semantic context.
Device identification configuration --agent-index-value {absolute,relative,type-relative} Controls device identification methodology in the converted output. Here are the values:
  • absolute: Utilizes hardware node identifiers such as Agent-0, Agent-2, and Agent-4, while bypassing container group abstractions.
  • relative: Employs logical node identifiers such as Agent-0, Agent-1, and Agent-2, while incorporating container group context. This is the Default value.
  • type-relative: Applies device-type-specific logical identifiers such as CPU-0, GPU-0, and GPU-1, with independent numbering sequence per device class.
Perfetto trace configuration --perfetto-backend {inprocess,system} Configures Perfetto data collection architecture. The value system requires active traced and perfetto daemon processes, while inprocess operates autonomously. The default value is inprocess.
--perfetto-buffer-fill-policy {discard,ring_buffer} Defines buffer overflow handling strategy. The value discard drops new records when capacity is exceeded and ring_​buffer overwrites oldest records. The default value is discard.
--perfetto-buffer-size KB Sets the trace buffer capacity (in kilobytes) for Perfetto output generation. The default value is 1,048,576 KB or 1 GB.
--perfetto-shmem-size-hint KB Specifies shared memory allocation hint (in kilobytes) for Perfetto interprocess communication. The default value is 64 KB.
--group-by-queue Organizes trace data by HIP stream abstractions rather than low-level HSA queue identifiers, providing higher-level application context for kernel and memory transfer operations.
Temporal filtering configuration --start START Defines trace window start boundary using percentage notation such as 50% or absolute nanosecond timestamps such as 781470909013049.
--start-marker START_MARKER Specifies named marker event identifier to establish trace window start boundary.
--end END Defines trace window end boundary using percentage notation such as 75% or absolute nanosecond timestamps such as 3543724246381057.
--end-marker END_MARKER Specifies named marker event identifier to establish trace window end boundary.
--inclusive INCLUSIVE Controls event inclusion criteria. The value True includes events with either start or end timestamps within the specified window while False requires both timestamps within the window. The default value is True.
Command-line help -h, --help Displays comprehensive command syntax, parameter descriptions, and usage examples.

Types of conversion#

Here are the types of conversion supported by rocpd:

  • Single database conversion to Perfetto format

    /opt/rocm/bin/rocpd convert -i db1.db --output-format pftrace
    
  • Multi-Database conversion with temporal filtering

    The following example converts multiple databases to Perfetto format while specifying custom output directory and filename with temporal window constraint set to the final 70% of the trace duration:

    /opt/rocm/bin/rocpd convert -i db1.db db2.db --output-format pftrace -d ./output/ -o twoFileTraces --start 30% --end 100%
    
  • Batch conversion into multiple formats

    The following example processes six database files simultaneously, generating both CSV and Perfetto trace outputs with custom output configuration:

    /opt/rocm/bin/rocpd convert -i db{0..5}.db -f csv pftrace -d ~/output_folder/ -o sixFileTraces
    
  • Comprehensive format conversion

    The following example converts multiple databases into all supported formats (CSV, OTF2, and Perfetto trace) in a single operation:

    /opt/rocm/bin/rocpd convert -i db{3,4}.db -f csv otf2 pftrace
    
  • Convert .rpdb package

    The following example converts a packaged database collection to Perfetto format:

    /opt/rocm/bin/rocpd convert -i my_profiling_data.rpdb -f pftrace
    

Automatic database merging:

When multiple database files are provided as input, rocpd convert automatically manages the database file count to stay within SQLite3’s attach limit:

  • SQLite3 has a maximum limit of 10 attached databases.

  • If the number of input databases exceeds the automerge limit (default: 1), the tool automatically merges them into a temporary .rpdb package.

  • The automerge limit can be controlled with the --automerge-limit parameter (max: 8, which is conservatively set below the SQLite3’s limit of 10).

  • For an explicit control over merging, use the rocpd merge or rocpd package commands before analysis (see Managing multiple database files with rocpd).

Example with automerge control:

# To allow up to 4 databases without automatic merging.
rocpd convert -i db1.db db2.db db3.db db4.db --automerge-limit 4 -f pftrace

Example with automerge control using an index.yaml package file:

# To avoid automerge and use the index.yaml package file to attach databases instead:
rocpd convert -i index.yaml --automerge-limit 6 -f csv

Automatic merging in action:

When multiple databases exceed the automerge limit, you’ll see an output similar to the following:

$ rocpd convert -i database1.db database2.db -f pftrace
Found 2 database files.
More than 1 database files found. It is recommended to merge and package databases
Original number of DBs: 2, Target number of DBs to merge per batch: 2
Adding database1.db
Tables found: 20
Adding database2.db
Tables found: 20
Merge completed successfully! Output saved to: rocpd-20260205-011104.rpdb/merged_db_0_e7e1a495-8185-4875-bdcd-6570e9b2fa81.db
Time: 1.02 sec
Reduced to 1 database files.
Converting database(s) to pftrace format:
...
Done. Exiting...

In the preceding output, the rocpd convert command automatically:

  • Detected 2 input databases exceeding the limit (default: 1).

  • Created a timestamped .rpdb folder (rocpd-20260205-011104.rpdb).

  • Merged the databases to stay within the SQLite3’s permissible attach limit of 10.

  • Proceeded with the requested conversion operation.

The merged database remains in the .rpdb folder for future use, eliminating the need to re-merge for subsequent operations.

For more information about explicit database merging and packaging, see Managing multiple database files with rocpd.

Dedicated conversion tools#

ROCprofiler-SDK provides specialized conversion utilities for efficient format-specific operations. The following tools offer streamlined interfaces for single-format conversions, which are particularly useful in automated workflows and scripts.

rocpd2csv - CSV export tool#

Purpose: To convert rocpd SQLite3 databases to CSV format for spreadsheet analysis and data processing workflows.

Location: /opt/rocm/bin/rocpd2csv

Syntax:

rocpd2csv -i INPUT [INPUT ...] [OPTIONS]

Key features:

  • Structured data export: Converts hierarchical database content to tabular CSV format.

  • Multidatabase support: Aggregates data from multiple database files into a unified CSV output.

  • Time window filtering: Applies temporal filters to limit exported data range.

  • Configurable output: Facilitates customized output file naming and directory structure.

Usage examples:

# Basic CSV conversion
rocpd2csv -i profile_data.db

# Convert multiple databases with custom output path
rocpd2csv -i db1.db db2.db db3.db -d ~/analysis_output/ -o combined_profile

# Apply time window filtering (export middle 70% of execution)
rocpd2csv -i large_profile.db --start 15% --end 85%

# Keep the input read-only while disassembling PC samples
rocpd2csv -i pc_profile.db --no-cache-disassembly

rocpd2csv accepts the same ISA decode options as rocpd convert (--no-cache-disassembly and --codeobj-path-prefix-map). See PC sampling data (schema 3.1.0).

Common output files:

  • out_hip_api_trace.csv: HIP API call trace data.

  • out_kernel_trace.csv: GPU kernel execution information.

  • out_counter_collection.csv: Hardware performance counter data.

  • out_pc_sampling_host_trap_trace.csv and out_pc_sampling_stochastic_trace.csv: PC samples, one row per sample. The leading Guid column identifies the recording, because local IDs such as Dispatch_Id repeat across merged recordings.

rocpd2otf2 - OTF2 export tool#

Purpose: To generate OTF2 files for high-performance trace analysis using tools such as Vampir, Tau, and Score-P viewers.

Location: /opt/rocm/bin/rocpd2otf2

Syntax:

rocpd2otf2 -i INPUT [INPUT ...] [OPTIONS]

Key features:

  • HPC-standard format: Produces traces compatible with scientific computing analysis tools.

  • Hierarchical timeline: Preserves process, thread, or queue relationships in trace structure.

  • Scalable storage: Efficient binary format for large-scale profiling data.

  • Agent indexing: Configurable GPU agent indexing strategies (absolute, relative, or type-relative).

Usage examples:

# Generate OTF2 trace archive
rocpd2otf2 -i gpu_workload.db

# Multi-process trace with custom indexing
rocpd2otf2 -i mpi_rank_*.db --agent-index-value type-relative -o mpi_trace

# Time-windowed trace export
rocpd2otf2 -i long_execution.db --start-marker "computation_begin" --end-marker "computation_end"

Output structure:

  • trace.otf2: Main trace archive containing timeline data.

  • trace.def: Trace definition file with metadata.

  • Supporting files for multistream trace data.

rocpd2pftrace - Perfetto trace export#

Purpose: To convert rocpd databases to Perfetto protocol buffer format for interactive visualization using the Perfetto UI.

Location: /opt/rocm/bin/rocpd2pftrace

Syntax:

rocpd2pftrace -i INPUT [INPUT ...] [OPTIONS]

Key features:

  • Interactive visualization: Optimized for modern web-based trace viewers.

  • Real-time analysis: Supports streaming analysis workflows.

  • GPU timeline integration: Specialized visualization of GPU execution patterns.

  • Configurable backend: Supports both in-process and system-wide tracing backends.

Backend configuration options:

# In-process backend (default)
rocpd2pftrace -i profile.db --perfetto-backend inprocess

# System-wide tracing backend
rocpd2pftrace -i system_profile.db --perfetto-backend system \
                --perfetto-buffer-size 64MB --perfetto-shmem-size-hint 32MB

Buffer management options:

# Ring buffer mode (overwrites old data)
rocpd2pftrace -i continuous_profile.db --perfetto-buffer-fill-policy ring_buffer

# Discard mode (stops recording when full)
rocpd2pftrace -i bounded_profile.db --perfetto-buffer-fill-policy discard

Usage examples:

# Basic Perfetto trace generation
rocpd2pftrace -i application.db

# High-throughput configuration
rocpd2pftrace -i heavy_workload.db --perfetto-buffer-size 128MB \
                --perfetto-buffer-fill-policy ring_buffer

# Multi-queue analysis
rocpd2pftrace -i multi_stream.db --group-by-queue -o queue_analysis

Visualization workflow:

Follow these steps to visualize traces on Perfetto:

  1. Generate .perfetto-trace file using rocpd2pftrace.

  2. Open https://ui.perfetto.dev in web browser.

  3. Load generated trace file for interactive analysis.

rocpd2summary - statistical analysis tool#

Purpose: To generate comprehensive statistical summaries and performance analysis reports from rocpd profiling data.

Location: /opt/rocm/bin/rocpd2summary

Syntax:

rocpd2summary -i INPUT [INPUT ...] [OPTIONS]

Key features:

  • Multiformat output: Supports console, CSV, HTML, JSON, Markdown, and PDF report generation.

  • Comprehensive statistics: Statistics include kernel execution times, API call frequencies, and memory transfer analysis.

  • Domain-specific analysis: Generates separate summaries for HIP, ROCr, Markers, and other trace domains.

  • Rank-based analysis: Per-process and per-rank performance breakdowns for MPI applications.

  • Configurable scope: Selective inclusion or exclusion of analysis categories.

Output format options:

# Console output (default)
rocpd2summary -i profile.db

# CSV format for data analysis
rocpd2summary -i profile.db --format csv -o performance_metrics

# HTML report with visualization
rocpd2summary -i profile.db --format html -d ~/reports/

# Multiple output formats
rocpd2summary -i profile.db --format csv html json

Analysis categories:

Here is how you can generate summary for specific trace domains:

# By default, all available domains will be processed and you can identify which domain regions are included in your profiled data.
rocpd2summary -i profile.db

# Only include HIP and HSA regions to speed up analysis and skip others.
rocpd2summary -i profile.db --region-categories HIP HSA

# Exclude all domain categories so that only the kernels and memory copies are analyzed, to speed up analysis.
rocpd2summary -i profile.db --region-categories NONE

Advanced analysis options:

The following commands demonstrate the usage of advanced analysis options such as domain-summary, summary-by-rank, and start/end:

# Include domain-specific statistics
rocpd2summary -i multi_gpu.db --domain-summary

# Per-rank analysis for MPI applications
rocpd2summary -i mpi_profile_*.db --summary-by-rank --format html

# Time-windowed summary analysis
rocpd2summary -i long_run.db --start 25% --end 75% --format csv

Report content:

  • Kernel statistics: Execution time distributions, call frequencies, and grid or block sizes.

  • API timing: HIP or HSA function call durations and frequencies.

  • Memory analysis: Transfer patterns, bandwidth utilization, and allocation statistics.

  • Device utilization: GPU occupancy patterns and idle time analysis.

  • Synchronization overhead: Barrier and synchronization point analysis.

Output files:

  • kernels_summary.{format} - GPU kernel execution summary.

  • hip_summary.{format} - HIP API call statistics.

  • hsa_summary.{format} - HSA runtime API analysis.

  • memory_summary.{format} - Memory operation statistics.

  • markers_summary.{format} - Marker event analysis.

Generating performance summary using rocpd#

The rocpd summary command provides statistical analysis and performance summaries, similar to the summary functionality available in rocprofv3. This command generates comprehensive reports from rocpd database files, offering the same analytical capabilities that were previously available through rocprofv3 --summary but on the structured database format.

Purpose: To generate statistical summaries and performance reports from rocpd database files, providing similar functionality as rocprofv3 builtin summary capabilities.

Location: /opt/rocm/bin/rocpd summary

Syntax:

rocpd summary -i INPUT [INPUT ...] [OPTIONS]

Key features:

  • Compatible analysis: Provides the same summary statistics and reports as rocprofv3 --summary.

  • Database-driven: Operates on structured rocpd database files for consistent and reproducible analysis.

  • Multidatabase aggregation: Combines and analyzes data from multiple profiling sessions, ranks, or nodes in a single operation. This is further explained in Multidatabase summary analysis.

  • Comparative analysis: Uses --summary-by-rank to compare performance across different ranks, nodes, or execution contexts.

  • Flexible output: Generates summaries in multiple formats such as console, CSV, HTML, and JSON.

  • Selective reporting: Allows generating reports based on specific performance domains and categories.

Multidatabase summary analysis#

The rocpd summary command excels at aggregating multiple database files, providing capabilities not available with single-session analysis.

Here are the benefits of using rocpd summary for multidatabase summary:

  • Unified summary reports:

    # Aggregates multiple databases into a single comprehensive summary.
    rocpd summary -i session1.db session2.db session3.db --format html -o unified_summary
    
    # Combines all MPI rank databases for an overall application analysis.
    rocpd summary -i rank_*.db -f csv -o mpi_application_summary
    
    # Time-series aggregation across multiple profiling runs.
    rocpd summary -i daily_profile_*.db -f json -o weekly_performance_trends
    
    # Generates summary from .rpdb package.
    rocpd summary -i simulation_run_001.rpdb -f html -o mpi_performance_summary
    
  • Rankwise comparative analysis:

    The --summary-by-rank option provides detailed comparative analysis, helping you to identify performance variations, load balancing issues, and optimization opportunities across different execution contexts:

    # Compares performance across MPI ranks.
    rocpd summary -i rank_0.db rank_1.db rank_2.db rank_3.db --summary-by-rank -f html -o rank_comparison
    
    # Analyzes multi-node performance characteristics.
    rocpd summary -i node_*.db --summary-by-rank -f csv -o node_performance_analysis
    
    # Compares GPU device performance in multi-GPU applications.
    rocpd summary -i gpu_0.db gpu_1.db gpu_2.db gpu_3.db --summary-by-rank -f json -o gpu_scaling_analysis
    

Use cases for multidatabase summary analysis#

  • MPI application performance analysis:

    # Profile distributed MPI application
    mpirun -np 8 rocprofv3 --hip-trace --output-format rocpd -- mpi_simulation
    
    # Generate unified summary for overall application performance
    rocpd summary -i results_rank_*.db --format html -o application_overview
    
    # Identify load balancing issues with rank-by-rank comparison
    rocpd summary -i results_rank_*.db --summary-by-rank --format csv -o load_balance_analysis
    
  • Multi-GPU scaling studies:

    # Profile scaling from 1 to 4 GPUs using HIP_VISIBLE_DEVICES.
    HIP_VISIBLE_DEVICES=0 rocprofv3 --hip-trace --output-format rocpd -o scaling_1gpu -- gpu_benchmark
    HIP_VISIBLE_DEVICES=0,1 rocprofv3 --hip-trace --output-format rocpd -o scaling_2gpu -- gpu_benchmark
    HIP_VISIBLE_DEVICES=0,1,2,3 rocprofv3 --hip-trace --output-format rocpd -o scaling_4gpu -- gpu_benchmark
    
    # Aggregate scaling analysis
    rocpd summary -i scaling_*.db --format html -o gpu_scaling_summary
    
    # Compare efficiency across different GPU counts
    rocpd summary -i scaling_*.db --summary-by-rank --format json -o scaling_efficiency
    
  • Performance regression testing:

    # Profile baseline and optimized versions
    rocprofv3 --hip-trace --output-format rocpd -o baseline.db -- application_v1
    rocprofv3 --hip-trace --output-format rocpd -o optimized.db -- application_v2
    
    # Generate unified performance comparison
    rocpd summary -i baseline.db optimized.db --summary-by-rank --format html -o regression_analysis
    
  • Cross-platform performance comparison:

    # Profile on different hardware platforms
    rocprofv3 --hip-trace --output-format rocpd -o platform_A.db -- benchmark
    rocprofv3 --hip-trace --output-format rocpd -o platform_B.db -- benchmark
    
    # Compare platform performance characteristics
    rocpd summary -i platform_*.db --summary-by-rank --format csv -o platform_comparison
    
  • Domain-specific reporting:

    # Cross-rank summary for MPI applications with domain focus
    rocpd summary -i rank_*.db --summary-by-rank --region-categories KERNEL HIP --format html
    
  • Time-windowed analysis:

    rocpd summary -i profile_*.db --start 25% --end 75% --summary-by-rank
    
  • Domain-specific comparative analysis:

    rocpd summary -i node_*.db --domain-summary --summary-by-rank --region-categories HIP ROCR
    

In summary, here is how the output generated using rocpd summary enhances performance analysis:

  • Unified summaries: Provides aggregate statistics across all input databases, showing combined performance metrics.

  • Rankwise summaries: Generates separate statistical reports for each input database, enabling direct comparison of performance characteristics.

  • Comparative metrics: Highlights performance variations, identifies outliers, and reveals load balancing opportunities.

Integration with rocprofv3 workflow#

The rocpd summary command maintains full compatibility with rocprofv3 summary analysis, while extending capabilities to multidatabase scenarios. Users familiar with rocprofv3 --summary will find identical statistical outputs and report formats when using rocpd summary on database files, with the added benefit of cross-session analysis capabilities.

Automatic database merging:

When multiple database files are provided as input, rocpd summary automatically manages database file counts to stay within the SQLite3’s attach limit:

  • SQLite3 has a maximum limit of 10 attached databases.

  • If the number of input databases exceeds the automerge limit (default: 1), the tool automatically merges them into a temporary .rpdb package.

  • The automerge limit can be controlled with the --automerge-limit parameter (max: 8, which is conservatively set below SQLite3’s limit of 10).

  • For an explicit control over merging, use the rocpd merge or rocpd package commands before the analysis (see Managing multiple database files with rocpd).

Example with automerge control:

# Allows up to 4 databases without automatic merging.
rocpd summary -i db1.db db2.db db3.db db4.db --automerge-limit 4 -f html

For more information about explicit control over database organization, see Managing multiple database files with rocpd.

Aggregating multiprofiling data using rocpd#

One of the key advantages of the rocpd format is its ability to aggregate and analyze data from multiple profiling sessions, ranks, or nodes within a unified framework. This capability enables comprehensive analysis workflows, which was not possible with earlier output formats.

Here are the use cases of data aggregation using rocpd:

  • Multidatabase analysis capabilities.

    Unlike the Perfetto output format used in earlier versions, rocpd databases can be seamlessly combined for cross-session analysis:

    # Aggregate analysis across multiple profiling sessions
    rocpd query -i session1.db session2.db session3.db \
                --query "SELECT name, AVG(duration) FROM kernels GROUP BY name"
    
    # Cross-rank performance comparison for MPI applications
    rocpd summary -i rank_0.db rank_1.db rank_2.db rank_3.db --summary-by-rank
    
    # Multi-node scaling analysis
    rocpd query -i node_*.db \
                --query "SELECT COUNT(*) as total_kernels, SUM(duration) as total_time FROM kernels"
    
  • Distributed computing workflows.

    rocpd can effectively aggregate data in a distributed computing environment. Here are the examples for single and multi-GPU analysis:

    • MPI application analysis:

      # Profile MPI application across multiple ranks
      mpirun -np 4 rocprofv3 --hip-trace --output-format rocpd -- mpi_application
      
      # Generate aggregated performance summary
      rocpd summary -i results_rank_*.db --summary-by-rank --format html -o mpi_performance_report
      
      # Analyze load balancing across ranks
      rocpd query -i results_rank_*.db \
                  --query "SELECT pid, COUNT(*) as kernel_count, AVG(duration) as avg_duration FROM kernels GROUP BY pid"
      
    • Multi-GPU scaling analysis:

      # Profile application with multiple GPU devices using HIP_VISIBLE_DEVICES for making the GPUs visible to the application.
      HIP_VISIBLE_DEVICES=0,1,2,3 rocprofv3 --hip-trace --output-format rocpd -o multi_gpu -- multi_gpu_app
      
      # Aggregate device utilization analysis.
      rocpd query -i multi_gpu*.db \
                  --query "SELECT agent_abs_index as device_id, COUNT(*) as operations, SUM(duration) as total_time FROM kernels GROUP BY device_id"
      
      # Cross-device performance comparison
      rocpd summary -i multi_gpu_results.db --domain-summary
      
  • Temporal aggregation.

    • Time-series analysis:

      # Collect profiles over time for performance monitoring
      for hour in {1..24}; do
          rocprofv3 --hip-trace --output-format rocpd -o "profile_hour_$hour.db" -- application
      done
      
      # Analyze performance trends over time
      rocpd query -i profile_hour_*.db \
                  --query "SELECT AVG(duration) as avg_kernel_time, COUNT(*) as kernel_count FROM kernels" \
                  --format csv -o performance_trends
      
    • Comparative analysis:

      # Compare baseline vs optimized performance
      rocpd query -i baseline.db optimized.db \
                  --query "SELECT kernel, AVG(duration) as avg_time FROM kernels GROUP BY name ORDER BY avg_time DESC"
      
      # Generate comparative summary reports
      rocpd summary -i baseline.db optimized.db --format html -o comparison_report
      

Here are the benefits of data aggregation using rocpd:

  • Unified analysis: Combines data from different execution contexts, hardware configurations, and time periods.

  • Scalability insights: Analyzes performance scaling across multiple nodes, ranks, or GPU devices.

  • Trend analysis: Tracks performance evolution over time or across different software versions.

  • Load balancing: Identifies performance bottlenecks and load distribution issues in distributed applications.

  • Cross-platform comparison: Compares performance across different hardware platforms using unified database schema.

The aggregation capabilities of rocpd format enable sophisticated analysis workflows that provide deeper insight into application performance characteristics across diverse computing environments.

Managing multiple database files with rocpd#

When working with distributed profiling sessions, multi-rank MPI applications, or multi-node GPU workloads, rocpd provides specialized tools for managing and consolidating multiple database files. The rocpd merge and rocpd package commands streamline database management, enabling efficient organization and analysis of large-scale profiling datasets.

rocpd merge - Database merging tool#

Purpose: To merge multiple SQLite3 databases into a single unified database file for simplified analysis and reduced file management overhead.

Location: /opt/rocm/bin/rocpd merge

Syntax:

rocpd merge -i INPUT [INPUT ...] [OPTIONS]

Key features:

  • Database consolidation: Combines multiple rocpd databases into a single unified database file.

  • Schema integrity: Validates before merging, that all input databases share the same schema version.

  • Comprehensive merging: Preserves all database objects including tables, views, indexes, and triggers.

  • Data aggregation: Creates UNION views that automatically aggregate data from all merged sources.

  • Integrity handling: Re-enables SQLite foreign key enforcement for subsequent operations on the merged database.

Command-line options:

The following table provides a detailed listing of the rocpd merge command-line options:

Category Option Description
Required arguments -i INPUT [INPUT ...], --input INPUT [INPUT ...] Specifies input database file paths. Accepts multiple SQLite3 database files separated by whitespace. Supports wildcard patterns and .rpdb folder inputs.
Output configuration -o OUTPUT_FILE``, --output-file OUTPUT_FILE Sets the base output filename. Default value: merged. The .db extension is added automatically if not provided.
-d OUTPUT_PATH``, --output-path OUTPUT_PATH Specifies the output directory path. Default value: ./rocpd-output-data.
Command-line help -h, --help Displays comprehensive command syntax, parameter descriptions, and usage examples.

Usage examples:

  • Basic database merge:

    # Merges three databases into a single file.
    rocpd merge -i db0.db db1.db db2.db
    
    # Output: ./rocpd-output-data/merged.db
    
  • Merge with custom output location:

    # Merges databases with custom output directory and filename.
    rocpd merge -i db0.db db1.db db2.db -d merged3DBs -o combined_results
    
    # Output: merged3DBs/combined_results.db
    
  • Merge multi-node profiling data:

    # Consolidates all databases from a node directory.
    rocpd merge -i node0/*.db -d node0_output -o largeMerged
    
    # Output: node0_output/largeMerged.db
    
  • Merge MPI rank databases:

    # Merges all rank databases from distributed MPI run
    rocpd merge -i results_rank_*.db -d mpi_merged -o unified_mpi_profile
    
    # Output: mpi_merged/unified_mpi_profile.db
    

Warning

Consider alternatives to merging large databases.

Merging multiple large databases creates a single, very large output file that might be difficult to manage, transfer, or analyze. For large profiling datasets, consider these alternatives:

  • Use rocpd package: Package databases with metadata files that reference them in their current locations, avoiding the creation of a single large file.

  • Selective merging: Merge only subsets of databases by node or rank, then package the merged subsets.

  • Direct analysis: Many rocpd commands such as convert, query, or summary can work directly with multiple database files or .rpdb packages without requiring a merge operation.

Example - Package instead of merge for large datasets:

# Instead of merging, which creates one large file:
rocpd merge -i node*/rank*.db -o huge_merged.db

# Package them to reference in-place (no large file created):
rocpd package -i node*/rank*.db -d my_large_dataset

# Then analyze the package:
rocpd summary -i my_large_dataset.rpdb -f html --automerge-limit 8

Use cases:

  • MPI application consolidation: Merge databases from all MPI ranks into a single unified database for comprehensive performance analysis.

  • Multi-node profiling: Consolidate profiling data collected across multiple compute nodes for centralized analysis.

  • Database file reduction: Reduce the number of database files when working with analysis tools having file count limitations.

  • Simplified analysis: Create a single database file for easier sharing, archiving, or integration with external analysis tools.

Important considerations:

  • Schema version compatibility: All input databases must have matching schema versions. Merging fails if version mismatches are detected.

  • Table name uniqueness: The merge operation assumes globally unique table names across all input databases.

  • Merged database size: The output database size equals the sum of all input database sizes plus index overhead.

  • Large dataset management: For very large databases, consider using rocpd package to avoid creating a single unwieldy file.

rocpd package - database packaging tool#

Purpose: To create organized database collections with metadata files, enabling efficient management and distribution of multiple profiling databases.

Location: /opt/rocm/bin/rocpd package

Syntax:

rocpd package -i INPUT [INPUT ...] [OPTIONS]

Key features:

  • Metadata generation: Creates .rpdb folders consisting of index.yaml metadata files that reference database collections.

  • Consolidation support: Optionally moves or copies database files into a centralized .rpdb folder structure.

  • Flexible input handling: Accepts database files, directories, wildcard patterns, and existing .rpdb folders as input.

  • In-place referencing: Can create metadata files that reference databases in their original locations without moving them.

Command-line options:

The following table provides a detailed listing of the rocpd package command-line options:

Purpose Option Description
Required arguments -i INPUT [INPUT ...], --input INPUT [INPUT ...] Specifies paths to the input database files or directories. Supports multiple inputs, wildcard patterns (such as *.db), directories containing databases, and existing .rpdb folders.
Package configuration -c, --consolidate Consolidates database files by moving them into the output ``.rpdb`` folder and generates metadata file.
--copy When used with --consolidate, copies database files instead of moving them.
-d OUTPUT_PATH, --output-path OUTPUT_PATH Specifies the output folder name. Without --consolidate, creates metadata in current directory. With --consolidate, creates a .rpdb folder with the specified name.
Command-line help -h, --help Displays comprehensive command syntax, parameter descriptions, and usage examples.

The .rpdb package format:

The .rpdb (ROCProfiler database) package is a standardized folder structure for organizing multiple profiling databases:

  • Folder structure: A directory with the .rpdb extension consisting of database files and metadata.

  • Metadata file: Consists of an index.yaml file in YAML format with database inventory and configuration.

  • Portability: The entire .rpdb folder can be moved, archived, or shared as a self-contained profiling dataset.

  • Tool integration: .rpdb folders are recognized as valid input by all rocpd commands such as convert, query, summary, or merge.

Metadata file structure (index.yaml):

rocprofiler-sdk:
  rocpd:
    rocpd_package_version: "1.0"
    path: "."
    files:
      - "database1.db"
      - "database2.db"
      - "database3.db"

Note

Advantages of using rocpd package over rocpd merge:

  • Large databases: Packaging avoids creating a single large merged file while maintaining organized access to all the data.

  • Distributed storage: When databases reside on different folders within the same filesystem, packaging can reference them in-place.

  • Iterative analysis: Package databases once, then run multiple analysis operations without repeated merging overhead.

  • Flexible organization: Easily add or remove databases from a package by updating the metadata file.

Usage examples:

  • Create metadata file for in-place databases:

    # Index databases without moving them
    rocpd package -i node0/db0.db node1/db1.db node2/db2.db
    
    # Output: ./index.yaml (references databases in original locations)
    
  • Package and consolidate databases:

    # Copy all databases into a new .rpdb folder
    rocpd package -i node0/db0.db node1/db1.db node2/db2.db \
                   -d my_MPI_run_1 --consolidate --copy
    
    # Output: my_MPI_run_1.rpdb/ folder containing copies of all databases and index.yaml
    
  • Append databases to an existing package:

    # Add additional databases to existing .rpdb folder
    rocpd package -i my_MPI_run_1.rpdb node5/db5.db \
                   -d my_MPI_run_1_append_5 --consolidate --copy
    
    # Output: my_MPI_run_1_append_5.rpdb/ folder with expanded database collection
    
  • Consolidate with move operation:

    # Move databases into .rpdb folder (original files are moved, not copied)
    rocpd package -i my_MPI_run_1.rpdb node7/db7.db \
                   -d my_MPI_run_1 --consolidate
    
    # Output: my_MPI_run_1.rpdb/ folder with all databases moved into it
    
  • Package databases from a directory:

    # Package all databases from a directory
    rocpd package -i ./profiling_results/ -d archived_run --consolidate --copy
    
    # Output: archived_run.rpdb/ folder with all .db files from profiling_results/
    

Use cases:

  • MPI profiling organization: Package databases from all MPI ranks into a single portable .rpdb folder for easy distribution and analysis.

  • Archival and backup: Create self-contained .rpdb packages for long-term storage or sharing with collaborators.

  • Distributed profiling management: Organize databases from multi-node runs with clear metadata tracking.

  • Analysis workflow simplification: Create .rpdb packages that can be directly consumed by rocpd convert, query, and summary commands.

  • Database consolidation: Reduce filesystem clutter by collecting scattered database files into organized .rpdb folders.

Integration with other rocpd commands#

The rocpd merge and rocpd package commands integrate seamlessly with other rocpd functionality, enabling sophisticated profiling workflows.

Using .rpdb folders as input:

All rocpd commands accept .rpdb folders as input, automatically processing the metadata and loading referenced databases:

# Convert .rpdb package to Perfetto format
rocpd convert -i my_profiling_data.rpdb -f pftrace

# Generate summary from packaged databases
rocpd summary -i archived_run.rpdb -f html -o performance_report

# Query packaged databases
rocpd query -i mpi_results.rpdb --query "SELECT * FROM top_kernels LIMIT 10"

Automatic merging in analysis workflows:

The rocpd convert, query, and summary commands automatically merge databases when the file count exceeds optimal thresholds:

# Convert many databases - automatic merging if count exceeds threshold
rocpd convert -i node_*/rank_*.db -f pftrace

# The tool automatically merges databases into manageable batches for optimal performance

Complete profiling and analysis workflow:

Here is an example workflow demonstrating merge and package integration:

# Step 1: Profile distributed MPI application (8 ranks)
mpirun -np 8 rocprofv3 --hip-trace --output-format rocpd -o results_rank_%rank% -- mpi_simulation

# Step 2: Package all rank databases with consolidation
rocpd package -i results_rank_*.db -d simulation_run_001 --consolidate --copy

# Step 3: Generate comprehensive summary from package
rocpd summary -i simulation_run_001.rpdb --summary-by-rank -f html \
              -o mpi_performance_analysis

# Step 4: Convert to Perfetto for visualizing the data
rocpd convert -i simulation_run_001.rpdb -f pftrace

# Step 5: Perform custom analysis with SQL queries
rocpd query -i simulation_run_001.rpdb \
            --query "SELECT pid, COUNT(*) as kernel_count, \
                     AVG(duration) as avg_duration \
                     FROM kernels GROUP BY pid" \
            --format csv -o rank_comparison

Alternative workflow with merge:

# Step 1: Profile distributed application
mpirun -np 4 rocprofv3 --hip-trace --output-format rocpd -o results_rank_%rank% -- gpu_workload

# Step 2: Merge all rank databases into single unified database
rocpd merge -i results_rank_*.db -d merged_results -o unified_profile

# Step 3: Analyze merged database (simpler than handling multiple files)
rocpd convert -i merged_results/unified_profile.db -f pftrace csv

# Step 4: Generate unified summary
rocpd summary -i merged_results/unified_profile.db -f html

Combining merge and package:

# Merge subsets of databases, then package the merged results

# Merge databases by node
rocpd merge -i node0/rank_*.db -d merged_by_node -o node0_merged
rocpd merge -i node1/rank_*.db -d merged_by_node -o node1_merged
rocpd merge -i node2/rank_*.db -d merged_by_node -o node2_merged

# Package the node-level merged databases
rocpd package -i merged_by_node/*.db -d cluster_profiling_data \
               --consolidate --copy

# Analyze the packaged multi-node data
rocpd summary -i cluster_profiling_data.rpdb --summary-by-rank \
              -f html -o cluster_performance_report

Benefits of integrated workflows:

  • Simplified file management: Reduce complexity of handling dozens or hundreds of database files.

  • Portable analysis: Create self-contained .rpdb packages for easy sharing and archival.

  • Flexible analysis: Choose between merged single-database or packaged multi-database approaches based on analysis requirement.

  • Scalable processing: Automatic merging ensures optimal performance regardless of database file count.

  • Reproducible workflows: Metadata files document database provenance and organization for reproducible analysis.

Tool integration and workflow examples#

The following examples demonstrate how to use the rocpd conversion tools for automated performance monitoring and multiformat analysis.

  • Multiformat analysis pipeline:

    # Generate all analysis formats for comprehensive review
    rocpd2csv -i profile.db -o analysis_data
    rocpd2summary -i profile.db --format html -o performance_report
    rocpd2pftrace -i profile.db -o interactive_trace
    
  • Automated performance monitoring:

    #!/bin/bash
    # Performance analysis automation script
    
    PROFILE_DB="$1"
    OUTPUT_DIR="analysis_$(date +%Y%m%d_%H%M%S)"
    
    mkdir -p "$OUTPUT_DIR"
    
    # Generate CSV data for automated analysis
    rocpd2csv -i "$PROFILE_DB" -d "$OUTPUT_DIR" -o raw_data
    
    # Create summary reports
    rocpd2summary -i "$PROFILE_DB" --format csv html \
                    -d "$OUTPUT_DIR" -o performance_summary
    
    # Generate interactive trace for detailed investigation
    rocpd2pftrace -i "$PROFILE_DB" -d "$OUTPUT_DIR" -o interactive_trace
    

Executing SQL queries with rocpd#

The rocpd query command provides comprehensive SQL-based analysis capabilities for exploring and extracting data from rocpd databases. This command enables custom analysis workflows and automated reporting for GPU profiling data analysis, and integration with external analysis pipelines.

Purpose: To execute custom SQL queries against rocpd databases with support for multiple output formats, automated reporting, and Email delivery.

Location: /opt/rocm/bin/rocpd query

Syntax:

rocpd query -i INPUT [INPUT ...] --query "SQL_STATEMENT" [OPTIONS]

Key features:

  • Standard SQL support: Supports full SQLite3 SQL syntax including JOINs, aggregate functions, and complex WHERE clauses.

  • Multidatabase aggregation: Facilitates queries across multiple database files as a unified virtual database.

  • Multiple output formats: Supports output formats such as console, CSV, HTML, JSON, Markdown, PDF, and interactive dashboards.

  • Script execution: Can execute complex SQL scripts with view definitions and custom functions.

  • Automated reporting: Supports Email delivery with SMTP configuration and attachment management.

  • Time window integration: Allows temporal filtering before query execution.

Database schema and views#

rocpd databases provide the following comprehensive views for analysis. It is generally recommended to build a query using data_views:

  • Core data views:

    -- System and hardware information
    SELECT * FROM rocpd_info_agents;
    SELECT * FROM rocpd_info_node;
    
    -- Kernel execution data
    SELECT * FROM kernels;
    SELECT * FROM top_kernels;
    
    -- API trace information
    SELECT * FROM regions_and_samples WHERE category LIKE 'HIP_%';
    SELECT * FROM regions_and_samples WHERE category LIKE 'RCCL_%';
    
    -- Performance counters
    SELECT * FROM counters_collection;
    
    -- Memory operations
    SELECT * FROM memory_copies;
    SELECT * FROM memory_allocations;
    
    -- Process and thread information
    SELECT * FROM processes;
    SELECT * FROM threads;
    
    -- Marker and region data
    SELECT * FROM regions;
    SELECT * FROM regions_and_samples WHERE category LIKE 'MARKERS_%';
    
  • Summary and analysis views:

    -- Top performing kernels by execution time
    SELECT * FROM top_kernels LIMIT 10;
    
    -- Top Analysis
    SELECT * FROM top;
    
    -- Busy Analysis
    SELECT * FROM busy;
    

rocpd query examples#

The following examples demonstrate the usage of rocpd query for performing some useful operations:

  • Simple data exploration:

    # List available GPU agents
    rocpd query -i profile.db --query "SELECT * FROM rocpd_info_agents"
    
    # Show top 10 longest-running kernels
    rocpd query -i profile.db --query "SELECT name, duration FROM kernels ORDER BY duration DESC LIMIT 10"
    
    # Count total number of kernel dispatches
    rocpd query -i profile.db --query "SELECT COUNT(*) as total_kernels FROM kernels"
    
  • Multidatabase aggregation:

    # Combine data from multiple profiling sessions
    rocpd query -i session1.db session2.db session3.db \
                --query "SELECT pid, COUNT(*) as kernel_count FROM kernels GROUP BY pid"
    
    # Cross-session performance comparison
    rocpd query -i baseline.db optimized.db \
                --query "SELECT name as kernel_name, AVG(duration) as avg_duration FROM kernels GROUP BY kernel_name"
    
  • Query .rpdb package:

    # Execute SQL query against packaged database collection
    rocpd query -i cluster_data.rpdb \
                --query "SELECT * FROM top_kernels LIMIT 10" \
                --format csv -o top_kernels_report
    
  • Advanced analytics:

    # Kernel performance analysis with statistics
    rocpd query -i profile.db --query "
    SELECT
        name as kernel_name,
        COUNT(*) as dispatch_count,
        MIN(duration) as min_duration,
        AVG(duration) as avg_duration,
        MAX(duration) as max_duration,
        SUM(duration) as total_duration
    FROM kernels
    GROUP BY kernel_name
    ORDER BY total_duration DESC"
    
  • Memory transfer analysis:

    # Memory copy analysis by direction
    rocpd query -i profile.db --query "
    SELECT
        name as kernel_name,
        src_agent_type,
        src_agent_abs_index,
        dst_agent_type,
        dst_agent_abs_index,
        COUNT(*) as transfer_count,
        SUM(size) as total_bytes,
        SUM(duration) as total_duration
    FROM memory_copies
    GROUP BY src_agent_abs_index
    ORDER BY total_bytes DESC"
    

Output format options#

rocpd query supports the following output formats:

  • Console output (default):

    # Display results in terminal
    rocpd query -i profile.db --query "SELECT * FROM top_kernels LIMIT 5"
    
  • CSV export for data analysis:

    # Export to CSV file
    rocpd query -i profile.db --query "SELECT * FROM kernels" --format csv -o kernel_analysis
    
    # Specify custom output directory
    rocpd query -i profile.db --query "SELECT * FROM kernels" --format csv -d ~/analysis/ -o kernel_data
    
  • HTML reports:

    # Generate HTML table
    rocpd query -i profile.db --query "SELECT * FROM top_kernels" --format html -o performance_report
    
  • Interactive dashboard:

    # Create interactive HTML dashboard
    rocpd query -i profile.db --query "SELECT * FROM device_utilization" --format dashboard -o utilization_dashboard
    
    # Use custom dashboard template
    rocpd query -i profile.db --query "SELECT * FROM kernels" --format dashboard \
                --template-path ~/templates/custom_dashboard.html -o custom_report
    
  • JSON for programmatic integration:

    # Export structured JSON data
    rocpd query -i profile.db --query "SELECT * FROM counters_collection" --format json -o counter_data
    
  • PDF reports:

    # Generate PDF report with monospace formatting
    rocpd query -i profile.db --query "SELECT name, duration FROM top_kernels" --format pdf -o kernel_report
    

Automatic database merging:

When multiple database files are provided as input, rocpd query automatically manages the database file count to stay within the SQLite3’s attach limit:

  • SQLite3 has a maximum limit of 10 attached databases.

  • If the number of input databases exceeds the automerge limit (default: 1), the tool automatically merges them into a .rpdb package.

  • The automerge limit can be controlled with the --automerge-limit parameter (max: 8, which is conservatively set below the SQLite3’s limit of 10).

  • For explicit control over merging, use the rocpd merge or rocpd package commands before the analysis. For details, see Managing multiple database files with rocpd.

Example with automerge control:

# Allow up to 4 databases without automatic merging
rocpd query -i db1.db db2.db db3.db db4.db --automerge-limit 4 \
            --query "SELECT COUNT(*) FROM kernels"

Example referencing an index.yaml package file with automerge control (don’t automerge, use the package as is):

# Don't automerge, use the index.yaml package file to attach databases as is
rocpd query -i index.yaml --automerge-limit 6 \
            --query "SELECT * FROM top_kernels"

To learn more about database merging and packaging workflows, see Managing multiple database files with rocpd.

Script-based analysis#

You can use rocpd query to execute complex SQL scripts with view definitions and custom analysis logic.

Here is a sample SQL script (analysis.sql):

-- Create temporary views for complex analysis
CREATE TEMP VIEW kernel_stats AS
SELECT
    name as kernel_name,
    COUNT(*) as dispatch_count,
    AVG(duration) as avg_duration,
    STDDEV(duration) as duration_stddev
FROM kernels
GROUP BY kernel_name;

CREATE TEMP VIEW performance_outliers AS
SELECT k.*, ks.avg_duration, ks.duration_stddev
FROM kernels k
JOIN kernel_stats ks ON k.name = ks.name
WHERE ABS(k.duration - ks.avg_duration) > 2 * ks.duration_stddev;

Here is how to execute the preceding script using rocpd query:

# Run script then execute query
rocpd query -i profile.db --script analysis.sql \
            --query "SELECT * FROM performance_outliers" --format html -o outlier_analysis

Time window integration#

Here is how to apply temporal filtering before query execution:

# Query only middle 50% of execution timeline
rocpd query -i profile.db --start 25% --end 75% \
            --query "SELECT COUNT(*) as kernel_count FROM kernels"

# Use marker-based time windows
rocpd query -i profile.db --start-marker "computation_begin" --end-marker "computation_end" \
            --query "SELECT * FROM kernels ORDER BY start_time"

# Absolute timestamp filtering
rocpd query -i profile.db --start 1000000000 --end 2000000000 \
            --query "SELECT * FROM kernels WHERE start_time BETWEEN 1000000000 AND 2000000000"

Automated Email reporting#

rocpd query facilitates sending reports through Email. Here is how you can send reports in different formats and use various Email configuration options:

  • Basic Email delivery:

    # Send CSV report via email
    rocpd query -i profile.db --query "SELECT * FROM top_kernels" --format csv \
                --email-to analyst@company.com --email-from profiler@company.com \
                --email-subject "Weekly Performance Report"
    
  • Advanced Email configuration:

    # Multiple recipients with SMTP authentication
    rocpd query -i profile.db --query "SELECT * FROM device_utilization" --format html \
                --email-to "team@company.com,manager@company.com" \
                --email-from profiler@company.com \
                --email-subject "GPU Utilization Analysis" \
                --smtp-server smtp.company.com --smtp-port 587 \
                --smtp-user profiler@company.com --smtp-password $(cat ~/.smtp_pass) \
                --inline-preview --zip-attachments
    
  • Dashboard Email reports:

    # Send interactive dashboard via email
    rocpd query -i profile.db --query "SELECT * FROM kernels" --format dashboard \
                --template-path ~/templates/executive_summary.html \
                --email-to executives@company.com --email-from profiler@company.com \
                --email-subject "Executive Performance Dashboard" \
                --inline-preview
    

Integration workflows#

  • Automated analysis pipeline:

    #!/bin/bash
    # Automated reporting script
    
    DB_FILE="$1"
    REPORT_DATE=$(date +%Y-%m-%d)
    
    # Generate multiple analysis reports
    rocpd query -i "$DB_FILE" --query "SELECT * FROM top_kernels LIMIT 20" \
                --format html -o "top_kernels_$REPORT_DATE"
    
    rocpd query -i "$DB_FILE" --query "SELECT * FROM memory_copy_summary" \
                --format csv -o "memory_analysis_$REPORT_DATE"
    
    rocpd query -i "$DB_FILE" --query "SELECT * FROM device_utilization" \
                --format dashboard -o "utilization_dashboard_$REPORT_DATE" \
                --email-to team@company.com --email-from automation@company.com
    
  • Performance regression detection:

    # Compare current performance against baseline
    rocpd query -i baseline.db current.db --script performance_comparison.sql \
                --query "SELECT * FROM performance_regression_analysis" \
                --format html -o regression_report \
                --email-to devteam@company.com --email-from ci@company.com \
                --email-subject "Performance Regression Analysis"
    
  • Custom analysis functions:

    rocpd databases support custom SQL functions for advanced analysis:

    # Use built-in rocpd functions
    rocpd query -i profile.db --query "
    SELECT
        name,
        rocpd_get_string(name_id, 0, nid, pid) as full_kernel_name,
        duration
    FROM kernels
    WHERE rocpd_get_string(name_id, 0, nid, pid) LIKE '%gemm%'"
    

rocpd query command-line reference#

The command-line options as displayed using rocpd query --help are listed here:

usage: rocpd query [-h] -i INPUT [INPUT ...] --query QUERY [--script SCRIPT]
                   [--format {console,csv,html,json,md,pdf,dashboard,clipboard}]
                   [-o OUTPUT_FILE] [-d OUTPUT_PATH] [--automerge-limit LIMIT]
                   [--email-to EMAIL_TO] [--email-from EMAIL_FROM]
                   [--email-subject EMAIL_SUBJECT] [--smtp-server SMTP_SERVER]
                   [--smtp-port SMTP_PORT] [--smtp-user SMTP_USER]
                   [--smtp-password SMTP_PASSWORD] [--zip-attachments]
                   [--inline-preview] [--template-path TEMPLATE_PATH]
                   [--start START | --start-marker START_MARKER]
                   [--end END | --end-marker END_MARKER]

The following table provides a detailed listing of the rocpd query command-line options:

Category Option Description
Required arguments -i INPUT [INPUT ...], --input INPUT [INPUT ...] The input database file paths. Multiple databases are merged into a unified view.
--query QUERY The SQL SELECT statement to be executed. Enclose complex queries in quotes.
Query options --script SCRIPT The SQL script file to be executed before running the main query. Useful for creating views and functions.
--format {console,csv,html,json,md,pdf,dashboard,clipboard} The output format. Dashboard format creates interactive HTML reports. The default value is "console".
Output configuration -o OUTPUT_FILE, --output-file OUTPUT_FILE Base filename for exported files.
-d OUTPUT_PATH, --output-path OUTPUT_PATH Output directory path.
--automerge-limit LIMIT Controls the database auto-merge limit. When the number of input databases exceeds this limit, they are automatically merged into a .rpdb package to stay below SQLite3's attach limit of 10. Default value: 1, maximum permissible value: 8.
--template-path TEMPLATE_PATH Jinja2 template file for dashboard format customization.
Email reporting --email-to EMAIL_TO Recipient Email addresses (comma-separated for multiple recipients).
--email-from EMAIL_FROM Sender Email address. This is required when using Email delivery.
--email-subject EMAIL_SUBJECT Email subject line.
--smtp-server SMTP_SERVER, --smtp-port SMTP_PORT SMTP server configuration. The default value is "localhost:25".
--smtp-user SMTP_USER, --smtp-password SMTP_PASSWORD SMTP authentication credentials.
--zip-attachments Bundles all attachments into a single ZIP file.
--inline-preview Embeds HTML reports as Email body content.
Time window filtering --start START, --end END Temporal boundaries using percentage such as 25% or absolute timestamps.
--start-marker START_MARKER, --end-marker END_MARKER Named marker events defining time window boundaries.

rocpd database visualization methods#

You can use the following methods to access the data in the rocpd database:

  • Sql queries (command-line):

    sqlite3 hostname/12345_results.db
    sqlite> .tables # List all tables
    sqlite> SELECT * FROM kernel_dispatch;
    sqlite> SELECT * FROM hip_api_trace WHERE duration > 1000000;
    
  • Python analysis:

    import sqlite3
    conn = sqlite3.connect('12345_results.db')
    df = pd.read_sql_query("SELECT * FROM kernel_dispatch", conn)
    
  • DB browser for SQLite (GUI):

    1. Open .db file in DB browser.

    2. Browse tables, run queries, or export data.

  • Convert and visualize on Perfetto:

    1. Convert into Perfetto format using:

      rocpd convert -i results.db --f pftrace
      
    2. Open in Perfetto UI to see the visual timeline.

PC sampling data (schema 3.1.0)#

PC sampling support was introduced in schema version 3.1.0. When rocprofv3 is run with --pc-sampling-beta-enabled and --output-format rocpd, the following tables are populated.

rocpd_gpu_pc_sample#

One row per collected PC sample. Key columns:

Column

Type

Description

event_id

INTEGER

Foreign key into rocpd_event. The corresponding event’s parent_id points to the dispatch event.

timestamp

BIGINT

Sample timestamp in nanoseconds.

dispatch_id

INTEGER

Dispatch ID of the kernel being sampled.

correlation_id

INTEGER

User-assigned external correlation ID, matching the linked rocpd_event.correlation_id. The internal ID is stored in the linked event’s stack_id.

exec_mask

TEXT

Bitmask of active SIMD lanes at sample time. Stored as a decimal string so the full 64-bit mask is preserved without integer overflow. SQLite integers are signed 64-bit values, with maximum 2^63 - 1 (9223372036854775807), whereas a valid 64-lane mask can reach 2^64 - 1 (18446744073709551615). No mask needs to exceed 2^64 - 1 for TEXT storage to be necessary. Because the column is TEXT, <, >, ORDER BY, MIN, and MAX compare masks as text ('255' > '1000'). For exact equality, use a quoted decimal string, such as exec_mask = '18446744073709551615', or bind a string parameter. Numeric literals above 2^63 - 1 are interpreted as REAL and may not match the stored text. CAST(exec_mask AS INTEGER) orders masks below 2^63 numerically but clamps larger masks, including a full 64-lane mask, to 9223372036854775807.

code_object_id

INTEGER

Identifies the loaded code object containing the sampled instruction. Zero denotes an unknown object. This column deliberately has no foreign-key constraint so samples are retained when code-object metadata is unavailable.

code_object_offset

BIGINT

Byte offset within the code object (the program counter value).

agent_id

INTEGER

GPU agent from the parent dispatch, falling back to recorded code-object metadata when dispatch context is unavailable. NULL when neither is known.

tid

INTEGER

CPU thread that launched the parent kernel dispatch. NULL when dispatch context is unavailable.

wave_issued

INTEGER

Whether the wave issued an instruction (1 = yes, 0 = no). Stochastic only; NULL for host-trap.

inst_type

INTEGER

Instruction type if wave_issued = 1. Stochastic only; NULL for host-trap.

stall_reason

INTEGER

Stall reason if wave_issued = 0. Stochastic only; NULL for host-trap.

wave_count

INTEGER

Total active waves on the compute unit at sample time. Stochastic only; NULL for host-trap.

INTEGER and BIGINT have the same integer affinity in SQLite; BIGINT does not provide unsigned storage. Metadata IDs consistently use INTEGER; timestamps and code-object offsets retain the existing BIGINT declaration.

Execution-mask operations#

TEXT storage preserves every mask bit, but does not make integer operations automatic. SQLite’s bitwise operators and CAST(... AS INTEGER) convert decimal text to a signed 64-bit integer. Masks above 2^63 - 1 are clamped, not reinterpreted as an unsigned bit pattern. For example, casting the full 64-lane mask before counting bits gives 63 active lanes instead of 64.

Standard SQLite has no built-in popcount. A client-provided function must parse the decimal text without losing the high bit. Python integers can represent the full mask; this example registers a connection-local function and works with Python 3.6 and later:

import sqlite3

def mask_popcount(mask):
  if mask is None:
    return None
  return bin(int(mask)).count("1")

with sqlite3.connect("results.db") as connection:
  connection.create_function("mask_popcount", 1, mask_popcount)
  rows = connection.execute(
    "SELECT mask_popcount(exec_mask) AS active_lanes, COUNT(*) AS samples "
    "FROM rocpd_gpu_pc_sample GROUP BY active_lanes"
  ).fetchall()

On Python 3.10 and later, int(mask).bit_count() can replace bin(int(mask)).count("1"). The function above is registered only on that connection; it is not a built-in rocpd query function. For bitwise analysis, likewise parse the original decimal text in an unsigned-capable client rather than casting it to SQLite INTEGER first.

Compared with INTEGER storage, decimal TEXT uses more bytes and requires decimal-to-integer conversion for each evaluated mask. A Python SQL function also incurs SQLite-to-Python call overhead. Thus equal results do not imply equal performance; measure the intended query and workload. Grouping identical exec_mask strings before computing their population counts can reduce conversions when many samples share a mask. These costs are separate from unpacking architecture-specific blob fields or decoding ISA instructions.

rocpd_blob_event, rocpd_info_blob_schema, rocpd_info_blob_field#

Each PC sample also has a companion row in rocpd_blob_event containing a packed binary blob of architecture-specific hardware state. Both rows contain the same event_id and independently reference rocpd_event.id; there is no blob_event_id column. Join the rows on both guid and event_id. The blob layout is self-describing through two metadata tables:

  • rocpd_info_blob_schema: Describes the overall layout (name pc_sample_extdata_hosttrap for host-trap, or pc_sample_extdata_stochastic with an optional _gfx12 or _gfx1250 suffix for stochastic sampling, followed by _no_memory_counters when those samples carry no memory counters; total byte size, byte order little, alignment 1).

  • rocpd_info_blob_field: One row per field with name, byte offset, size, and data type. Consumers can decode the binary without hard-coding struct offsets.

Schema names are internal diagnostic labels, not a versioned architecture or capability API. Consumers must discover fields from rocpd_info_blob_field using (guid, schema_id) instead of inferring memory-counter presence or field lists from a name, suffix, or GPU architecture. Layouts with and without memory counters can coexist for the same architecture, according to each sample’s validity flag.

The blob contains HW ID fields (hw_id_chiplet, hw_id_wave_id, hw_id_simd_id, hw_id_pipe_id, hw_id_cu_or_wgp_id, hw_id_shader_array_id, hw_id_shader_engine_id, hw_id_workgroup_id, hw_id_vm_id, hw_id_queue_id, hw_id_microengine_id) and workgroup coordinates (workgroup_id_x/y/z, wave_in_group). For stochastic sampling, arbiter-state fields (arb_state_issue_* and arb_state_stall_*) are also present. Each boolean is currently stored as a one-byte uint8_t containing 0 or 1, not a packed bit. Bit packing would require additional bit-offset/width metadata and decoder support and is not part of this layout.

The architecture is recovered from the parent dispatch or, without kernel tracing, from the sample’s code-object agent. If neither identifies an agent, the writer uses the architecture only when all enumerated GPUs agree. An ambiguous architecture uses an _unknown_arch layout containing only shared fields; it is not silently classified as GFX9.

Code-object agent attribution#

For a known code object captured by rocprofv3, its load callback records both code_object_id and the SDK agent ID. get_code_object_agent() looks up that recorded ID, and get_agent() supplies the GPU target version and ROCpd node ID. This does not require --kernel-trace. The load metadata is retained when the object unloads; the unload callback flushes pending data rather than deleting the mapping. FILE and MEMORY storage types use the same mapping path. A memory:// URI can prevent later ISA decoding, but does not by itself prevent agent attribution.

The mapping is not unconditional for every sample:

  • code_object_id = 0 is ROCPROFILER_CODE_OBJECT_ID_NONE, not a loaded object. A PC outside the parser’s registered address ranges has no object mapping. If dispatch correlation also fails, the parser reports PCSAMPLE_STATUS_PARSER_ERROR and leaves both the object and dispatch IDs unset. An unmapped PC whose dispatch correlation succeeds is instead marked invalid by the parser. Neither case establishes a missing mapping for a successfully identified code object.

  • A nonzero sample ID with no captured load record, or a recorded agent ID absent from the agent metadata, indicates incomplete or inconsistent attribution data. This is a defensive case, not an expected consequence of disabling kernel tracing, unloading a known object, or using MEMORY storage.

ROCpd does not invent a code-object or agent mapping for such records. If no parent-dispatch or code-object agent is available, agent_id remains NULL. For stochastic samples, a single shared architecture among enumerated GPUs can still select the blob layout without selecting a particular agent. Otherwise the unknown-architecture layout preserves only shared fields. An unsupported GPU target can also select that layout despite having a valid agent mapping; unsupported architecture and missing attribution are distinct conditions.

Joined view#

The persistent pc_samples view joins raw PC samples to their event, blob, agent index, and code-object URI using recording-qualified keys. Its left joins retain samples without dispatch or code-object metadata. It does not unpack blob bytes or decode instructions; use the decoded view below for those fields.

Decoded view#

The rocpd post-processing tools provide a rocpd_gpu_pc_sample_decoded temporary view that automatically unpacks the binary blob fields alongside all columns from rocpd_gpu_pc_sample. The view is created when a RocpdImportData connection is opened: setup_blob_views reads rocpd_info_blob_schema and, for each registered blob schema (keyed by its (guid, schema_id)), builds a generic decoded view over the schema’s source_table (rocpd_gpu_pc_sample here), which setup_isa_decode_views then enriches with the instruction and enum-name columns. The packed blob bytes are only unpacked when the decoded view is queried. Reading rocpd_gpu_pc_sample directly (for example, through sqlite3) does not unpack them, but rocpd query rewrites references to rocpd_gpu_pc_sample to the decoded view (see below). The intermediate rocpd_gpu_pc_sample_blob_decoded temporary view contains the base sample columns and unpacked blob fields, without instruction or enum-name enrichment. If the PC-sampling layer cannot be set up, rocpd prints a warning, does not create rocpd_gpu_pc_sample_decoded, and skips the PC-sampling CSV files; the blob fields remain available through rocpd_gpu_pc_sample_blob_decoded.

The decoded view exposes every column from rocpd_gpu_pc_sample plus one additional column per blob field, including hw_id_chiplet, hw_id_wave_id, hw_id_simd_id, hw_id_pipe_id, hw_id_cu_or_wgp_id, hw_id_shader_array_id, hw_id_shader_engine_id, hw_id_workgroup_id, hw_id_vm_id, hw_id_queue_id, hw_id_microengine_id, wave_in_group, workgroup_id_x, workgroup_id_y, workgroup_id_z, and the architecture’s arb_state_issue_* and arb_state_stall_* fields. GFX9 includes dual_issue_valu and the *_matrix, *_flat, and *_misc arbiters. GFX12 and GFX1250 omit those GFX9-only fields and instead include the *_lds_direct and *_brmsg arbiter variants and, when supplied by the sample, the memory_counter_load, memory_counter_store, memory_counter_bvh, memory_counter_sample, memory_counter_ds, and memory_counter_km fields. GFX1250 additionally exposes sampling_lock_error, memory_counter_async, memory_counter_tensor, and memory_counter_xnack. Architecture-specific or unavailable memory-counter fields are NULL for rows whose blob schema does not contain them. The view additionally exposes the disassembled instruction and instruction_comment text for the sampled program counter and the human-readable inst_type_name and stall_reason_name columns. An empty instruction comment is exposed as NULL, whether the instruction is decoded lazily or read from stored disassembly.

Instruction disassembly#

By default, instruction text is decoded lazily during post-processing. Each time the rocpd_gpu_pc_sample_decoded view is queried, rocpd disassembles the sampled program counters on demand from the referenced code objects and returns the instruction text in the query result.

Important

Disassembly caching is enabled by default (cache_disassembly=True) in the Python API and the rocpd post-processing tools. This applies to existing PC-sampling databases as well as new captures, with no feature-flag gate. A query or conversion that decodes previously unstored instructions can therefore modify its input database at normal Python interpreter shutdown, even when the command is used only for analysis. This is independent of --complete-isa-decode, which persists instruction text during collection.

Write-back is best-effort: it requires an existing rocpd_disassembly_data table and a successful decode from code-object bytes that match a recorded content hash. Caching does not create missing tables or migrate older input schemas. Simply opening a database or reading already-stored instruction text does not add cache rows.

Pass --no-cache-disassembly to rocpd convert, rocpd query, rocpd summary, rocpd2csv, or python -m rocpd.query (or cache_disassembly=False to the RocpdImportData API) to keep the input database read-only, in which case the decoded text is cached only in memory for the lifetime of the process and the .db file is left unchanged.

A code object on disk can change after profiling, for example when the application is rebuilt or --codeobj-path-prefix-map points at a different file. When rocprofv3 decodes a code object during PC sampling with ROCpd output enabled, it records a hash of the code object’s bytes in the content_hash field of rocpd_info_code_object.extdata. Before decoding lazily, rocpd compares the bytes it reads against that hash and reports Decode unavailable for a mismatch rather than returning instructions from a different binary. Code objects without a recorded hash are still decoded, but their instructions are never written back. The internal capture_content_hash gate skips hashing for CSV/JSON-only PC-sampling captures; it does not skip reading the bytes needed for decoding.

Passing --complete-isa-decode to rocprofv3 instead persists the disassembly at collection time into the rocpd_disassembly_data table (deduplicated per code_object_id and code_object_offset). Only the sampled program counters that were successfully resolved to an instruction during collection are stored. A program counter is absent from the table when it could not be disassembled at collection time, for example because the referenced code object was no longer loaded when the disassembly pass ran, or the address did not map to a decodable instruction. When the table is present, the decoded view reads the stored text for the program counters it contains and falls back to on-demand decoding (the default lazy behavior described above) only for those absent program counters. Because the fallback decodes from the same code objects, a query still returns instruction text for those program counters as long as the code objects remain accessible.

By default (unless --no-cache-disassembly or cache_disassembly=False is passed), the rocpd post-processing tools persist instructions that are decoded lazily from code objects that match their recorded hash. Program counters that are disassembled on demand while a query is served (for example by rocpd query or rocpd convert) are written into the same rocpd_disassembly_data table when the process exits, so a later open reads the stored text instead of decoding again. Unlike --complete-isa-decode (which runs at collection time), this persists only the program counters that were actually decoded during post-processing, and only those that resolved to an instruction.

Note

Disassembly write-back is on by default and best-effort:

  • It writes to the database(s) that were opened unless disabled with --no-cache-disassembly (or cache_disassembly=False via the API), which keeps the input read-only. When several inputs are merged automatically, the merged copy is the database that was opened.

  • Rows are written at process exit and routed to each recording’s table by guid, including every recording of a merged database, so the cache benefits the next open rather than the current run.

  • Only program counters that a query actually decodes are cached: the query must project the instruction or instruction_comment columns, the program counter must not already be stored, it must decode successfully, and its code object must match its recorded content hash. Memory-resident code objects that cannot be decoded (see the note below) are therefore not cached.

  • It is skipped silently when the database cannot be written (a read-only file or filesystem, lock contention, a database deleted or replaced after

it was opened, or a pre-3.1.0 database without the

rocpd_disassembly_data table); the database is left untouched.

  • Writes are de-duplicated per guid, code_object_id, and code_object_offset, so the cache never changes query results.

By default the cache is populated whenever a command decodes program counters that are not already stored. For example:

# A query that projects the instruction/comment columns of the decoded view.
rocpd query -i results.db --query "
  SELECT dispatch_id, instruction, instruction_comment
  FROM rocpd_gpu_pc_sample_decoded LIMIT 100"

# A query written against rocpd_gpu_pc_sample is rewritten to the decoded
# view automatically (update_query_for_blob_views), so it decodes -- and
# caches -- the projected program counters too.
rocpd query -i results.db --query "
  SELECT instruction FROM rocpd_gpu_pc_sample LIMIT 100"

# CSV conversion: the PC sampling CSV writers select instruction and
# instruction_comment, so a convert run caches every decoded sample.
rocpd convert -i results.db -f csv

Any direct SELECT of the instruction or instruction_comment columns of rocpd_gpu_pc_sample_decoded has the same effect. In every case the decoded program counters are written to rocpd_disassembly_data when the process exits, so the next time the database is opened the stored text is served through the decoded view instead of being disassembled again. Reading the stored text is unaffected by --no-cache-disassembly – that flag only controls whether new decodes are persisted:

# First run: decodes lazily and, at exit, persists the disassembly.
rocpd convert -i results.db -f csv

# Later runs: the decoded view serves the stored text (no re-decoding);
# --no-cache-disassembly does not affect reading already-stored text.
rocpd query -i results.db --query "
  SELECT instruction FROM rocpd_gpu_pc_sample_decoded LIMIT 100"

Warning

Persisting disassembly grows the input database by roughly one row per distinct sampled program counter that is decoded. This is bounded (only sampled program counters, not whole code objects) but non-zero; pass --no-cache-disassembly (or cache_disassembly=False via the API) to keep the input read-only if you do not want the instruction text written back.

Note

Lazy decoding requires accessible code-object bytes through a local file:// URI. Both storage_type = FILE and storage_type = MEMORY objects can be decoded when such a URI resolves to matching bytes; MEMORY objects must also match the recorded memory_size. PC sampling does not snapshot purely memory-resident objects to disk. Objects available only through a memory:// URI therefore cannot be decoded after the profiled process exits, and the view reports Decode unavailable. To retain their disassembly, pass --complete-isa-decode at collection time; it decodes the live bytes in-process and persists the text into the database.

Moving a recording to another machine#

The database stores code-object metadata (URI, load information, and optional content hash), not complete code-object binaries. ROCpd capture and database merge/package operations do not automatically bundle the referenced binaries. Copying the whole .rocprofv3 directory is therefore not a guarantee that all code objects are present: a file:// URI can point to the application’s executable or a shared library elsewhere, including a byte range within that file.

For lazy decoding on another machine, copy the database and the exact binary files referenced by its sampled code objects. Keep any URI offset/size fragment valid, and pass --codeobj-path-prefix-map to map the recorded directory prefixes to their new locations. The decoder still checks the captured content hash; a rebuilt library is not an equivalent replacement. The referenced objects can be listed without decoding instructions:

SELECT DISTINCT C.guid, C.id, C.storage_type, C.uri
FROM rocpd_info_code_object C
JOIN rocpd_gpu_pc_sample S
  ON S.guid = C.guid AND S.code_object_id = C.id
WHERE C.id != 0;

Instructions already persisted by --complete-isa-decode or a verified lazy decode remain readable from the copied database without the original binaries. This covers only successfully decoded sampled PCs, not the whole code object. Pure memory:// objects have no file to copy in the current implementation; preserving their loaded bytes requires a separate capture-time snapshot feature. storage_type = MEMORY alone does not imply this limitation: an object can still have a usable file:// URI.

Querying PC sampling data#

Note

A recording is the profiling data captured by a single rocprofv3 run (one profiled process). Each run writes its own .db file and stamps every row it produces with a unique guid (a UUID generated once per process), so within one database all rows share the same guid. Merging or packaging databases (for example, from multiple MPI ranks, GPUs, or repeated runs) combines rows from several recordings, and each retains its original guid. The guid is therefore what distinguishes one recording from another after the data has been combined.

dispatch_id is unique only within a single recording (guid). When querying a database merged from multiple recordings, add a guid predicate to any join on dispatch_id (for example ... AND KD.guid = S.guid) so that samples are not associated with dispatches from a different recording.

Each PC sample event’s parent_id in rocpd_event points to the dispatch event for the kernel being sampled when --kernel-trace is active. Its parent_stack_id records that dispatch event’s internal stack_id. Both parent fields are NULL when the dispatch context was not captured:

rocpd query -i results.db --query "SELECT dispatch_id, arb_state_issue_valu FROM rocpd_gpu_pc_sample_decoded LIMIT 10"

Verify parent-dispatch linkage:

rocpd query -i profile.db --query "
SELECT
  COUNT(*) AS total_samples,
  SUM(CASE WHEN E.parent_id IS NOT NULL THEN 1 ELSE 0 END) AS linked_samples
FROM rocpd_gpu_pc_sample S
JOIN rocpd_event E ON E.id = S.event_id AND E.guid = S.guid"

VALU issue rate per kernel:

rocpd query -i profile.db --query "
WITH per_kernel AS (
  SELECT
    KD.guid,
    KD.kernel_id,
    SUM(CASE WHEN S.arb_state_issue_valu = 1 THEN 1 ELSE 0 END) AS issued_valu_samples,
    COUNT(*) AS total_samples
  FROM rocpd_gpu_pc_sample_decoded S
  INNER JOIN rocpd_kernel_dispatch KD ON KD.dispatch_id = S.dispatch_id
    AND KD.guid = S.guid
  GROUP BY KD.guid, KD.kernel_id
)
SELECT
  PK.kernel_id,
  KS.display_name,
  PK.issued_valu_samples,
  PK.total_samples,
  (100.0 * PK.issued_valu_samples) / NULLIF(PK.total_samples, 0) AS issued_valu_pct
FROM per_kernel PK
LEFT JOIN rocpd_info_kernel_symbol KS ON KS.id = PK.kernel_id
  AND KS.guid = PK.guid
ORDER BY PK.issued_valu_samples DESC
LIMIT 20"

Dispatches with highest execution stall back-pressure:

rocpd query -i profile.db --query "
SELECT
  S.guid,
  S.dispatch_id,
  COUNT(*) AS samples,
  SUM(CASE WHEN
        S.arb_state_stall_valu=1 OR S.arb_state_stall_matrix=1 OR
        S.arb_state_stall_lds=1 OR
        S.arb_state_stall_scalar=1 OR S.arb_state_stall_vmem_tex=1 OR
        S.arb_state_stall_flat=1 OR S.arb_state_stall_exp=1 OR
        S.arb_state_stall_misc=1
      THEN 1 ELSE 0 END) AS stall_samples,
  (100.0 * SUM(CASE WHEN
        S.arb_state_stall_valu=1 OR S.arb_state_stall_matrix=1 OR
        S.arb_state_stall_lds=1 OR
        S.arb_state_stall_scalar=1 OR S.arb_state_stall_vmem_tex=1 OR
        S.arb_state_stall_flat=1 OR S.arb_state_stall_exp=1 OR
        S.arb_state_stall_misc=1
      THEN 1 ELSE 0 END)) / NULLIF(COUNT(*), 0) AS stall_pct
FROM rocpd_gpu_pc_sample_decoded S
GROUP BY S.guid, S.dispatch_id
ORDER BY stall_samples DESC, samples DESC
LIMIT 20"

Kernels with highest stall ratio:

This counts a sample as stalled when every execution pipeline failed to make forward progress in the sampled cycle, that is arb_state_issue_PIPE = 0 OR arb_state_stall_PIPE = 1 for all pipes (SIMD latency). It is therefore broader than the back-pressure query above, which counts only samples where a pipeline backpressured an issued instruction (arb_state_stall_PIPE = 1).

rocpd query -i profile.db --query "
WITH per_kernel AS (
  SELECT
    KD.guid,
    KD.kernel_id,
    COUNT(*) AS samples,
    SUM(CASE WHEN
          (S.arb_state_issue_valu=0       OR S.arb_state_stall_valu=1) AND
          (S.arb_state_issue_matrix=0     OR S.arb_state_stall_matrix=1) AND
          (S.arb_state_issue_lds=0        OR S.arb_state_stall_lds=1) AND
          (S.arb_state_issue_scalar=0     OR S.arb_state_stall_scalar=1) AND
          (S.arb_state_issue_vmem_tex=0   OR S.arb_state_stall_vmem_tex=1) AND
          (S.arb_state_issue_flat=0       OR S.arb_state_stall_flat=1) AND
          (S.arb_state_issue_exp=0        OR S.arb_state_stall_exp=1) AND
          (S.arb_state_issue_misc=0       OR S.arb_state_stall_misc=1)
        THEN 1 ELSE 0 END) AS stall_samples
  FROM rocpd_gpu_pc_sample_decoded S
  INNER JOIN rocpd_kernel_dispatch KD ON KD.dispatch_id = S.dispatch_id
    AND KD.guid = S.guid
  GROUP BY KD.guid, KD.kernel_id
)
SELECT
  PK.kernel_id, KS.display_name, PK.samples, PK.stall_samples,
  (100.0 * PK.stall_samples) / NULLIF(PK.samples, 0) AS stall_pct
FROM per_kernel PK
LEFT JOIN rocpd_info_kernel_symbol KS ON KS.id = PK.kernel_id
  AND KS.guid = PK.guid
ORDER BY stall_pct DESC, PK.samples DESC
LIMIT 20"

Top sampled instruction offsets per kernel:

rocpd query -i profile.db --query "
SELECT
  KD.guid, KD.kernel_id, KS.display_name, S.code_object_offset,
  COUNT(*) AS samples_at_offset
FROM rocpd_gpu_pc_sample_decoded S
INNER JOIN rocpd_kernel_dispatch KD ON KD.dispatch_id = S.dispatch_id
  AND KD.guid = S.guid
LEFT JOIN rocpd_info_kernel_symbol KS ON KS.id = KD.kernel_id
  AND KS.guid = KD.guid
GROUP BY KD.guid, KD.kernel_id, KS.display_name, S.code_object_offset
ORDER BY samples_at_offset DESC
LIMIT 50"

Instruction histogram per kernel:

rocpd query -i profile.db --query "
WITH per_instruction AS (
  SELECT KD.guid, KD.kernel_id, S.code_object_offset, COUNT(*) AS instruction_samples
  FROM rocpd_gpu_pc_sample_decoded S
  INNER JOIN rocpd_kernel_dispatch KD ON KD.dispatch_id = S.dispatch_id
    AND KD.guid = S.guid
  GROUP BY KD.guid, KD.kernel_id, S.code_object_offset
),
histogram AS (
  SELECT
    guid,
    kernel_id,
    CASE
      WHEN instruction_samples < 10   THEN '1-9'
      WHEN instruction_samples < 100  THEN '10-99'
      WHEN instruction_samples < 1000 THEN '100-999'
      ELSE '1000+'
    END AS sample_bucket,
    COUNT(*) AS instruction_count
  FROM per_instruction
  GROUP BY guid, kernel_id, sample_bucket
)
SELECT H.kernel_id, KS.display_name, H.sample_bucket, H.instruction_count
FROM histogram H
LEFT JOIN rocpd_info_kernel_symbol KS ON KS.id = H.kernel_id
  AND KS.guid = H.guid
ORDER BY H.kernel_id, H.sample_bucket"

Internal pipeline latency hole detection:

rocpd query -i profile.db --query "
WITH sk AS (
  SELECT KD.guid, KD.kernel_id, S.code_object_id, S.code_object_offset,
         S.stall_reason_name,
         S.dual_issue_valu,
         S.arb_state_issue_valu, S.arb_state_stall_valu,
         S.arb_state_issue_matrix, S.arb_state_stall_matrix,
         S.arb_state_issue_lds, S.arb_state_stall_lds,
         S.arb_state_issue_scalar, S.arb_state_stall_scalar,
         S.arb_state_issue_vmem_tex, S.arb_state_stall_vmem_tex,
         S.arb_state_issue_flat, S.arb_state_stall_flat,
         S.arb_state_issue_exp, S.arb_state_stall_exp,
         S.arb_state_issue_misc, S.arb_state_stall_misc
  FROM rocpd_gpu_pc_sample_decoded S
  INNER JOIN rocpd_kernel_dispatch KD ON KD.dispatch_id = S.dispatch_id
    AND KD.guid = S.guid
),
per_pc AS (
  SELECT guid, kernel_id, code_object_id, code_object_offset,
         COUNT(*) AS samples,
         SUM(CASE WHEN stall_reason_name =
                       'ROCPROFILER_PC_SAMPLING_INSTRUCTION_NOT_ISSUED_REASON_INTERNAL_INSTRUCTION'
                  THEN 1 ELSE 0 END) AS internal_instruction_samples,
         SUM(CASE WHEN dual_issue_valu=1 OR
                       arb_state_issue_valu=1 OR arb_state_issue_matrix=1 OR
                       arb_state_issue_lds=1 OR
                       arb_state_issue_scalar=1 OR arb_state_issue_vmem_tex=1 OR
                       arb_state_issue_flat=1 OR arb_state_issue_exp=1 OR
                       arb_state_issue_misc=1
                  THEN 1 ELSE 0 END) AS any_pipeline_issue_samples,
         SUM(CASE WHEN
                       (arb_state_issue_valu=1       AND arb_state_stall_valu=0) OR
                       (arb_state_issue_matrix=1     AND arb_state_stall_matrix=0) OR
                       (arb_state_issue_lds=1        AND arb_state_stall_lds=0) OR
                       (arb_state_issue_scalar=1     AND arb_state_stall_scalar=0) OR
                       (arb_state_issue_vmem_tex=1   AND arb_state_stall_vmem_tex=0) OR
                       (arb_state_issue_flat=1       AND arb_state_stall_flat=0) OR
                       (arb_state_issue_exp=1        AND arb_state_stall_exp=0) OR
                       (arb_state_issue_misc=1       AND arb_state_stall_misc=0)
                  THEN 1 ELSE 0 END) AS useful_issue_samples
  FROM sk
  GROUP BY guid, kernel_id, code_object_id, code_object_offset
)
SELECT guid, kernel_id, code_object_id, code_object_offset, samples,
       internal_instruction_samples, any_pipeline_issue_samples, useful_issue_samples,
       CASE WHEN internal_instruction_samples>0 AND useful_issue_samples=0 THEN 1 ELSE 0 END
         AS internal_latency_hole
FROM per_pc
WHERE internal_instruction_samples>0
ORDER BY internal_latency_hole DESC, any_pipeline_issue_samples DESC, samples DESC"

For information about enabling PC sampling in rocprofv3, see Using PC sampling and ROCPD output format.