Cluster Validation Suite (CVS) scalability and parallel SSH performance

Cluster Validation Suite (CVS) scalability and parallel SSH performance#

2026-09-24

3 min read time

Applies to Linux

CVS automatically scales to handle clusters from small lab setups to large enterprise deployments with thousands of nodes.

Parallel execution#

CVS runs cluster-wide SSH concurrently—it does not connect to hosts one at a time.

  • Default (up to 32 hosts): A single process uses gevent-based concurrent SSH (ParallelSSHClient) across all hosts.

  • Large host lists (more than 32 hosts, or ``CVS_HOSTS_PER_SHARD``): CVS additionally splits hosts into shards and runs each shard in a separate worker process. Each worker still uses gevent concurrency inside the process.

  • Results: Output is merged while preserving the original host order.

Example: with 50 hosts and default settings, CVS:

  1. Splits hosts into shards of 32

  2. Runs each shard in a parallel worker process (gevent concurrency within each shard)

  3. Merges results maintaining original host order

Example: with 8 hosts, CVS runs one gevent-based ParallelSSHClient over all hosts—no process sharding.

Environment variables#

Configure CVS parallel SSH operations and optimize performance for your cluster size:

CVS_HOSTS_PER_SHARD (default: 32)

When the host count exceeds this value, CVS splits work across multiple worker processes. Each process still uses gevent-based concurrent SSH. Lower the value to enable process sharding on smaller clusters; set 0 to disable process sharding (always one gevent client in the parent process).

export CVS_HOSTS_PER_SHARD=64  # Process 64 hosts per shard instead of default 32
CVS_WORKERS_PER_CPU (default: 4)

Sets the number of worker processes per CPU core for parallel operations. The total number of workers is calculated as CPU_COUNT * CVS_WORKERS_PER_CPU.

export CVS_WORKERS_PER_CPU=8  # Use 8 workers per CPU core instead of default 4

Performance tuning examples#

The following examples show recommended settings for common cluster sizes.

For large clusters (1,000 or more nodes):

export CVS_HOSTS_PER_SHARD=64
export CVS_WORKERS_PER_CPU=6

For smaller clusters or resource-constrained environments:

export CVS_HOSTS_PER_SHARD=16
export CVS_WORKERS_PER_CPU=2
Contents