Core API

Core API#

2026-03-31

24 min read time

Applies to Linux

Cluster management#

class dask_cuda.local_cuda_cluster.LocalCUDACluster(HIP_VISIBLE_DEVICES=None, n_workers=None, threads_per_worker=1, memory_limit='auto', device_memory_limit='default', enable_cudf_spill=False, cudf_spill_stats=0, local_directory=None, shared_filesystem=None, protocol=None, enable_tcp_over_ucx=None, enable_infiniband=None, enable_rocm_ipc=None, enable_rdmacm=None, rmm_pool_size=None, rmm_maximum_pool_size=None, rmm_managed_memory=False, rmm_async=False, rmm_allocator_external_lib_list=None, rmm_release_threshold=None, rmm_log_directory=None, rmm_track_allocations=False, jit_unspill=None, log_spilling=False, pre_import=None, CUDA_VISIBLE_DEVICES=None, enable_nvlink=None, **kwargs)#

Bases: distributed.LocalCluster

A variant of dask.distributed.LocalCluster that uses one GPU per process.

This assigns a rotated list of visible devices to each Dask worker process, such that each worker sees a different default device.

For machines with a complex architecture mapping CPUs, GPUs, and network hardware, this class creates a local cluster that tries to respect this hardware as much as possible.

Each worker process is automatically assigned the correct CPU cores and network interface cards to maximize performance. If UCX and distributed-ucxx are available, InfiniBand and XGMI connections can be used to optimize data transfer performance.

Parameters:

  • HIP_VISIBLE_DEVICES (str, list of int, or None, default None) – GPUs to restrict activity to. Can be a string (like "0,1,2,3"), list (like [0, 1, 2, 3]), or None to use all available GPUs. For compatibility with existing code targeting CUDA, CUDA_VISIBLE_DEVICES can also be used instead of HIP_VISIBLE_DEVICES.

  • n_workers (int or None, default None) – Number of workers. Can be an integer or None to fall back on the GPUs specified by HIP_VISIBLE_DEVICES. The value of n_workers must be smaller or equal to the number of GPUs specified in HIP_VISIBLE_DEVICES when the latter is specified, and if smaller, only the first n_workers GPUs will be used.

  • threads_per_worker (int, default 1) – Number of threads to be used for each Dask worker process.

  • memory_limit (int, float, str, or None, default "auto") – Size of the host LRU cache, which is used to determine when the worker starts spilling to disk (not available if JIT-Unspill is enabled). Can be an integer (bytes), float (fraction of total system memory), string (like "5GB" or "5000M"), or "auto", 0, or None for no memory management.

  • device_memory_limit (int, float, str, or None, default "default") – Size of the CUDA device LRU cache, which is used to determine when the worker starts spilling to host memory. Can be an integer (bytes), float (fraction of total device memory), string (like "5GB" or "5000M"), "auto", 0 or None to disable spilling to host (i.e. allow full device memory usage). Another special value "default" (which happens to be the default) is also available and uses the recommended Dask-CUDA’s defaults and means 80% of the total device memory (analogous to 0.8), and disabled spilling (analogous to auto/0) on devices without a dedicated memory resource, such as system on a chip (SoC) devices.

  • enable_cudf_spill (bool, default False) –

    Enable automatic cuDF spilling.

    Warning

    This should NOT be used together with JIT-Unspill.

  • cudf_spill_stats (int, default 0) – Set the cuDF spilling statistics level. This option has no effect if enable_cudf_spill=False.

  • local_directory (str or None, default None) – Path on local machine to store temporary files. Can be a string (like "path/to/files") or None to fall back on the value of dask.temporary-directory in the local Dask configuration, using the current working directory if this is not set.

  • shared_filesystem (bool or None, default None) – Whether the local_directory above is shared between all workers or not. If None, the “jit-unspill-shared-fs” config value are used, which defaults to True. Notice, in all other cases this option defaults to False, but on a local cluster it defaults to True – we assume all workers use the same filesystem.

  • protocol (str or None, default None) – Protocol to use for communication. Can be a string (like "tcp" or "ucx"), or None to automatically choose the correct protocol.

  • enable_tcp_over_ucx (bool, default None) – Set environment variables to enable TCP over UCX, even if InfiniBand and NVLink are not supported or disabled.

  • enable_infiniband (bool, default None) – Set environment variables to enable UCX over InfiniBand, requires protocol="ucx", and implies enable_tcp_over_ucx=True when True.

  • enable_rocm_ipc (bool, default None) – Set environment variables to enable UCX over ROCm-IPC, requires protocol="ucx", and implies enable_tcp_over_ucx=True when True.

  • enable_rdmacm (bool, default None) – Set environment variables to enable UCX RDMA connection manager support, requires protocol="ucx", and enable_infiniband=True.

  • rmm_pool_size (int, str or None, default None) –

    RMM pool size to initialize each worker with. Can be an integer (bytes), float (fraction of total device memory), string (like "5GB" or "5000M"), or None to disable RMM pools.

    Note

    This size is a per-worker configuration, and not cluster-wide.

  • rmm_maximum_pool_size (int, str or None, default None) –

    When rmm_pool_size is set, this argument indicates the maximum pool size. Can be an integer (bytes), float (fraction of total device memory), string (like "5GB" or "5000M") or None. By default, the total available memory on the GPU is used. rmm_pool_size must be specified to use RMM pool and to set the maximum pool size.

    Note

    When paired with --enable-rmm-async the maximum size cannot be guaranteed due to fragmentation.

    Note

    This size is a per-worker configuration, and not cluster-wide.

  • rmm_managed_memory (bool, default False) –

    Initialize each worker with RMM and set it to use managed memory. If disabled, RMM may still be used by specifying rmm_pool_size.

    Warning

    Managed memory is currently incompatible with NVLink. Trying to enable both will result in an exception.

  • rmm_async (bool, default False) –

    Initialize each worker with RMM and set it to use RMM’s asynchronous allocator. See rmm.mr.CudaAsyncMemoryResource for more info.

    Warning

    The asynchronous allocator is incompatible with RMM pools and managed memory. Trying to enable both will result in an exception.

  • rmm_allocator_external_lib_list (str, list or None, default None) – List of external libraries for which to set RMM as the allocator. Supported options are: ["torch", "cupy"]. Can be a comma-separated string (like "torch,cupy") or a list of strings (like ["torch", "cupy"]). If None, no external libraries will use RMM as their allocator.

  • rmm_release_threshold (int, str or None, default None) –

    When rmm.async is True and the pool size grows beyond this value, unused memory held by the pool will be released at the next synchronization point. Can be an integer (bytes), float (fraction of total device memory), string (like "5GB" or "5000M") or None. By default, this feature is disabled.

    Note

    This size is a per-worker configuration, and not cluster-wide.

  • rmm_log_directory (str or None, default None) –

    Directory to write per-worker RMM log files to. The client and scheduler are not logged here. Can be a string (like "/path/to/logs/") or None to disable logging.

    Note

    Logging will only be enabled if rmm_pool_size is specified or rmm_managed_memory=True.

  • rmm_track_allocations (bool, default False) –

    If True, wraps the memory resource used by each worker with a rmm.mr.TrackingResourceAdaptor, which tracks the amount of memory allocated.

    Note

    This option enables additional diagnostics to be collected and reported by the Dask dashboard. However, there is significant overhead associated with this and it should only be used for debugging and memory profiling.

  • jit_unspill (bool or None, default None) –

    Enable just-in-time unspilling. Can be a boolean or None to fall back on the value of dask.jit-unspill in the local Dask configuration, disabling unspilling if this is not set.

    Note

    This is experimental and doesn’t support memory spilling to disk. See proxy_object.ProxyObject and proxify_host_file.ProxifyHostFile for more info.

  • log_spilling (bool, default True) – Enable logging of spilling operations directly to distributed.Worker with an INFO log level.

  • pre_import (str, list or None, default None) – Pre-import libraries as a Worker plugin to prevent long import times bleeding through later Dask operations. Should be a list of comma-separated names, such as “cudf,rmm” or a list of strings such as [“cudf”, “rmm”].

Examples

>>> from dask_cuda import LocalCUDACluster
>>> from dask.distributed import Client
>>> cluster = LocalCUDACluster()
>>> client = Client(cluster)
Raises:

  • TypeError – If InfiniBand or NVLink are enabled and protocol != "ucx".

  • ValueError – If RMM pool, RMM managed memory or RMM async allocator are requested but RMM cannot be imported. If RMM managed memory and asynchronous allocator are both enabled. If RMM maximum pool size is set but RMM pool size is not. If RMM maximum pool size is set but RMM async allocator is used. If RMM release threshold is set but the RMM async allocator is not being used.

Parameters:

  • threads_per_worker (int)

  • memory_limit (str)

  • device_memory_limit (str)

  • enable_cudf_spill (bool)

  • cudf_spill_stats (int)

  • rmm_managed_memory (bool)

  • rmm_async (bool)

  • rmm_track_allocations (bool)

  • log_spilling (bool)

See also

LocalCluster

cuda_visible_devices: _typeshed.Incomplete#
cudf_spill_stats: _typeshed.Incomplete#
data: _typeshed.Incomplete#
device_memory_limit: _typeshed.Incomplete#
enable_cudf_spill: _typeshed.Incomplete#
host: _typeshed.Incomplete#
memory_limit: _typeshed.Incomplete#
new_worker_spec()#
pre_import: _typeshed.Incomplete#
rmm_allocator_external_lib_list: _typeshed.Incomplete#
rmm_async: _typeshed.Incomplete#
rmm_log_directory: _typeshed.Incomplete#
rmm_managed_memory: _typeshed.Incomplete#
rmm_maximum_pool_size: _typeshed.Incomplete#
rmm_pool_size: _typeshed.Incomplete#
rmm_release_threshold: _typeshed.Incomplete#
rmm_track_allocations: _typeshed.Incomplete#

Worker#

class dask_cuda.cuda_worker.CUDAWorker(scheduler=None, host=None, nthreads=1, name=None, memory_limit='auto', device_memory_limit='default', enable_cudf_spill=False, cudf_spill_stats=0, rmm_pool_size=None, rmm_maximum_pool_size=None, rmm_managed_memory=False, rmm_async=False, rmm_allocator_external_lib_list=None, rmm_release_threshold=None, rmm_log_directory=None, rmm_track_allocations=False, pid_file=None, resources=None, dashboard=True, dashboard_address=':0', local_directory=None, shared_filesystem=None, scheduler_file=None, interface=None, preload=[], dashboard_prefix=None, security=None, enable_tcp_over_ucx=None, enable_infiniband=None, enable_rocm_ipc=None, enable_rdmacm=None, jit_unspill=None, worker_class=None, pre_import=None, enable_nvlink=None, **kwargs)#

Bases: distributed.core.Server

Parameters:

  • nthreads (int)

  • memory_limit (str)

  • device_memory_limit (str)

  • enable_cudf_spill (bool)

  • cudf_spill_stats (int)

  • rmm_managed_memory (bool)

  • rmm_async (bool)

  • rmm_track_allocations (bool)

  • dashboard (bool)

  • dashboard_address (str)

async close(timeout=5)#
Parameters:

timeout (int)

Return type:

None

async finished()#
Return type:

None

nannies: _typeshed.Incomplete#

Worker specification#

dask_cuda.worker_spec.worker_spec(interface=None, protocol=None, dashboard_address=':8787', threads_per_worker=1, silence_logs=True, CUDA_VISIBLE_DEVICES=None, enable_tcp_over_ucx=False, enable_infiniband=False, enable_rocm_ipc=False, enable_nvlink=None, **kwargs)#

Create a Spec for a CUDA worker.

The Spec created by this function can be used as a recipe for CUDA workers that can be passed to a SpecCluster.

Parameters:

  • interface (str) – The external interface used to connect to the scheduler.

  • protocol (str) – The protocol to used for data transfer, e.g., “tcp” or “ucx”.

  • dashboard_address (str) – The address for the scheduler dashboard. Defaults to “:8787”.

  • threads_per_worker (int) – Number of threads to be used for each CUDA worker process.

  • silence_logs (bool) – Disable logging for all worker processes

  • CUDA_VISIBLE_DEVICES (str) – String like "0,1,2,3" or [0, 1, 2, 3] to restrict activity to different GPUs

  • enable_tcp_over_ucx (bool) – Set environment variables to enable TCP over UCX, even if InfiniBand and NVLink are not supported or disabled.

  • enable_infiniband (bool) – Set environment variables to enable UCX InfiniBand support. Implies enable_tcp_over_ucx=True.

  • enable_rocm_ipc (bool) – Set environment variables to enable UCX ROCm-IPC support. Implies enable_tcp_over_ucx=True.

Examples

>>> from dask_cuda.worker_spec import worker_spec
>>> worker_spec(interface="enp1s0f0", CUDA_VISIBLE_DEVICES=[0, 2])
{0: {'cls': distributed.nanny.Nanny,
  'options': {'env': {'CUDA_VISIBLE_DEVICES': '0,2'},
   'interface': 'enp1s0f0',
   'protocol': None,
   'nthreads': 1,
   'data': dict,
   'dashboard_address': ':8787',
   'plugins': [<dask_cuda.utils.CPUAffinity at 0x7fbb8748a860>],
   'silence_logs': True,
   'memory_limit': 135263611392.0,
   'preload': ['dask_cuda.initialize'],
   'preload_argv': ['--create-cuda-context']}},
 2: {'cls': distributed.nanny.Nanny,
  'options': {'env': {'CUDA_VISIBLE_DEVICES': '2,0'},
   'interface': 'enp1s0f0',
   'protocol': None,
   'nthreads': 1,
   'data': dict,
   'dashboard_address': ':8787',
   'plugins': [<dask_cuda.utils.CPUAffinity at 0x7fbb8748a0f0>],
   'silence_logs': True,
   'memory_limit': 135263611392.0,
   'preload': ['dask_cuda.initialize'],
   'preload_argv': ['--create-cuda-context']}}}

Client initialization#

dask_cuda.initialize.initialize(create_cuda_context=True, enable_tcp_over_ucx=None, enable_infiniband=None, enable_rocm_ipc=None, enable_rdmacm=None, enable_nvlink=None)#

Create CUDA context and initialize UCXX configuration.

Sometimes it is convenient to initialize the CUDA context, particularly before starting up Dask worker processes which create a variety of threads.

To ensure UCX works correctly, it is important to ensure it is initialized with the correct options. This is especially important for the client, which cannot be configured to use UCX with arguments like LocalCUDACluster and dask cuda worker. This function will ensure that they are provided a UCX configuration based on the flags and options passed by the user.

This function can also be used within a worker preload script for UCX configuration of mainline Dask.distributed. https://docs.dask.org/en/latest/setup/custom-startup.html

You can add it to your global config with the following YAML:

distributed:
  worker:
    preload:
      - dask_cuda.initialize

See https://docs.dask.org/en/latest/configuration.html for more information about Dask configuration.

Parameters:

  • create_cuda_context (bool, default True) – Create CUDA context on initialization.

  • enable_tcp_over_ucx (bool, default None) – Set environment variables to enable TCP over UCX, even if InfiniBand and NVLink are not supported or disabled.

  • enable_infiniband (bool, default None) – Set environment variables to enable UCX over InfiniBand, implies enable_tcp_over_ucx=True when True.

  • enable_rocm_ipc (bool, default None) – Set environment variables to enable UCX over ROCm-IPC, implies enable_tcp_over_ucx=True when True.

  • enable_rdmacm (bool, default None) – Set environment variables to enable UCX RDMA connection manager support, requires enable_infiniband=True.

  • enable_nvlink (bool, default None) – Unsupported on AMD, only provided for easy portability. Please use enable_rocm_ipc instead.

Return type:

None

Explicit communication#

class dask_cuda.explicit_comms.comms.CommsContext(client=None)#

Communication handler for explicit communication

Parameters:

client (Client, optional) – Specify client to use for communication. If None, use the default client.

client: distributed.Client#
run(coroutine, *args, workers=None, lock_workers=False)#

Run a coroutine on multiple workers

The coroutine is given the worker’s state dict as the first argument and *args as the following arguments.

Parameters:

  • coroutine (coroutine) – The function to run on each worker

  • *args – Arguments for coroutine

  • workers (list, optional) – List of workers. Default is all workers

  • lock_workers (bool, optional) – Use distributed.MultiLock to get exclusive access to the workers. Use this flag to support parallel runs.

Returns:

ret – List of the output from each worker

Return type:

list

sessionId: int#
stage_keys(name, keys)#

Staging keys on workers under the given name

In an explicit-comms task, use pop_staging_area(..., name) to access the staged keys and the associated data.

Notes

In the context of explicit-comms, staging is the act of duplicating the responsibility of Dask keys. When staging a key, the worker owning the key (as assigned by the Dask scheduler) save a reference to the key and the associated data to its local staging area. From this point on, if the scheduler cancels the key, the worker (and the task running on the worker) now has exclusive access to the key and the associated data. This way, staging makes it possible for long running explicit-comms tasks to free input data ASAP.

Parameters:

  • name (str) – Name for the staging area

  • keys (iterable) – The keys to stage

Returns:

dict that maps each worker-rank to the workers set of staged keys

Return type:

dict

submit(worker, coroutine, *args, wait=False)#

Run a coroutine on a single worker

The coroutine is given the worker’s state dict as the first argument and *args as the following arguments.

Parameters:

  • worker (str) – Worker to run the coroutine

  • coroutine (coroutine) – The function to run on the worker

  • *args – Arguments for coroutine

  • wait (boolean, optional) – If True, waits for the coroutine to finished before returning.

Returns:

ret – If wait=True, the result of coroutine If wait=False, Future that can be waited on later.

Return type:

object or Future

worker_addresses: list[str]#
worker_direct_addresses: _typeshed.Incomplete#
dask_cuda.explicit_comms.comms.default_comms(client=None)#

Return the default comms object for client.

Creates a new default comms object if one does not already exist for client.

Parameters:

client (Client, optional) – If no default comm object exists, create the new comm on client are returned.

Returns:

comms – The default comms object

Return type:

CommsContext

Notes

There are some subtle points around explicit-comms and the lifecycle of a Dask Cluster.

A CommsContext establishes explicit communication channels between the workers at the time it’s created. If workers are added or removed, they will not be included in the communication channels with the other workers.

If you need to refresh the explicit communications channels, then create a new CommsContext object or call default_comms again after workers have been added to or removed from the cluster.

dask_cuda.explicit_comms.dataframe.shuffle.shuffle(df, column_names, npartitions=None, ignore_index=False, batchsize=None)#

Order divisions of DataFrame so that all values within column(s) align

This enacts a task-based shuffle using explicit-comms. It requires a full dataset read, serialization and shuffle. This is expensive. If possible you should avoid shuffles.

This does not preserve a meaningful index/partitioning scheme. This is not deterministic if done in parallel.

Requires an activate client.

Parameters:

  • df (dask.dataframe.DataFrame) – Dataframe to shuffle

  • column_names (list of strings) – List of column names on which we want to split.

  • npartitions (int or None) – The desired number of output partitions. If None, the number of output partitions equals df.npartitions

  • ignore_index (bool) – Ignore index during shuffle. If True, performance may improve, but index values will not be preserved.

  • batchsize (int) – A shuffle consist of multiple rounds where each worker partitions and then all-to-all communicates a number of its dataframe partitions. The batch size is the number of partitions each worker will handle in each round. If -1, each worker will handle all its partitions in a single round and all techniques to reduce memory usage are disabled, which might be faster when memory pressure isn’t an issue. If None, the value of DASK_EXPLICIT_COMMS_BATCHSIZE is used or 1 if not set thus by default, we prioritize robustness over performance.

Returns:

  • df (dask.dataframe.DataFrame) – Shuffled dataframe

  • Developer Notes

  • ---------------

  • The implementation consist of three steps –

    1. Stage the partitions of df on all workers and then cancel them thus at this point the Dask Scheduler doesn’t know about any of the the partitions.

    2. Submit a task on each worker that shuffle (all-to-all communicate) the staged partitions and return a list of dataframe-partitions.

    3. Submit a dask graph that extract (using getitem()) individual dataframe-partitions from (b).

Return type:

dask.dataframe.DataFrame

CLI#

dask-cuda-worker#

Launch a distributed worker with GPUs attached to an existing scheduler.

A scheduler can be specified either through a URI passed through the SCHEDULER argument or a scheduler file passed through the --scheduler-file option.

See https://docs.rapids.ai/api/dask-cuda/stable/quickstart.html#dask-cuda-worker for info.

Usage

dask cuda worker [OPTIONS] [SCHEDULER] [PRELOAD_ARGV]...

Options

--host <host>#

IP address of serving host; should be visible to the scheduler and other workers. Can be a string (like "127.0.0.1") or None to fall back on the address of the interface specified by --interface or the default interface.

--nthreads <nthreads>#

Number of threads to be used for each Dask worker process.

Default:

1

--name <name>#

A unique name for the worker. Can be a string (like "worker-1") or None for a nameless worker.

--memory-limit <memory_limit>#

Size of the host LRU cache, which is used to determine when the worker starts spilling to disk (not available if JIT-Unspill is enabled). Can be an integer (bytes), float (fraction of total system memory), string (like "5GB" or "5000M"), or "auto" or 0 for no memory management.

Default:

'auto'

--device-memory-limit <device_memory_limit>#

Size of the CUDA device LRU cache, which is used to determine when the worker starts spilling to host memory. Can be an integer (bytes), float (fraction of total device memory), string (like "5GB" or "5000M"), "auto" or 0 to disable spilling to host (i.e. allow full device memory usage). Another special value "default" (which happens to be the default) is also available and uses the recommended Dask-CUDA’s defaults and means 80% of the total device memory (analogous to 0.8), and disabled spilling (analogous to auto/0) on devices without a dedicated memory resource, such as system on a chip (SoC) devices.

Default:

'default'

--enable-cudf-spill, --disable-cudf-spill#

Enable automatic cuDF spilling. WARNING: This should NOT be used with JIT-Unspill.

Default:

False

--cudf-spill-stats <cudf_spill_stats>#

Set the cuDF spilling statistics level. This option has no effect if --enable-cudf-spill is not specified.

--rmm-pool-size <rmm_pool_size>#

RMM pool size to initialize each worker with. Can be an integer (bytes), float (fraction of total device memory), string (like "5GB" or "5000M"), or None to disable RMM pools.

Note

This size is a per-worker configuration, and not cluster-wide.

--rmm-maximum-pool-size <rmm_maximum_pool_size>#

When --rmm-pool-size is specified, this argument indicates the maximum pool size. Can be an integer (bytes), float (fraction of total device memory), string (like "5GB" or "5000M") or None. By default, the total available memory on the GPU is used. rmm_pool_size must be specified to use RMM pool and to set the maximum pool size.

Note

When paired with --enable-rmm-async the maximum size cannot be guaranteed due to fragmentation.

Note

This size is a per-worker configuration, and not cluster-wide.

--rmm-managed-memory, --no-rmm-managed-memory#

Initialize each worker with RMM and set it to use managed memory. If disabled, RMM may still be used by specifying --rmm-pool-size.

Warning

Managed memory is currently incompatible with NVLink. Trying to enable both will result in failure.

Default:

False

--rmm-async, --no-rmm-async#

Initialize each worker with RMM and set it to use RMM’s asynchronous allocator. See rmm.mr.CudaAsyncMemoryResource for more info.

Warning

The asynchronous allocator is incompatible with RMM pools and managed memory, trying to enable both will result in failure.

Default:

False

--set-rmm-allocator-for-libs <rmm_allocator_external_lib_list>#

Set RMM as the allocator for external libraries. Provide a comma-separated list of libraries to set, e.g., “torch,cupy”.

Options:

cupy | torch

--rmm-release-threshold <rmm_release_threshold>#

When rmm.async is True and the pool size grows beyond this value, unused memory held by the pool will be released at the next synchronization point. Can be an integer (bytes), float (fraction of total device memory), string (like "5GB" or "5000M") or None. By default, this feature is disabled.

Note

This size is a per-worker configuration, and not cluster-wide.

--rmm-log-directory <rmm_log_directory>#

Directory to write per-worker RMM log files to. The client and scheduler are not logged here. Can be a string (like "/path/to/logs/") or None to disable logging.

Note

Logging will only be enabled if --rmm-pool-size or --rmm-managed-memory are specified.

--rmm-track-allocations, --no-rmm-track-allocations#

Track memory allocations made by RMM. If True, wraps the memory resource of each worker with a rmm.mr.TrackingResourceAdaptor that allows querying the amount of memory allocated by RMM.

Default:

False

--pid-file <pid_file>#

File to write the process PID.

--resources <resources>#

Resources for task constraints like "GPU=2 MEM=10e9".

--dashboard, --no-dashboard#

Launch the dashboard.

Default:

True

--dashboard-address <dashboard_address>#

Relative address to serve the dashboard (if enabled).

Default:

':0'

--local-directory <local_directory>#

Path on local machine to store temporary files. Can be a string (like "path/to/files") or None to fall back on the value of dask.temporary-directory in the local Dask configuration, using the current working directory if this is not set.

--shared-filesystem, --no-shared-filesystem#

If --shared-filesystem is specified, inform JIT-Unspill that local_directory is a shared filesystem available for all workers, whereas --no-shared-filesystem informs it may not assume it’s a shared filesystem. If neither is specified, JIT-Unspill will decide based on the Dask config value specified by "jit-unspill-shared-fs". Notice, a shared filesystem must support the os.link() operation.

--scheduler-file <scheduler_file>#

Filename to JSON encoded scheduler information. To be used in conjunction with the equivalent dask scheduler option.

--protocol <protocol>#

Protocol like tcp, tls, or ucx

--interface <interface>#

External interface used to connect to the scheduler. Usually an ethernet interface is used for connection, and not an InfiniBand interface (if one is available). Can be a string (like "eth0" for NVLink or "ib0" for InfiniBand) or None to fall back on the default interface.

--preload <preload>#

Module that should be loaded by each worker process like "foo.bar" or "/path/to/foo.py".

--death-timeout <death_timeout>#

Seconds to wait for a scheduler before closing

--dashboard-prefix <dashboard_prefix>#

Prefix for the dashboard. Can be a string (like …) or None for no prefix.

--tls-ca-file <tls_ca_file>#

CA certificate(s) file for TLS (in PEM format). Can be a string (like "path/to/certs"), or None for no certificate(s).

--tls-cert <tls_cert>#

Certificate file for TLS (in PEM format). Can be a string (like "path/to/certs"), or None for no certificate(s).

--tls-key <tls_key>#

Private key file for TLS (in PEM format). Can be a string (like "path/to/certs"), or None for no private key.

--enable-tcp-over-ucx, --disable-tcp-over-ucx#

Set environment variables to enable TCP over UCX, even if InfiniBand and NVLink are not supported or disabled.

--enable-infiniband, --disable-infiniband#

Set environment variables to enable UCX over InfiniBand, implies --enable-tcp-over-ucx when enabled.

--enable-rocm-ipc, --disable-rocm-ipc#

Set environment variables to enable rocm_ipc TL for UCX, implies --enable-tcp-over-ucx when enabled.

Same effect as --enable-rocm-ipc/--disable-rocm-ipc, as NVLink in not valid/supported for AMD. Only supported for easy portability. A warning will be shown unless suppressed using the ‘DASK_HIP_SUPPRESS_NVLINK_WARNING’ environment variable. Please use --enable-rocm-ipc/--disable-rocm-ipc instead.

--enable-rdmacm, --disable-rdmacm#

Set environment variables to enable UCX RDMA connection manager support, requires --enable-infiniband.

--enable-jit-unspill, --disable-jit-unspill#

Enable just-in-time unspilling. Can be a boolean or None to fall back on the value of dask.jit-unspill in the local Dask configuration, disabling unspilling if this is not set.

Note

This is experimental and doesn’t support memory spilling to disk. See proxy_object.ProxyObject and proxify_host_file.ProxifyHostFile for more info.

--worker-class <worker_class>#

Use a different class than Distributed’s default (distributed.Worker) to spawn distributed.Nanny.

--pre-import <pre_import>#

Pre-import libraries as a Worker plugin to prevent long import times bleeding through later Dask operations. Should be a list of comma-separated names, such as “cudf,rmm”.

--multiprocessing-method <multiprocessing_method>#

Method used to start new processes with multiprocessing

Options:

spawn | fork | forkserver

Arguments

SCHEDULER#

Optional argument

PRELOAD_ARGV#

Optional argument(s)

dask-cuda-config#

Query an existing GPU cluster’s configuration.

A cluster can be specified either through a URI passed through the SCHEDULER argument or a scheduler file passed through the --scheduler-file option.

Usage

dask cuda config [OPTIONS] [SCHEDULER] [PRELOAD_ARGV]...

Options

--scheduler-file <scheduler_file>#

Filename to JSON encoded scheduler information. To be used in conjunction with the equivalent dask scheduler option.

--tls-ca-file <tls_ca_file>#

CA certificate(s) file for TLS (in PEM format). Can be a string (like "path/to/certs"), or None for no certificate(s).

--tls-cert <tls_cert>#

Certificate file for TLS (in PEM format). Can be a string (like "path/to/certs"), or None for no certificate(s).

--tls-key <tls_key>#

Private key file for TLS (in PEM format). Can be a string (like "path/to/certs"), or None for no private key.

Arguments

SCHEDULER#

Optional argument

PRELOAD_ARGV#

Optional argument(s)

Compatibility notes#

For portability, LocalCUDACluster, CUDAWorker, and the dask cuda worker CLI accept enable_nvlink / --enable-nvlink parameters, mapping them to enable_rocm_ipc / --enable-rocm-ipc with a deprecation warning. This allows scripts written for dask-cuda on NVIDIA platforms to work on dask-hip without modification.

The deprecation warning can be suppressed by setting the DASK_HIP_SUPPRESS_NVLINK_WARNING=1 environment variable.