Resources

Contents

Resources#

2025-10-17

21 min read time

Applies to Linux

All resources which are specific to a computing environment like host or device are contained within, and managed by, raft::resources. This design simplifies the APIs and eases user burden by making the APIs opaque by default but allowing customization based on user preference.

Vocabulary#

#include <raft/core/resource/resource_types.hpp>

namespace raft::resource

enum resource_type#

Resource types can apply to any resource and don’t have to be host- or device-specific.

Values:

enumerator CUBLAS_HANDLE#
enumerator CUSOLVER_DN_HANDLE#
enumerator CUSOLVER_SP_HANDLE#
enumerator CUSPARSE_HANDLE#
enumerator CUDA_STREAM_VIEW#
enumerator CUDA_STREAM_POOL#
enumerator CUDA_STREAM_SYNC_EVENT#
enumerator COMMUNICATOR#
enumerator SUB_COMMUNICATOR#
enumerator DEVICE_PROPERTIES#
enumerator DEVICE_ID#
enumerator STREAM_VIEW#
enumerator THRUST_POLICY#
enumerator WORKSPACE_RESOURCE#
enumerator CUBLASLT_HANDLE#
enumerator CUSTOM#
enumerator LARGE_WORKSPACE_RESOURCE#
enumerator NCCL_CLIQUE#
enumerator LAST_KEY#
class resource#
#include <resource_types.hpp>

A resource constructs and contains an instance of some pre-determined object type and facades that object behind a common API.

Subclassed by raft::resource::comms_resource, raft::resource::cublas_resource, raft::resource::cublaslt_resource, raft::resource::cuda_event_resource, raft::resource::cuda_stream_pool_resource, raft::resource::cuda_stream_resource, raft::resource::cusolver_dn_resource, raft::resource::cusolver_sp_resource, raft::resource::cusparse_resource, raft::resource::custom_resource, raft::resource::device_id_resource, raft::resource::device_memory_resource, raft::resource::device_properties_resource, raft::resource::empty_resource, raft::resource::limiting_memory_resource, raft::resource::nccl_clique_resource, raft::resource::stream_view_resource, raft::resource::sub_comms_resource, raft::resource::thrust_policy_resource

class empty_resource : public raft::resource::resource#
class resource_factory#
#include <resource_types.hpp>

A resource factory knows how to construct an instance of a specific raft::resource::resource.

Subclassed by raft::resource::comms_resource_factory, raft::resource::cublas_resource_factory, raft::resource::cublaslt_resource_factory, raft::resource::cuda_stream_pool_resource_factory, raft::resource::cuda_stream_resource_factory, raft::resource::cusolver_dn_resource_factory, raft::resource::cusolver_sp_resource_factory, raft::resource::cusparse_resource_factory, raft::resource::custom_resource_factory, raft::resource::detail::cuda_stream_sync_event_resource_factory, raft::resource::device_id_resource_factory, raft::resource::device_properties_resource_factory, raft::resource::empty_resource_factory, raft::resource::large_workspace_resource_factory, raft::resource::nccl_clique_resource_factory, raft::resource::stream_view_resource_factory, raft::resource::sub_comms_resource_factory, raft::resource::thrust_policy_resource_factory, raft::resource::workspace_resource_factory

Public Functions

virtual resource_type get_resource_type(
) = 0#

Return the resource_type associated with the current factory.

Returns:

resource_type corresponding to the current factory

virtual resource *make_resource(
) = 0#

Construct an instance of the factory’s underlying resource.

Returns:

resource instance

class empty_resource_factory : public raft::resource::resource_factory#
#include <resource_types.hpp>

A resource factory knows how to construct an instance of a specific raft::resource::resource.

Public Functions

inline virtual resource_type get_resource_type(
) override#

Return the resource_type associated with the current factory.

Returns:

resource_type corresponding to the current factory

inline virtual resource *make_resource(
) override#

Construct an instance of the factory’s underlying resource.

Returns:

resource instance

Device Resources#

raft::device_resources is a convenience over using raft::resources directly. It provides accessor methods to retrieve resources such as the CUDA stream, stream pool, and handles to the various CUDA math libraries like cuBLAS and cuSOLVER.

#include <raft/core/device_resources.hpp>

namespace raft::core

class device_resources : public raft::resources#

Main resource container object that stores all necessary resources used for calling necessary device functions, cuda kernels and/or libraries.

Subclassed by raft::handle_t

Public Functions

inline device_resources(
rmm::cuda_stream_view stream_view = rmm::cuda_stream_per_thread,
std::shared_ptr<rmm::cuda_stream_pool> stream_pool = {nullptr},
std::shared_ptr<rmm::mr::device_memory_resource> workspace_resource = {nullptr},
std::optional<std::size_t> allocation_limit = std::nullopt
)#

Construct a resources instance with a stream view and stream pool.

Parameters:

  • stream_view – the default stream (which has the default per-thread stream if unspecified)

  • stream_pool – [in] the stream pool used (which has default of nullptr if unspecified)

  • workspace_resource – [in] an optional resource used by some functions for allocating temporary workspaces.

  • allocation_limit – [in] the total amount of memory in bytes available to the temporary workspace resources.

inline virtual ~device_resources(
)#

Destroys all held-up resources

inline void sync_stream(
rmm::cuda_stream_view stream
) const#

synchronize a stream on the current container

inline void sync_stream(
) const#

synchronize main stream on the current container

inline rmm::cuda_stream_view get_stream(
) const#

returns main stream on the current container

inline bool is_stream_pool_initialized(
) const#

returns whether stream pool was initialized on the current container

inline const rmm::cuda_stream_pool &get_stream_pool(
) const#

returns stream pool on the current container

inline rmm::cuda_stream_view get_stream_from_stream_pool(
) const#

return stream from pool

inline rmm::cuda_stream_view get_stream_from_stream_pool(
std::size_t stream_idx
) const#

return stream from pool at index

inline rmm::cuda_stream_view get_next_usable_stream(
) const#

return stream from pool if size > 0, else main stream on current container

inline rmm::cuda_stream_view get_next_usable_stream(
std::size_t stream_idx
) const#

return stream from pool at index if size > 0, else main stream on current container

Parameters:

stream_idx – [in] the required index of the stream in the stream pool if available

inline void sync_stream_pool(
) const#

synchronize the stream pool on the current container

inline void sync_stream_pool(
const std::vector<std::size_t> stream_indices
) const#

synchronize subset of stream pool

Parameters:

stream_indices – [in] the indices of the streams in the stream pool to synchronize

inline void wait_stream_pool_on_stream(
) const#

ask stream pool to wait on last event in main stream

Device Resources Manager#

While raft::device_resources provides a convenient way to access device-related resources for a sequence of RAFT calls, it is sometimes useful to be able to limit those resources across an entire application. For instance, in highly multi-threaded applications, it can be helpful to limit the total number of streams rather than relying on the default stream per thread. raft::device_resources_manager offers a way to access raft::device_resources instances that draw from a limited pool of underlying device resources.

#include <raft/core/device_resources_manager.hpp>

namespace raft::core

struct device_resources_manager#

A singleton used to easily generate a raft::device_resources object.

Many calls to RAFT functions require a raft::device_resources object to provide CUDA resources like streams and stream pools. The raft::device_resources_manager singleton provides a straightforward method to create those objects in a way that allows consumers of RAFT to limit total consumption of device resources without actively managing streams or other CUDA-specific objects.

To control the resources a consuming application will use, the resource manager provides setters for a variety of values. For instance, to ensure that no more than N CUDA streams are used per device, a consumer might call raft::device_resources_manager::set_streams_per_device(N). Note that all of these setters must be used prior to retrieving the first device_resources from the manager. Setters invoked after this will log a warning but have no effect.

After calling all desired setters, consumers can simply call auto res = raft::device_resources_manager::get_device_resources(); to get a valid device_resources object for the current device based on previously-set parameters. Importantly, calling get_device_resources() again from the same thread is guaranteed to return a device_resources object with the same underlying CUDA stream and (if a non-zero number of stream pools has been requested) stream pool.

Typical usage might look something like the following: 

void initialize_application() {
  raft::device_resources_manager::set_streams_per_device(16);
}

void foo_called_from_multiple_threads() {
  auto res = raft::device_resources_manager::get_device_resources();
  // Call RAFT function using res
  res.sync_stream() // Ensure work completes before returning
}

Note that all public methods of the device_resources_manager are thread-safe, but the manager is designed to minimize locking required for retrieving device_resources objects. Each thread must acquire a lock exactly once per device when calling get_device_resources. Subsequent calls will still be thread-safe but will not require a lock.

All public methods of the device_resources_manager are static. Please see documentation of those methods for additional usage information.

Public Static Functions

static inline auto const &get_device_resources(
int device_id = device_setter::get_current_device()
)#

Retrieve device_resources to be used with the RAFT API.

This thread-safe method ensures that a device_resources object with the same underlying stream and stream pool is returned every time it is called by the same host thread. This means that if get_device_resources is used to provide all device_resources in an application, then raft::get_device_resources().sync_stream() and (if a stream pool is used) raft::get_device_resources().sync_stream_pool() are guaranteed to synchronize all work previously submitted to the device by this host thread.

If the max memory pool size set with set_max_mem_pool_size is non-zero, the first call of this method will also create a memory pool to be used for all RMM-based allocations on device.

Parameters:

device_id – int If provided, the device for which resources should be returned. Defaults to active CUDA device.

static inline void set_streams_per_device(
std::optional<std::size_t> num_streams
)#

Set the total number of CUDA streams to be used per device.

If nullopt, the default stream per thread will be used (essentially allowing as many streams as there are host threads). Otherwise, all returned device_resources will draw their streams from this limited pool.

Limiting the total number of streams can be desirable for a number of reasons, but it is most often used in consuming applications to prevent a large number of host threads from flooding the device with simultaneous requests that may exhaust device memory or other resources.

If called after the first call to raft::device_resources_manager::get_device_resources, no change will be made, and a warning will be emitted.

static inline void set_stream_pools_per_device(
std::size_t num_pools,
std::size_t num_streams = rmm::cuda_stream_pool::default_size
)#

Set the total number and size of CUDA stream pools to be used per device.

Setting the number of stream pools to a non-zero value will provide a pool of stream pools that can be shared among host threads. This can be useful for the same reason it is useful to limit the total number of primary streams assigned to device_resoures for each host thread. Repeated calls to get_device_resources on a given host thread are guaranteed to return device_resources with the same underlying stream pool.

If called after the first call to raft::device_resources_manager::get_device_resources, no change will be made, and a warning will be emitted.

static inline void set_workspace_allocation_limit(
std::size_t memory_limit
)#

Set the maximum size of temporary RAFT workspaces.

Note that this limits only the size of temporary workspace allocations. To cap the device memory generally available for all device allocations made with RMM, use raft::device_manager::set_max_mem_pool_size

If called after the first call to raft::device_resources_manager::get_device_resources, no change will be made, and a warning will be emitted.

static inline void set_max_mem_pool_size(
std::optional<std::size_t> max_mem
)#

Set the maximum size of the device memory pool.

If set to 0, no memory pool will be used. If set to nullopt, the memory pool is allowed to grow to the size of available device memory.

Note that the pool will not actually be created until the first call to raft::device_manager::get_device_resources(device_id), after which it will become the current RMM device memory resource for the indicated device. If the current RMM device memory resource has already been set to some non-default resource, no pool resource will be created and a warning will be emitted. It is assumed that applications which have set a memory resource already wish to manage RMM themselves.

If called after the first call to raft::device_resources_manager::get_device_resources, no change will be made, and a warning will be emitted.

static inline void set_init_mem_pool_size(
std::optional<std::size_t> init_mem
)#

Set the initial size of the device memory pool.

If set to nullopt, the memory pool starts with half of the available device memory.

If called after the first call to raft::device_resources_manager::get_device_resources, no change will be made, and a warning will be emitted.

static inline void set_mem_pool(
std::optional<std::size_t> init_mem = std::nullopt,
std::optional<std::size_t> max_mem = std::nullopt
)#

Request a device memory pool with specified parameters.

This convenience method essentially combines set_init_mem_pool_size and set_max_mem_pool_size. It is provided primarily to allow users who want a memory pool but do not want to choose specific pool sizes to simply call raft::device_manager::set_memory_pool() and enable a memory pool using RMM defaults (initialize with half of available memory, allow to grow to all available memory).

If called after the first call to raft::device_resources_manager::get_device_resources, no change will be made, and a warning will be emitted.

static inline void set_workspace_memory_resource(
std::shared_ptr<rmm::mr::device_memory_resource> mr,
int device_id = device_setter::get_current_device()
)#

Set the workspace memory resource to be used on a specific device.

RAFT device_resources objects can be built with a separate memory resource for allocating temporary workspaces. If a (non-nullptr) memory resource is provided by this setter, it will be used as the workspace memory resource for all device_resources returned for the indicated device.

If called after the first call to raft::device_resources_manager::get_device_resources, no change will be made, and a warning will be emitted.

Resource Functions#

Comms#

#include <raft/core/resource/comms.hpp>

namespace raft::resource

inline bool comms_initialized(
resources const &res
)#
inline comms::comms_t const &get_comms(
resources const &res
)#
inline void set_comms(
resources const &res,
std::shared_ptr<comms::comms_t> communicator
)#

cuBLAS Handle#

#include <raft/core/resource/cublas_handle.hpp>

namespace raft::resource

inline cublasHandle_t get_cublas_handle(
resources const &res
)#

Load a cublasHandle_t from raft res if it exists, otherwise add it and return it.

Parameters:

res – [in] the raft resources object

Returns:

cublas handle

cuBLASLt Handle#

#include <raft/core/resource/cublaslt_handle.hpp>

namespace raft::resource

inline auto get_cublaslt_handle(
resources const &res
) -> cublasLtHandle_t#

Load a cublasLtHandle_t from raft res if it exists, otherwise add it and return it.

Parameters:

res – [in] the raft resources object

Returns:

cublasLt handle

CUDA Stream#

#include <raft/core/resource/cuda_stream.hpp>

namespace raft::resource

inline rmm::cuda_stream_view get_cuda_stream(
resources const &res
)#

Load a rmm::cuda_stream_view from a resources instance (and populate it on the res if needed).

Parameters:

res – raft res object for managing resources

Returns:

inline void set_cuda_stream(
resources const &res,
rmm::cuda_stream_view stream_view
)#

Load a rmm::cuda_stream_view from a resources instance (and populate it on the res if needed).

Parameters:

  • res – [in] raft resources object for managing resources

  • stream_view – cuda stream view

inline void sync_stream(
const resources &res,
rmm::cuda_stream_view stream
)#

synchronize a specific stream

Parameters:

  • res – [in] the raft resources object

  • stream – [in] stream to synchronize

inline void sync_stream(
const resources &res
)#

synchronize main stream on the resources instance

CUDA Stream Pool#

#include <raft/core/resource/cuda_stream_pool.hpp>

namespace raft::resource

inline const rmm::cuda_stream_pool &get_cuda_stream_pool(
const resources &res
)#

Load a cuda_stream_pool, and create a new one if it doesn’t already exist

Parameters:

res – raft res object for managing resources

Returns:

inline void set_cuda_stream_pool(
const resources &res,
std::shared_ptr<rmm::cuda_stream_pool> stream_pool
)#

Explicitly set a stream pool on the current res. Note that this will overwrite an existing stream pool on the res.

Parameters:

  • res –

  • stream_pool –

inline std::size_t get_stream_pool_size(
const resources &res
)#
inline rmm::cuda_stream_view get_stream_from_stream_pool(
const resources &res
)#

return stream from pool

inline rmm::cuda_stream_view get_stream_from_stream_pool(
const resources &res,
std::size_t stream_idx
)#

return stream from pool at index

inline rmm::cuda_stream_view get_next_usable_stream(
const resources &res
)#

return stream from pool if size > 0, else main stream on res

inline rmm::cuda_stream_view get_next_usable_stream(
const resources &res,
std::size_t stream_idx
)#

return stream from pool at index if size > 0, else main stream on res

Parameters:

  • res – [in] the raft resources object

  • stream_idx – [in] the required index of the stream in the stream pool if available

inline void sync_stream_pool(
const resources &res
)#

synchronize the stream pool on the res

Parameters:

res – [in] the raft resources object

inline void sync_stream_pool(
const resources &res,
const std::vector<std::size_t> stream_indices
)#

synchronize subset of stream pool

Parameters:

  • res – [in] the raft resources object

  • stream_indices – [in] the indices of the streams in the stream pool to synchronize

inline void wait_stream_pool_on_stream(
const resources &res
)#

ask stream pool to wait on last event in main stream

Parameters:

res – [in] the raft resources object

cuSolverDn Handle#

#include <raft/core/resource/cusolver_dn_handle.hpp> namespace raft::resource

inline cusolverDnHandle_t get_cusolver_dn_handle(
resources const &res
)#

Load a cusolverSpres_t from raft res if it exists, otherwise add it and return it.

Parameters:

res – [in] the raft resources object

Returns:

cusolver dn handle

class cusolver_dn_resource_factory : public raft::resource::resource_factory#
#include <cusolver_dn_handle.hpp>

Factory that knows how to construct a specific raft::resource to populate the res_t.

Public Functions

inline virtual resource_type get_resource_type(
) override#

Return the resource_type associated with the current factory.

Returns:

resource_type corresponding to the current factory

inline virtual resource *make_resource(
) override#

Construct an instance of the factory’s underlying resource.

Returns:

resource instance

cuSolverSp Handle#

#include <raft/core/resource/cusolver_sp_handle.hpp>

namespace raft::resource

inline cusolverSpHandle_t get_cusolver_sp_handle(
resources const &res
)#

Load a cusolverSpres_t from raft res if it exists, otherwise add it and return it.

Parameters:

res – [in] the raft resources object

Returns:

cusolver sp handle

cuSparse Handle#

#include <raft/core/resource/cusparse_handle.hpp>

namespace raft::resource

inline cusparseHandle_t get_cusparse_handle(
resources const &res
)#

Load a cusparseres_t from raft res if it exists, otherwise add it and return it.

Parameters:

res – [in] the raft resources object

Returns:

cusparse handle

Device ID#

#include <raft/core/resource/device_id.hpp>

namespace raft::resource

inline int get_device_id(
resources const &res
)#

Load a device id from a res (and populate it on the res if needed).

Parameters:

res – raft res object for managing resources

Returns:

device id

Device Memory Resource#

#include <raft/core/resource/device_memory_resource.hpp>

namespace raft::resource

inline auto get_workspace_resource(
resources const &res
) -> rmm::mr::limiting_resource_adaptor<rmm::mr::device_memory_resource>*#

Load a temp workspace resource from a resources instance (and populate it on the res if needed).

Parameters:

res – raft resources object for managing resources

Returns:

device memory resource object

inline auto get_workspace_total_bytes(
resources const &res
) -> size_t#

Get the total size of the workspace resource.

inline auto get_workspace_used_bytes(
resources const &res
) -> size_t#

Get the already allocated size of the workspace resource.

inline auto get_workspace_free_bytes(
resources const &res
) -> size_t#

Get the available size of the workspace resource.

inline void set_workspace_resource(
resources const &res,
std::shared_ptr<rmm::mr::device_memory_resource> mr = {nullptr},
std::optional<std::size_t> allocation_limit = std::nullopt,
std::optional<std::size_t> alignment = std::nullopt
)#

Set a temporary workspace resource on a resources instance.

Parameters:

  • res – raft resources object for managing resources

  • mr – an optional RMM device_memory_resource

  • allocation_limit – the total amount of memory in bytes available to the temporary workspace resources.

  • alignment – optional alignment requirements passed to RMM allocations

inline void set_workspace_to_pool_resource(
resources const &res,
std::optional<std::size_t> allocation_limit = std::nullopt
)#

Set the temporary workspace resource to a pool on top of the global memory resource (rmm::mr::get_current_device_resource().

Parameters:

  • res – raft resources object for managing resources

  • allocation_limit – the total amount of memory in bytes available to the temporary workspace resources; if not provided, a last used or default limit is used.

inline void set_workspace_to_global_resource(
resources const &res,
std::optional<std::size_t> allocation_limit = std::nullopt
)#

Set the temporary workspace resource the same as the global memory resource (rmm::mr::get_current_device_resource().

Note, the workspace resource is always limited; the limit here defines how much of the global memory resource can be consumed by the workspace allocations.

Parameters:

  • res – raft resources object for managing resources

  • allocation_limit – the total amount of memory in bytes available to the temporary workspace resources.

inline auto get_large_workspace_resource(
resources const &res
) -> rmm::mr::device_memory_resource*#
inline void set_large_workspace_resource(
resources const &res,
std::shared_ptr<rmm::mr::device_memory_resource> mr = {nullptr}
)#
class device_memory_resource : public raft::resource::resource#
class limiting_memory_resource : public raft::resource::resource#
class large_workspace_resource_factory : public raft::resource::resource_factory#
#include <device_memory_resource.hpp>

Factory that knows how to construct a specific raft::resource to populate the resources instance.

Public Functions

inline virtual auto get_resource_type(
) -> resource_type override#

Return the resource_type associated with the current factory.

Returns:

resource_type corresponding to the current factory

inline virtual auto make_resource(
) -> resource* override#

Construct an instance of the factory’s underlying resource.

Returns:

resource instance

class workspace_resource_factory : public raft::resource::resource_factory#
#include <device_memory_resource.hpp>

Factory that knows how to construct a specific raft::resource to populate the resources instance.

Public Functions

inline virtual auto get_resource_type(
) -> resource_type override#

Return the resource_type associated with the current factory.

Returns:

resource_type corresponding to the current factory

inline virtual auto make_resource(
) -> resource* override#

Construct an instance of the factory’s underlying resource.

Returns:

resource instance

Public Static Functions

static inline auto default_pool_resource(
std::size_t limit
) -> std::shared_ptr<rmm::mr::device_memory_resource>#

Construct a sensible default pool memory resource.

static inline auto default_plain_resource(
) -> std::shared_ptr<rmm::mr::device_memory_resource>#

Get the global memory resource wrapped into an unmanaged shared_ptr (with no deleter).

Note: the lifetime of the underlying rmm::mr::get_current_device_resource() is managed somewhere else, since it’s passed by a raw pointer. Hence, this shared_ptr wrapper is not allowed to delete the pointer on destruction.

Device Properties#

#include <raft/core/resource/device_properties.hpp>

namespace raft::resource

inline cudaDeviceProp &get_device_properties(
resources const &res
)#

Load a cudaDeviceProp from a res (and populate it on the res if needed).

Parameters:

res – raft res object for managing resources

Returns:

populated cuda device properties instance

class device_properties_resource_factory : public raft::resource::resource_factory#
#include <device_properties.hpp>

Factory that knows how to construct a specific raft::resource to populate the res_t.

Public Functions

inline virtual resource_type get_resource_type(
) override#

Return the resource_type associated with the current factory.

Returns:

resource_type corresponding to the current factory

inline virtual resource *make_resource(
) override#

Construct an instance of the factory’s underlying resource.

Returns:

resource instance

Sub Communicators#

#include <raft/core/resource/sub_comms.hpp>

namespace raft::resource

inline const comms::comms_t &get_subcomm(
const resources &res,
std::string key
)#
inline void set_subcomm(
resources const &res,
std::string key,
std::shared_ptr<comms::comms_t> subcomm
)#

Thrust Exec Policy#

#include <raft/core/resource/thrust_policy.hpp>

namespace raft::resource

inline rmm::exec_policy_nosync &get_thrust_policy(
resources const &res
)#

Load a thrust policy from a res (and populate it on the res if needed).

Parameters:

res – raft res object for managing resources

Returns:

thrust execution policy

Custom runtime-shared resources#

A custom resource is an arbitrary default-constructible C++ class. The consumer of the API can keep such a resource in the raft::resources handle. For example, consider a function that is expected to be called repeatedly and involves a costly kernel configuration. One can cache the kernel configuration in a custom resource. The cost of accessing it is one hashmap lookup.

#include <raft/core/resource/custom_resource.hpp>

namespace raft::resource

template<typename ResourceT>
auto get_custom_resource(
resources const &res
) -> ResourceT*#

Get the custom default-constructible resource if it exists, create it otherwise.

Note: in contrast to the other, hard-coded resources, there’s no information about the custom resources at compile time. Hence, custom resources are kept in a hashmap and looked-up at runtime. This leads to slightly slower access times.

Template Parameters:

ResourceT – the type of the resource; it must be complete and default-constructible.

Parameters:

res – [in] the raft resources object

Returns:

a pointer to the custom resource.

Contents