ONNX Runtime
Loading...
Searching...
No Matches
OrtCUDAProviderOptions Struct Reference

CUDA Provider Options. More...

#include <onnxruntime_c_api.h>

Public Member Functions

 OrtCUDAProviderOptions ()
 

Public Attributes

int device_id
 CUDA device Id Defaults to 0.
 
OrtCudnnConvAlgoSearch cudnn_conv_algo_search
 CUDA Convolution algorithm search configuration. See enum OrtCudnnConvAlgoSearch for more details. Defaults to OrtCudnnConvAlgoSearchExhaustive.
 
size_t gpu_mem_limit
 CUDA memory limit (To use all possible memory pass in maximum size_t) Defaults to SIZE_MAX.
 
int arena_extend_strategy
 Strategy used to grow the memory arena 0 = kNextPowerOfTwo
1 = kSameAsRequested
Defaults to 0.
 
int do_copy_in_default_stream
 Flag indicating if copying needs to take place on the same stream as the compute stream in the CUDA EP 0 = Use separate streams for copying and compute. 1 = Use the same stream for copying and compute. Defaults to 1. WARNING: Setting this to 0 may result in data races for some models. Please see issue #4829 for more details.
 
int has_user_compute_stream
 Flag indicating if there is a user provided compute stream Defaults to 0.
 
void * user_compute_stream
 User provided compute stream. If provided, please set has_user_compute_stream to 1.
 
OrtArenaCfgdefault_memory_arena_cfg
 CUDA memory arena configuration parameters.
 
int tunable_op_enable
 Enable TunableOp for using. Set it to 1/0 to enable/disable TunableOp. Otherwise, it is disabled by default. This option can be overridden by environment variable ORT_CUDA_TUNABLE_OP_ENABLE.
 
int tunable_op_tuning_enable
 Enable TunableOp for tuning. Set it to 1/0 to enable/disable TunableOp tuning. Otherwise, it is disabled by default. This option can be overridden by environment variable ORT_CUDA_TUNABLE_OP_TUNING_ENABLE.
 
int tunable_op_max_tuning_duration_ms
 Max tuning duration time limit for each instance of TunableOp. Defaults to 0 to disable the limit.
 

Detailed Description

Constructor & Destructor Documentation

◆ OrtCUDAProviderOptions()

OrtCUDAProviderOptions::OrtCUDAProviderOptions ( )
inline

Member Data Documentation

◆ arena_extend_strategy

int OrtCUDAProviderOptions::arena_extend_strategy

Strategy used to grow the memory arena 0 = kNextPowerOfTwo
1 = kSameAsRequested
Defaults to 0.

Note
If a OrtArenaCfg has been applied, it will override this field

◆ cudnn_conv_algo_search

OrtCudnnConvAlgoSearch OrtCUDAProviderOptions::cudnn_conv_algo_search

CUDA Convolution algorithm search configuration. See enum OrtCudnnConvAlgoSearch for more details. Defaults to OrtCudnnConvAlgoSearchExhaustive.

◆ default_memory_arena_cfg

OrtArenaCfg* OrtCUDAProviderOptions::default_memory_arena_cfg

CUDA memory arena configuration parameters.

◆ device_id

int OrtCUDAProviderOptions::device_id

CUDA device Id Defaults to 0.

◆ do_copy_in_default_stream

int OrtCUDAProviderOptions::do_copy_in_default_stream

Flag indicating if copying needs to take place on the same stream as the compute stream in the CUDA EP 0 = Use separate streams for copying and compute. 1 = Use the same stream for copying and compute. Defaults to 1. WARNING: Setting this to 0 may result in data races for some models. Please see issue #4829 for more details.

◆ gpu_mem_limit

size_t OrtCUDAProviderOptions::gpu_mem_limit

CUDA memory limit (To use all possible memory pass in maximum size_t) Defaults to SIZE_MAX.

Note
If a OrtArenaCfg has been applied, it will override this field

◆ has_user_compute_stream

int OrtCUDAProviderOptions::has_user_compute_stream

Flag indicating if there is a user provided compute stream Defaults to 0.

◆ tunable_op_enable

int OrtCUDAProviderOptions::tunable_op_enable

Enable TunableOp for using. Set it to 1/0 to enable/disable TunableOp. Otherwise, it is disabled by default. This option can be overridden by environment variable ORT_CUDA_TUNABLE_OP_ENABLE.

◆ tunable_op_max_tuning_duration_ms

int OrtCUDAProviderOptions::tunable_op_max_tuning_duration_ms

Max tuning duration time limit for each instance of TunableOp. Defaults to 0 to disable the limit.

◆ tunable_op_tuning_enable

int OrtCUDAProviderOptions::tunable_op_tuning_enable

Enable TunableOp for tuning. Set it to 1/0 to enable/disable TunableOp tuning. Otherwise, it is disabled by default. This option can be overridden by environment variable ORT_CUDA_TUNABLE_OP_TUNING_ENABLE.

◆ user_compute_stream

void* OrtCUDAProviderOptions::user_compute_stream

User provided compute stream. If provided, please set has_user_compute_stream to 1.