ONNX Runtime
Loading...
Searching...
No Matches
OrtCompileApi Struct Reference

The OrtCompileApi struct provides functions to compile ONNX models. More...

#include <onnxruntime_c_api.h>

Public Member Functions

OrtModelCompilationOptions
void ReleaseModelCompilationOptions (OrtModelCompilationOptions *input)
 
OrtStatus * CreateModelCompilationOptionsFromSessionOptions (const OrtEnv *env, const OrtSessionOptions *session_options, OrtModelCompilationOptions **out)
 Creates an OrtModelCompilationOptions object from an existing OrtSessionOptions object.
 
OrtStatus * ModelCompilationOptions_SetInputModelPath (OrtModelCompilationOptions *model_compile_options, const char *input_model_path)
 Sets the file path to the input ONNX model to compile.
 
OrtStatus * ModelCompilationOptions_SetInputModelFromBuffer (OrtModelCompilationOptions *model_compile_options, const void *input_model_data, size_t input_model_data_size)
 Sets the buffer that stores the bytes of the loaded ONNX model to compile.
 
OrtStatus * ModelCompilationOptions_SetOutputModelPath (OrtModelCompilationOptions *model_compile_options, const char *output_model_path)
 Sets the file path for the output ONNX model generated by CompileModel.
 
OrtStatus * ModelCompilationOptions_SetOutputModelExternalInitializersFile (OrtModelCompilationOptions *model_compile_options, const char *external_initializers_file_path, size_t external_initializers_size_threshold)
 Optionally sets the file that should store external initializers for the compiled ONNX model. If not set, initializers are stored within the model.
 
OrtStatus * ModelCompilationOptions_SetOutputModelBuffer (OrtModelCompilationOptions *model_compile_options, OrtAllocator *allocator, void **output_model_buffer_ptr, size_t *output_model_buffer_size_ptr)
 Configures model compilation to store the output compiled ONNX model in a buffer.
 
OrtStatus * ModelCompilationOptions_SetEpContextEmbedMode (OrtModelCompilationOptions *model_compile_options, bool embed_ep_context_in_model)
 Enables or disables the embedding of EPContext binary data into the ep_cache_context attribute of EPContext nodes. Defaults to false.
 
OrtStatus * CompileModel (const OrtEnv *env, const OrtModelCompilationOptions *model_options)
 Compiles an input ONNX model with the given compilation options.
 
OrtStatus * ModelCompilationOptions_SetFlags (OrtModelCompilationOptions *model_compile_options, uint32_t flags)
 Sets flags from OrtCompileApiFlags that represent one or more boolean options to enable.
 
OrtStatus * ModelCompilationOptions_SetEpContextBinaryInformation (OrtModelCompilationOptions *model_compile_options, const char *output_directory, const char *model_name)
 
OrtStatus * ModelCompilationOptions_SetGraphOptimizationLevel (OrtModelCompilationOptions *model_compile_options, GraphOptimizationLevel graph_optimization_level)
 
OrtStatus * ModelCompilationOptions_SetOutputModelWriteFunc (OrtModelCompilationOptions *model_compile_options, OrtWriteBufferFunc write_func, void *state)
 Sets a OrtWriteBufferFunc function that is called by ORT to write out the output model's serialized ONNX bytes.
 
OrtStatus * ModelCompilationOptions_SetOutputModelGetInitializerLocationFunc (OrtModelCompilationOptions *model_compile_options, OrtGetInitializerLocationFunc get_initializer_location_func, void *state)
 Sets a OrtGetInitializerLocationFunc function that is called by ORT for every initializer in the generated model. Allows implementer to specify whether initializers should be stored within the model or externally.
 
OrtStatus * ModelCompilationOptions_SetInputModel (OrtModelCompilationOptions *model_compile_options, const OrtModel *model)
 Sets the OrtModel to compile.
 
OrtStatus * ModelCompilationOptions_SetWeightlessEnabled (OrtModelCompilationOptions *model_compile_options, bool use_weightless)
 Enable or disable weightless mode for model compilation.
 
OrtStatus * ModelCompilationOptions_SetEpContextDataWriteFunc (OrtModelCompilationOptions *model_compile_options, OrtWriteNamedBufferFunc write_func, void *state)
 Register a callback that receives external EPContext binary data during model compilation.
 
OrtStatus * ModelCompilationOptions_SetOutputModelExternalInitializersBuffer (OrtModelCompilationOptions *model_compile_options, const char *logical_file_name, size_t external_initializers_size_threshold, OrtAllocator *allocator, void **output_buffer_ptr, size_t *output_buffer_size_ptr)
 Store external initializers for the compiled model in one caller-owned buffer.
 
OrtStatus * ModelCompilationOptions_SetOutputModelExternalInitializersAlignment (OrtModelCompilationOptions *model_compile_options, size_t alignment, size_t minimum_size)
 Configure additional alignment for externalized initializer offsets.
 

Detailed Description

The OrtCompileApi struct provides functions to compile ONNX models.

Execution providers that support compilation fuse a subgraph into an EPContext node that wraps a provider-specific binary representation of the subgraph. For more details about the EPContext design, refer to: EPContext design document.

Example (error handling not shown): OrtStatus* status = NULL; OrtCompileApi* compile_api = ort_api->GetCompileApi(); OrtModelCompilationOptions* compile_options = NULL;

status = compile_api->CreateModelCompilationOptionsFromSessionOptions(env, session_options, &compile_options); status = compile_api->ModelCompilationOptions_SetInputModelPath(compile_options, ORT_TSTR("model.onnx")); status = compile_api->ModelCompilationOptions_SetOutputModelPath(compile_options, ORT_TSTR("model.compiled.onnx")); status = compile_api->CompileModel(env, compile_options); compile_api->ReleaseModelCompilationOptions(compile_options);

Since
Version 1.22.

Member Function Documentation

◆ CompileModel()

OrtStatus * OrtCompileApi::CompileModel ( const OrtEnv *  env,
const OrtModelCompilationOptions *  model_options 
)

Compiles an input ONNX model with the given compilation options.

Parameters
[in]envOrtEnv object.
[in]model_optionsThe compilation options that defines compilation options for a model.

Returns
If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since
Version 1.22.

◆ CreateModelCompilationOptionsFromSessionOptions()

OrtStatus * OrtCompileApi::CreateModelCompilationOptionsFromSessionOptions ( const OrtEnv *  env,
const OrtSessionOptions *  session_options,
OrtModelCompilationOptions **  out 
)

Creates an OrtModelCompilationOptions object from an existing OrtSessionOptions object.

An OrtModelCompilationOptions object contains the settings used to generate a compiled ONNX model. The OrtSessionOptions object has the execution providers with which the model will be compiled.

ReleaseOrtModelCompilationsOptions must be called to free the OrtModelCompilationOptions after calling CompileModel.

Note
By default, the GraphOptimizationLevel is set to ORT_DISABLE_ALL. Use ModelCompilationOptions_SetGraphOptimizationLevel to enable graph optimizations.
Parameters
[in]envOrtEnv object.
[in]session_optionsThe OrtSessionOptions instance from which to create the OrtModelCompilationOptions.
[out]outThe created OrtModelCompilationOptions instance.
Returns
If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.
Since
Version 1.22.

◆ ModelCompilationOptions_SetEpContextBinaryInformation()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetEpContextBinaryInformation ( OrtModelCompilationOptions *  model_compile_options,
const char *  output_directory,
const char *  model_name 
)

Sets information related to EP context binary file.

EP uses this information to decide the location and context binary file name. Used while compiling model with input and output in memory buffer

Parameters
[in]model_compile_optionsThe OrtModelCompilationOptions instance.
[in]output_directoryNull terminated string of the path (wchar on Windows, char otherwise).
[in]model_nameNull terminated string of the model name (wchar on Windows, char otherwise).

Returns
If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since
Version 1.23.

◆ ModelCompilationOptions_SetEpContextDataWriteFunc()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetEpContextDataWriteFunc ( OrtModelCompilationOptions *  model_compile_options,
OrtWriteNamedBufferFunc  write_func,
void *  state 
)

Register a callback that receives external EPContext binary data during model compilation.

Execution providers that support external EPContext data retrieve this callback from an OrtEpContextConfig. The callback is used only when EPContext data is not embedded in the generated ONNX model. Passing NULL clears the callback and its state. If a compiling EP does not advertise WRITE support, compilation fails before that EP's Compile() call.

Parameters
[in]model_compile_optionsModel compilation options.
[in]write_funcWrite callback, or NULL to clear a previously registered callback.
[in]stateApplication-owned state passed to write_func. Ignored when write_func is NULL.

Returns
If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since
Version 1.31.

◆ ModelCompilationOptions_SetEpContextEmbedMode()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetEpContextEmbedMode ( OrtModelCompilationOptions *  model_compile_options,
bool  embed_ep_context_in_model 
)

Enables or disables the embedding of EPContext binary data into the ep_cache_context attribute of EPContext nodes. Defaults to false.

If enabled, the ep_cache_context attribute of EPContext nodes will store the context binary data, which may include weights for compiled subgraphs.

If disabled, the ep_cache_context attribute of EPContext nodes will contain the path to the file containing the context binary data. The path is set by the execution provider creating the EPContext node.

More details relate to EPContext design refers to: EPContext design document.

Parameters
[in]model_compile_optionsThe OrtModelCompilationOptions instance.
[in]embed_ep_context_in_modelTrue to embed EPContext binary data into the EPContext node ep_cache_context attributes.

Returns
If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since
Version 1.22.

◆ ModelCompilationOptions_SetFlags()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetFlags ( OrtModelCompilationOptions *  model_compile_options,
uint32_t  flags 
)

Sets flags from OrtCompileApiFlags that represent one or more boolean options to enable.

Parameters
[in]model_compile_optionsThe OrtModelCompilationOptions instance.
[in]flagsbitwise OR of flags in OrtCompileApiFlags to enable.

Returns
If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since
Version 1.23.

◆ ModelCompilationOptions_SetGraphOptimizationLevel()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetGraphOptimizationLevel ( OrtModelCompilationOptions *  model_compile_options,
GraphOptimizationLevel  graph_optimization_level 
)

Set the graph optimization level.

Parameters
[in]model_compile_optionsThe OrtModelCompilationOptions instance.
[in]graph_optimization_levelThe graph optimization level.

Returns
If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since
Version 1.23.

◆ ModelCompilationOptions_SetInputModel()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetInputModel ( OrtModelCompilationOptions *  model_compile_options,
const OrtModel *  model 
)

Sets the OrtModel to compile.

Sets an OrtModel created via the Model Editor API as the input for compilation.

The input model's source (file path, memory buffer, or OrtModel) must be set with one of: ModelCompilationOptions_SetInputModelPath, ModelCompilationOptions_SetInputModelFromBuffer, or ModelCompilationOptions_SetInputModel.

The OrtModel must have a complete graph with inputs, outputs, and nodes defined. The caller retains ownership of the OrtModel and must not release it until after CompileModel returns.

Parameters
[in]model_compile_optionsThe OrtModelCompilationOptions instance.
[in]modelThe OrtModel to compile. The model is borrowed (not copied or owned).

Returns
If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since
Version 1.24.

◆ ModelCompilationOptions_SetInputModelFromBuffer()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetInputModelFromBuffer ( OrtModelCompilationOptions *  model_compile_options,
const void *  input_model_data,
size_t  input_model_data_size 
)

Sets the buffer that stores the bytes of the loaded ONNX model to compile.

The input model's location (e.g., file path or memory buffer) must be set with either ModelCompilationOptions_SetInputModelPath or ModelCompilationOptions_SetInputModelFromBuffer.

Parameters
[in]model_compile_optionsThe OrtModelCompilationOptions instance.
[in]input_model_dataBuffer containing the loaded ONNX model bytes.
[in]input_model_data_sizeThe number of bytes in the input_model_data buffer.

Returns
If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since
Version 1.22.

◆ ModelCompilationOptions_SetInputModelPath()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetInputModelPath ( OrtModelCompilationOptions *  model_compile_options,
const char *  input_model_path 
)

Sets the file path to the input ONNX model to compile.

The input model's location (e.g., file path or memory buffer) must be set with either ModelCompilationOptions_SetInputModelPath or ModelCompilationOptions_SetInputModelFromBuffer.

Parameters
[in]model_compile_optionsThe OrtModelCompilationOptions instance.
[in]input_model_pathNull terminated string of the path (wchar on Windows, char otherwise).

Returns
If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since
Version 1.22.

◆ ModelCompilationOptions_SetOutputModelBuffer()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelBuffer ( OrtModelCompilationOptions *  model_compile_options,
OrtAllocator *  allocator,
void **  output_model_buffer_ptr,
size_t *  output_model_buffer_size_ptr 
)

Configures model compilation to store the output compiled ONNX model in a buffer.

The caller passes an OrtAllocator that ONNX Runtime uses to allocate memory for the buffer.

The output model's location (e.g., file path or memory buffer) can be set with either ModelCompilationOptions_SetOutputModelPath or ModelCompilationOptions_SetOutputModelBuffer.

If the output model's location is not set, ONNX Runtime will generate an output file with a path based on the input model's file path. Examples: /Path/my_model.onnx -> /Path/my_model_ctx.onnx /Path/my_model -> /Path/my_model_ctx.onnx

Parameters
[in]model_compile_optionsThe OrtModelCompilationOptions instance.
[in]allocatorThe allocator used to allocate the buffer for the compiled model.
[out]output_model_buffer_ptrPointer to the buffer that stores the compiled model.
[out]output_model_buffer_size_ptrPointer set to the size of output model in bytes.

Returns
If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since
Version 1.22.

◆ ModelCompilationOptions_SetOutputModelExternalInitializersAlignment()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelExternalInitializersAlignment ( OrtModelCompilationOptions *  model_compile_options,
size_t  alignment,
size_t  minimum_size 
)

Configure additional alignment for externalized initializer offsets.

Applies to file and buffer destinations for initializers at least minimum_size bytes. An alignment of zero disables the additional policy. Otherwise, alignment must be a power of two. By default, offsets for initializers of at least 1 MiB (1048576 bytes) are aligned to 4096 bytes, even if this setter is never called. Natural alignment for each tensor's element type is always applied, including when the additional policy is disabled. These settings align offsets, not the allocator-provided buffer address.

Parameters
[in]model_compile_optionsThe OrtModelCompilationOptions instance.
[in]alignmentRequired byte alignment, or zero to disable.
[in]minimum_sizeMinimum initializer size at which alignment is applied.
Returns
If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.
Since
Version 1.31.

◆ ModelCompilationOptions_SetOutputModelExternalInitializersBuffer()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelExternalInitializersBuffer ( OrtModelCompilationOptions *  model_compile_options,
const char *  logical_file_name,
size_t  external_initializers_size_threshold,
OrtAllocator *  allocator,
void **  output_buffer_ptr,
size_t *  output_buffer_size_ptr 
)

Store external initializers for the compiled model in one caller-owned buffer.

This destination replaces any external-initializer file or callback destination configured previously. The logical file name must be a non-empty relative path. Leading current-directory prefixes (./ or .\\) are removed before the name is recorded in each externalized TensorProto. This setter only configures the destination; CompileModel allocates the buffer using allocator. The allocator must provide host-accessible memory (CPU or host-accessible/pinned device memory). Device-only allocators are rejected with ORT_INVALID_ARGUMENT. On successful CompileModel completion, the caller owns the allocated buffer and must release it with allocator. If no data is externalized, the output buffer is NULL and its size is zero. Only main-graph initializers are externalized to this buffer; subgraph initializers remain embedded in the model. If CompileModel fails, both outputs are unchanged and temporary allocations are freed. The allocator and output pointer locations must remain valid until CompileModel returns. If the model also uses an output buffer, its buffer and size output locations must each be distinct from those provided here. CompileModel rejects aliased output locations with ORT_INVALID_ARGUMENT.

The output model may be written to a file, buffer, or write callback. When the output model is written to a file, the caller is responsible for persisting or otherwise supplying this buffer under logical_file_name when the model is loaded.

Parameters
[in]model_compile_optionsThe OrtModelCompilationOptions instance.
[in]logical_file_nameLogical external-data file name stored in the model.
[in]external_initializers_size_thresholdInitializers at least this size are externalized.
[in]allocatorAllocator providing host-accessible memory for the output buffer.
[out]output_buffer_ptrReceives the allocated buffer, or NULL when no data is externalized.
[out]output_buffer_size_ptrReceives the allocated buffer size.
Returns
If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.
Since
Version 1.31.

◆ ModelCompilationOptions_SetOutputModelExternalInitializersFile()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelExternalInitializersFile ( OrtModelCompilationOptions *  model_compile_options,
const char *  external_initializers_file_path,
size_t  external_initializers_size_threshold 
)

Optionally sets the file that should store external initializers for the compiled ONNX model. If not set, initializers are stored within the model.

Only initializers for nodes that were not compiled are stored in the external initializers file. Compiled nodes contain their initializer data within the ep_cache_context attribute of EPContext nodes. Refer to ModelCompilationOptions_SetEpContextEmbedMode.

Parameters
[in]model_compile_optionsThe OrtModelCompilationOptions instance.
[in]external_initializers_file_pathNull terminated string of the path to the file.
[in]external_initializers_size_thresholdInitializers larger than this threshold are stored in the file.

Returns
If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since
Version 1.22.

◆ ModelCompilationOptions_SetOutputModelGetInitializerLocationFunc()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelGetInitializerLocationFunc ( OrtModelCompilationOptions *  model_compile_options,
OrtGetInitializerLocationFunc  get_initializer_location_func,
void *  state 
)

Sets a OrtGetInitializerLocationFunc function that is called by ORT for every initializer in the generated model. Allows implementer to specify whether initializers should be stored within the model or externally.

Parameters
[in]model_compile_optionsThe OrtModelCompilationOptions instance.
[in]get_initializer_location_funcThe OrtGetInitializerLocationFunc function called by ORT when to determine the location of the initializer.
[in]stateOpaque state passed as the first argument to OrtGetInitializerLocationFunc. Can be NULL.

Returns
If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since
Version 1.23.

◆ ModelCompilationOptions_SetOutputModelPath()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelPath ( OrtModelCompilationOptions *  model_compile_options,
const char *  output_model_path 
)

Sets the file path for the output ONNX model generated by CompileModel.

The output model's location (e.g., file path or memory buffer) can be set with either ModelCompilationOptions_SetOutputModelPath or ModelCompilationOptions_SetOutputModelBuffer.

If the output model's location is not set, ONNX Runtime will generate an output file with a path based on the input model's file path. Examples: /Path/my_model.onnx -> /Path/my_model_ctx.onnx /Path/my_model -> /Path/my_model_ctx.onnx

Parameters
[in]model_compile_optionsThe OrtModelCompilationOptions instance.
[in]output_model_pathNull terminated string of the path (wchar on Windows, char otherwise).

Returns
If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since
Version 1.22.

◆ ModelCompilationOptions_SetOutputModelWriteFunc()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelWriteFunc ( OrtModelCompilationOptions *  model_compile_options,
OrtWriteBufferFunc  write_func,
void *  state 
)

Sets a OrtWriteBufferFunc function that is called by ORT to write out the output model's serialized ONNX bytes.

The provided write function may be called repeatedly until then entire output model has been written out. Each call to the write function is expected to consume the entire input buffer.

The output model's destination (e.g., file path, memory buffer, or stream) can be set with any of the functions that begin with ModelCompilationOptions_SetOutputModel____.

Parameters
[in]model_compile_optionsThe OrtModelCompilationOptions instance.
[in]write_funcThe OrtWriteBufferFunc function called by ORT when writing out the model.
[in]stateOpaque state passed as the first argument to OrtWriteBufferFunc. Can be NULL.

Returns
If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since
Version 1.23.

◆ ModelCompilationOptions_SetWeightlessEnabled()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabled ( OrtModelCompilationOptions *  model_compile_options,
bool  use_weightless 
)

Enable or disable weightless mode for model compilation.

When enabled, the compiled EPContext model will not embed constant initializer data in the EP's compiled binary. Instead, the initializer data must be provided when creating a session from the compiled model, either from the source model (via the "onnx_model_filename" EPContext node attribute or the "ep.context_source_model_path" session option) or from externalized weights.

This enables smaller compiled models and allows sharing initializer data across multiple compiled model variants (e.g., multi-platform caches for different hardware generations).

ORT verifies that the target EP supports weightless mode during CompileModel() by calling OrtEp::GetWeightlessSupport(). If the EP does not support weightless mode, CompileModel() returns an error.

Parameters
[in]model_compile_optionsThe OrtModelCompilationOptions instance.
[in]use_weightlessIf true, enable weightless mode. If false, disable (default behavior).

Returns
If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since
Version 1.29.

◆ ReleaseModelCompilationOptions()

void OrtCompileApi::ReleaseModelCompilationOptions ( OrtModelCompilationOptions *  input)