![]() |
ONNX Runtime
|
The OrtCompileApi struct provides functions to compile ONNX models. More...
#include <onnxruntime_c_api.h>
Public Member Functions | |
OrtModelCompilationOptions | |
| void | ReleaseModelCompilationOptions (OrtModelCompilationOptions *input) |
| OrtStatus * | CreateModelCompilationOptionsFromSessionOptions (const OrtEnv *env, const OrtSessionOptions *session_options, OrtModelCompilationOptions **out) |
| Creates an OrtModelCompilationOptions object from an existing OrtSessionOptions object. | |
| OrtStatus * | ModelCompilationOptions_SetInputModelPath (OrtModelCompilationOptions *model_compile_options, const char *input_model_path) |
| Sets the file path to the input ONNX model to compile. | |
| OrtStatus * | ModelCompilationOptions_SetInputModelFromBuffer (OrtModelCompilationOptions *model_compile_options, const void *input_model_data, size_t input_model_data_size) |
| Sets the buffer that stores the bytes of the loaded ONNX model to compile. | |
| OrtStatus * | ModelCompilationOptions_SetOutputModelPath (OrtModelCompilationOptions *model_compile_options, const char *output_model_path) |
| Sets the file path for the output ONNX model generated by CompileModel. | |
| OrtStatus * | ModelCompilationOptions_SetOutputModelExternalInitializersFile (OrtModelCompilationOptions *model_compile_options, const char *external_initializers_file_path, size_t external_initializers_size_threshold) |
| Optionally sets the file that should store external initializers for the compiled ONNX model. If not set, initializers are stored within the model. | |
| OrtStatus * | ModelCompilationOptions_SetOutputModelBuffer (OrtModelCompilationOptions *model_compile_options, OrtAllocator *allocator, void **output_model_buffer_ptr, size_t *output_model_buffer_size_ptr) |
| Configures model compilation to store the output compiled ONNX model in a buffer. | |
| OrtStatus * | ModelCompilationOptions_SetEpContextEmbedMode (OrtModelCompilationOptions *model_compile_options, bool embed_ep_context_in_model) |
Enables or disables the embedding of EPContext binary data into the ep_cache_context attribute of EPContext nodes. Defaults to false. | |
| OrtStatus * | CompileModel (const OrtEnv *env, const OrtModelCompilationOptions *model_options) |
| Compiles an input ONNX model with the given compilation options. | |
| OrtStatus * | ModelCompilationOptions_SetFlags (OrtModelCompilationOptions *model_compile_options, uint32_t flags) |
| Sets flags from OrtCompileApiFlags that represent one or more boolean options to enable. | |
| OrtStatus * | ModelCompilationOptions_SetEpContextBinaryInformation (OrtModelCompilationOptions *model_compile_options, const char *output_directory, const char *model_name) |
| OrtStatus * | ModelCompilationOptions_SetGraphOptimizationLevel (OrtModelCompilationOptions *model_compile_options, GraphOptimizationLevel graph_optimization_level) |
| OrtStatus * | ModelCompilationOptions_SetOutputModelWriteFunc (OrtModelCompilationOptions *model_compile_options, OrtWriteBufferFunc write_func, void *state) |
| Sets a OrtWriteBufferFunc function that is called by ORT to write out the output model's serialized ONNX bytes. | |
| OrtStatus * | ModelCompilationOptions_SetOutputModelGetInitializerLocationFunc (OrtModelCompilationOptions *model_compile_options, OrtGetInitializerLocationFunc get_initializer_location_func, void *state) |
| Sets a OrtGetInitializerLocationFunc function that is called by ORT for every initializer in the generated model. Allows implementer to specify whether initializers should be stored within the model or externally. | |
| OrtStatus * | ModelCompilationOptions_SetInputModel (OrtModelCompilationOptions *model_compile_options, const OrtModel *model) |
| Sets the OrtModel to compile. | |
| OrtStatus * | ModelCompilationOptions_SetWeightlessEnabled (OrtModelCompilationOptions *model_compile_options, bool use_weightless) |
| Enable or disable weightless mode for model compilation. | |
| OrtStatus * | ModelCompilationOptions_SetEpContextDataWriteFunc (OrtModelCompilationOptions *model_compile_options, OrtWriteNamedBufferFunc write_func, void *state) |
| Register a callback that receives external EPContext binary data during model compilation. | |
| OrtStatus * | ModelCompilationOptions_SetOutputModelExternalInitializersBuffer (OrtModelCompilationOptions *model_compile_options, const char *logical_file_name, size_t external_initializers_size_threshold, OrtAllocator *allocator, void **output_buffer_ptr, size_t *output_buffer_size_ptr) |
| Store external initializers for the compiled model in one caller-owned buffer. | |
| OrtStatus * | ModelCompilationOptions_SetOutputModelExternalInitializersAlignment (OrtModelCompilationOptions *model_compile_options, size_t alignment, size_t minimum_size) |
| Configure additional alignment for externalized initializer offsets. | |
The OrtCompileApi struct provides functions to compile ONNX models.
Execution providers that support compilation fuse a subgraph into an EPContext node that wraps a provider-specific binary representation of the subgraph. For more details about the EPContext design, refer to: EPContext design document.
Example (error handling not shown): OrtStatus* status = NULL; OrtCompileApi* compile_api = ort_api->GetCompileApi(); OrtModelCompilationOptions* compile_options = NULL;
status = compile_api->CreateModelCompilationOptionsFromSessionOptions(env, session_options, &compile_options); status = compile_api->ModelCompilationOptions_SetInputModelPath(compile_options, ORT_TSTR("model.onnx")); status = compile_api->ModelCompilationOptions_SetOutputModelPath(compile_options, ORT_TSTR("model.compiled.onnx")); status = compile_api->CompileModel(env, compile_options); compile_api->ReleaseModelCompilationOptions(compile_options);
| OrtStatus * OrtCompileApi::CompileModel | ( | const OrtEnv * | env, |
| const OrtModelCompilationOptions * | model_options | ||
| ) |
Compiles an input ONNX model with the given compilation options.
| [in] | env | OrtEnv object. |
| [in] | model_options | The compilation options that defines compilation options for a model. |
| OrtStatus * OrtCompileApi::CreateModelCompilationOptionsFromSessionOptions | ( | const OrtEnv * | env, |
| const OrtSessionOptions * | session_options, | ||
| OrtModelCompilationOptions ** | out | ||
| ) |
Creates an OrtModelCompilationOptions object from an existing OrtSessionOptions object.
An OrtModelCompilationOptions object contains the settings used to generate a compiled ONNX model. The OrtSessionOptions object has the execution providers with which the model will be compiled.
ReleaseOrtModelCompilationsOptions must be called to free the OrtModelCompilationOptions after calling CompileModel.
| [in] | env | OrtEnv object. |
| [in] | session_options | The OrtSessionOptions instance from which to create the OrtModelCompilationOptions. |
| [out] | out | The created OrtModelCompilationOptions instance.
|
| OrtStatus * OrtCompileApi::ModelCompilationOptions_SetEpContextBinaryInformation | ( | OrtModelCompilationOptions * | model_compile_options, |
| const char * | output_directory, | ||
| const char * | model_name | ||
| ) |
Sets information related to EP context binary file.
EP uses this information to decide the location and context binary file name. Used while compiling model with input and output in memory buffer
| [in] | model_compile_options | The OrtModelCompilationOptions instance. |
| [in] | output_directory | Null terminated string of the path (wchar on Windows, char otherwise). |
| [in] | model_name | Null terminated string of the model name (wchar on Windows, char otherwise). |
| OrtStatus * OrtCompileApi::ModelCompilationOptions_SetEpContextDataWriteFunc | ( | OrtModelCompilationOptions * | model_compile_options, |
| OrtWriteNamedBufferFunc | write_func, | ||
| void * | state | ||
| ) |
Register a callback that receives external EPContext binary data during model compilation.
Execution providers that support external EPContext data retrieve this callback from an OrtEpContextConfig. The callback is used only when EPContext data is not embedded in the generated ONNX model. Passing NULL clears the callback and its state. If a compiling EP does not advertise WRITE support, compilation fails before that EP's Compile() call.
| [in] | model_compile_options | Model compilation options. |
| [in] | write_func | Write callback, or NULL to clear a previously registered callback. |
| [in] | state | Application-owned state passed to write_func. Ignored when write_func is NULL. |
| OrtStatus * OrtCompileApi::ModelCompilationOptions_SetEpContextEmbedMode | ( | OrtModelCompilationOptions * | model_compile_options, |
| bool | embed_ep_context_in_model | ||
| ) |
Enables or disables the embedding of EPContext binary data into the ep_cache_context attribute of EPContext nodes. Defaults to false.
If enabled, the ep_cache_context attribute of EPContext nodes will store the context binary data, which may include weights for compiled subgraphs.
If disabled, the ep_cache_context attribute of EPContext nodes will contain the path to the file containing the context binary data. The path is set by the execution provider creating the EPContext node.
More details relate to EPContext design refers to: EPContext design document.
| [in] | model_compile_options | The OrtModelCompilationOptions instance. |
| [in] | embed_ep_context_in_model | True to embed EPContext binary data into the EPContext node ep_cache_context attributes. |
| OrtStatus * OrtCompileApi::ModelCompilationOptions_SetFlags | ( | OrtModelCompilationOptions * | model_compile_options, |
| uint32_t | flags | ||
| ) |
Sets flags from OrtCompileApiFlags that represent one or more boolean options to enable.
| [in] | model_compile_options | The OrtModelCompilationOptions instance. |
| [in] | flags | bitwise OR of flags in OrtCompileApiFlags to enable. |
| OrtStatus * OrtCompileApi::ModelCompilationOptions_SetGraphOptimizationLevel | ( | OrtModelCompilationOptions * | model_compile_options, |
| GraphOptimizationLevel | graph_optimization_level | ||
| ) |
Set the graph optimization level.
| [in] | model_compile_options | The OrtModelCompilationOptions instance. |
| [in] | graph_optimization_level | The graph optimization level. |
| OrtStatus * OrtCompileApi::ModelCompilationOptions_SetInputModel | ( | OrtModelCompilationOptions * | model_compile_options, |
| const OrtModel * | model | ||
| ) |
Sets the OrtModel to compile.
Sets an OrtModel created via the Model Editor API as the input for compilation.
The input model's source (file path, memory buffer, or OrtModel) must be set with one of: ModelCompilationOptions_SetInputModelPath, ModelCompilationOptions_SetInputModelFromBuffer, or ModelCompilationOptions_SetInputModel.
The OrtModel must have a complete graph with inputs, outputs, and nodes defined. The caller retains ownership of the OrtModel and must not release it until after CompileModel returns.
| [in] | model_compile_options | The OrtModelCompilationOptions instance. |
| [in] | model | The OrtModel to compile. The model is borrowed (not copied or owned). |
| OrtStatus * OrtCompileApi::ModelCompilationOptions_SetInputModelFromBuffer | ( | OrtModelCompilationOptions * | model_compile_options, |
| const void * | input_model_data, | ||
| size_t | input_model_data_size | ||
| ) |
Sets the buffer that stores the bytes of the loaded ONNX model to compile.
The input model's location (e.g., file path or memory buffer) must be set with either ModelCompilationOptions_SetInputModelPath or ModelCompilationOptions_SetInputModelFromBuffer.
| [in] | model_compile_options | The OrtModelCompilationOptions instance. |
| [in] | input_model_data | Buffer containing the loaded ONNX model bytes. |
| [in] | input_model_data_size | The number of bytes in the input_model_data buffer. |
| OrtStatus * OrtCompileApi::ModelCompilationOptions_SetInputModelPath | ( | OrtModelCompilationOptions * | model_compile_options, |
| const char * | input_model_path | ||
| ) |
Sets the file path to the input ONNX model to compile.
The input model's location (e.g., file path or memory buffer) must be set with either ModelCompilationOptions_SetInputModelPath or ModelCompilationOptions_SetInputModelFromBuffer.
| [in] | model_compile_options | The OrtModelCompilationOptions instance. |
| [in] | input_model_path | Null terminated string of the path (wchar on Windows, char otherwise). |
| OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelBuffer | ( | OrtModelCompilationOptions * | model_compile_options, |
| OrtAllocator * | allocator, | ||
| void ** | output_model_buffer_ptr, | ||
| size_t * | output_model_buffer_size_ptr | ||
| ) |
Configures model compilation to store the output compiled ONNX model in a buffer.
The caller passes an OrtAllocator that ONNX Runtime uses to allocate memory for the buffer.
The output model's location (e.g., file path or memory buffer) can be set with either ModelCompilationOptions_SetOutputModelPath or ModelCompilationOptions_SetOutputModelBuffer.
If the output model's location is not set, ONNX Runtime will generate an output file with a path based on the input model's file path. Examples: /Path/my_model.onnx -> /Path/my_model_ctx.onnx /Path/my_model -> /Path/my_model_ctx.onnx
| [in] | model_compile_options | The OrtModelCompilationOptions instance. |
| [in] | allocator | The allocator used to allocate the buffer for the compiled model. |
| [out] | output_model_buffer_ptr | Pointer to the buffer that stores the compiled model. |
| [out] | output_model_buffer_size_ptr | Pointer set to the size of output model in bytes. |
| OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelExternalInitializersAlignment | ( | OrtModelCompilationOptions * | model_compile_options, |
| size_t | alignment, | ||
| size_t | minimum_size | ||
| ) |
Configure additional alignment for externalized initializer offsets.
Applies to file and buffer destinations for initializers at least minimum_size bytes. An alignment of zero disables the additional policy. Otherwise, alignment must be a power of two. By default, offsets for initializers of at least 1 MiB (1048576 bytes) are aligned to 4096 bytes, even if this setter is never called. Natural alignment for each tensor's element type is always applied, including when the additional policy is disabled. These settings align offsets, not the allocator-provided buffer address.
| [in] | model_compile_options | The OrtModelCompilationOptions instance. |
| [in] | alignment | Required byte alignment, or zero to disable. |
| [in] | minimum_size | Minimum initializer size at which alignment is applied.
|
| OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelExternalInitializersBuffer | ( | OrtModelCompilationOptions * | model_compile_options, |
| const char * | logical_file_name, | ||
| size_t | external_initializers_size_threshold, | ||
| OrtAllocator * | allocator, | ||
| void ** | output_buffer_ptr, | ||
| size_t * | output_buffer_size_ptr | ||
| ) |
Store external initializers for the compiled model in one caller-owned buffer.
This destination replaces any external-initializer file or callback destination configured previously. The logical file name must be a non-empty relative path. Leading current-directory prefixes (./ or .\\) are removed before the name is recorded in each externalized TensorProto. This setter only configures the destination; CompileModel allocates the buffer using allocator. The allocator must provide host-accessible memory (CPU or host-accessible/pinned device memory). Device-only allocators are rejected with ORT_INVALID_ARGUMENT. On successful CompileModel completion, the caller owns the allocated buffer and must release it with allocator. If no data is externalized, the output buffer is NULL and its size is zero. Only main-graph initializers are externalized to this buffer; subgraph initializers remain embedded in the model. If CompileModel fails, both outputs are unchanged and temporary allocations are freed. The allocator and output pointer locations must remain valid until CompileModel returns. If the model also uses an output buffer, its buffer and size output locations must each be distinct from those provided here. CompileModel rejects aliased output locations with ORT_INVALID_ARGUMENT.
The output model may be written to a file, buffer, or write callback. When the output model is written to a file, the caller is responsible for persisting or otherwise supplying this buffer under logical_file_name when the model is loaded.
| [in] | model_compile_options | The OrtModelCompilationOptions instance. |
| [in] | logical_file_name | Logical external-data file name stored in the model. |
| [in] | external_initializers_size_threshold | Initializers at least this size are externalized. |
| [in] | allocator | Allocator providing host-accessible memory for the output buffer. |
| [out] | output_buffer_ptr | Receives the allocated buffer, or NULL when no data is externalized. |
| [out] | output_buffer_size_ptr | Receives the allocated buffer size.
|
| OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelExternalInitializersFile | ( | OrtModelCompilationOptions * | model_compile_options, |
| const char * | external_initializers_file_path, | ||
| size_t | external_initializers_size_threshold | ||
| ) |
Optionally sets the file that should store external initializers for the compiled ONNX model. If not set, initializers are stored within the model.
Only initializers for nodes that were not compiled are stored in the external initializers file. Compiled nodes contain their initializer data within the ep_cache_context attribute of EPContext nodes. Refer to ModelCompilationOptions_SetEpContextEmbedMode.
| [in] | model_compile_options | The OrtModelCompilationOptions instance. |
| [in] | external_initializers_file_path | Null terminated string of the path to the file. |
| [in] | external_initializers_size_threshold | Initializers larger than this threshold are stored in the file. |
| OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelGetInitializerLocationFunc | ( | OrtModelCompilationOptions * | model_compile_options, |
| OrtGetInitializerLocationFunc | get_initializer_location_func, | ||
| void * | state | ||
| ) |
Sets a OrtGetInitializerLocationFunc function that is called by ORT for every initializer in the generated model. Allows implementer to specify whether initializers should be stored within the model or externally.
| [in] | model_compile_options | The OrtModelCompilationOptions instance. |
| [in] | get_initializer_location_func | The OrtGetInitializerLocationFunc function called by ORT when to determine the location of the initializer. |
| [in] | state | Opaque state passed as the first argument to OrtGetInitializerLocationFunc. Can be NULL. |
| OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelPath | ( | OrtModelCompilationOptions * | model_compile_options, |
| const char * | output_model_path | ||
| ) |
Sets the file path for the output ONNX model generated by CompileModel.
The output model's location (e.g., file path or memory buffer) can be set with either ModelCompilationOptions_SetOutputModelPath or ModelCompilationOptions_SetOutputModelBuffer.
If the output model's location is not set, ONNX Runtime will generate an output file with a path based on the input model's file path. Examples: /Path/my_model.onnx -> /Path/my_model_ctx.onnx /Path/my_model -> /Path/my_model_ctx.onnx
| [in] | model_compile_options | The OrtModelCompilationOptions instance. |
| [in] | output_model_path | Null terminated string of the path (wchar on Windows, char otherwise). |
| OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelWriteFunc | ( | OrtModelCompilationOptions * | model_compile_options, |
| OrtWriteBufferFunc | write_func, | ||
| void * | state | ||
| ) |
Sets a OrtWriteBufferFunc function that is called by ORT to write out the output model's serialized ONNX bytes.
The provided write function may be called repeatedly until then entire output model has been written out. Each call to the write function is expected to consume the entire input buffer.
The output model's destination (e.g., file path, memory buffer, or stream) can be set with any of the functions that begin with ModelCompilationOptions_SetOutputModel____.
| [in] | model_compile_options | The OrtModelCompilationOptions instance. |
| [in] | write_func | The OrtWriteBufferFunc function called by ORT when writing out the model. |
| [in] | state | Opaque state passed as the first argument to OrtWriteBufferFunc. Can be NULL. |
| OrtStatus * OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabled | ( | OrtModelCompilationOptions * | model_compile_options, |
| bool | use_weightless | ||
| ) |
Enable or disable weightless mode for model compilation.
When enabled, the compiled EPContext model will not embed constant initializer data in the EP's compiled binary. Instead, the initializer data must be provided when creating a session from the compiled model, either from the source model (via the "onnx_model_filename" EPContext node attribute or the "ep.context_source_model_path" session option) or from externalized weights.
This enables smaller compiled models and allows sharing initializer data across multiple compiled model variants (e.g., multi-platform caches for different hardware generations).
ORT verifies that the target EP supports weightless mode during CompileModel() by calling OrtEp::GetWeightlessSupport(). If the EP does not support weightless mode, CompileModel() returns an error.
| [in] | model_compile_options | The OrtModelCompilationOptions instance. |
| [in] | use_weightless | If true, enable weightless mode. If false, disable (default behavior). |
| void OrtCompileApi::ReleaseModelCompilationOptions | ( | OrtModelCompilationOptions * | input | ) |