The OrtCompileApi struct provides functions to compile ONNX models. More...

#include <onnxruntime_c_api.h>

Public Member Functions
OrtModelCompilationOptions
void	ReleaseModelCompilationOptions (OrtModelCompilationOptions *input)

OrtStatus *	CreateModelCompilationOptionsFromSessionOptions (const OrtEnv env, const OrtSessionOptions session_options, OrtModelCompilationOptions **out)
	Creates an OrtModelCompilationOptions object from an existing OrtSessionOptions object.

OrtStatus *	ModelCompilationOptions_SetInputModelPath (OrtModelCompilationOptions model_compile_options, const char input_model_path)
	Sets the file path to the input ONNX model to compile.

OrtStatus *	ModelCompilationOptions_SetInputModelFromBuffer (OrtModelCompilationOptions model_compile_options, const void input_model_data, size_t input_model_data_size)
	Sets the buffer that stores the bytes of the loaded ONNX model to compile.

OrtStatus *	ModelCompilationOptions_SetOutputModelPath (OrtModelCompilationOptions model_compile_options, const char output_model_path)
	Sets the file path for the output ONNX model generated by CompileModel.

OrtStatus *	ModelCompilationOptions_SetOutputModelExternalInitializersFile (OrtModelCompilationOptions model_compile_options, const char external_initializers_file_path, size_t external_initializers_size_threshold)
	Optionally sets the file that should store external initializers for the compiled ONNX model. If not set, initializers are stored within the model.

OrtStatus *	ModelCompilationOptions_SetOutputModelBuffer (OrtModelCompilationOptions model_compile_options, OrtAllocator allocator, void *output_model_buffer_ptr, size_t output_model_buffer_size_ptr)
	Configures model compilation to store the output compiled ONNX model in a buffer.

OrtStatus *	ModelCompilationOptions_SetEpContextEmbedMode (OrtModelCompilationOptions *model_compile_options, bool embed_ep_context_in_model)
	Enables or disables the embedding of EPContext binary data into the `ep_cache_context` attribute of EPContext nodes. Defaults to false.

OrtStatus *	CompileModel (const OrtEnv env, const OrtModelCompilationOptions model_options)
	Compiles an input ONNX model with the given compilation options.

OrtStatus *	ModelCompilationOptions_SetFlags (OrtModelCompilationOptions *model_compile_options, uint32_t flags)
	Sets flags from OrtCompileApiFlags that represent one or more boolean options to enable.

OrtStatus *	ModelCompilationOptions_SetEpContextBinaryInformation (OrtModelCompilationOptions model_compile_options, const char output_directory, const char *model_name)

OrtStatus *	ModelCompilationOptions_SetGraphOptimizationLevel (OrtModelCompilationOptions *model_compile_options, GraphOptimizationLevel graph_optimization_level)

OrtStatus *	ModelCompilationOptions_SetOutputModelWriteFunc (OrtModelCompilationOptions model_compile_options, OrtWriteBufferFunc write_func, void state)
	Sets a OrtWriteBufferFunc function that is called by ORT to write out the output model's serialized ONNX bytes.

OrtStatus *	ModelCompilationOptions_SetOutputModelGetInitializerLocationFunc (OrtModelCompilationOptions model_compile_options, OrtGetInitializerLocationFunc get_initializer_location_func, void state)
	Sets a OrtGetInitializerLocationFunc function that is called by ORT for every initializer in the generated model. Allows implementer to specify whether initializers should be stored within the model or externally.

Detailed Description

The OrtCompileApi struct provides functions to compile ONNX models.

Execution providers that support compilation fuse a subgraph into an EPContext node that wraps a provider-specific binary representation of the subgraph. For more details about the EPContext design, refer to: EPContext design document.

Example (error handling not shown): OrtStatus* status = NULL; OrtCompileApi* compile_api = ort_api->GetCompileApi(); OrtModelCompilationOptions* compile_options = NULL;

status = compile_api->CreateModelCompilationOptionsFromSessionOptions(env, session_options, &compile_options); status = compile_api->ModelCompilationOptions_SetInputModelPath(compile_options, ORT_TSTR("model.onnx")); status = compile_api->ModelCompilationOptions_SetOutputModelPath(compile_options, ORT_TSTR("model.compiled.onnx")); status = compile_api->CompileModel(env, compile_options); compile_api->ReleaseModelCompilationOptions(compile_options);

Since: Version 1.22.

Member Function Documentation

◆ CompileModel()

OrtStatus * OrtCompileApi::CompileModel	(	const OrtEnv *	env,
		const OrtModelCompilationOptions *	model_options
	)

Compiles an input ONNX model with the given compilation options.

Parameters

[in]	env	OrtEnv object.
[in]	model_options	The compilation options that defines compilation options for a model.

Returns: If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since: Version 1.22.

◆ CreateModelCompilationOptionsFromSessionOptions()

OrtStatus * OrtCompileApi::CreateModelCompilationOptionsFromSessionOptions	(	const OrtEnv *	env,
		const OrtSessionOptions *	session_options,
		OrtModelCompilationOptions **	out
	)

Creates an OrtModelCompilationOptions object from an existing OrtSessionOptions object.

An OrtModelCompilationOptions object contains the settings used to generate a compiled ONNX model. The OrtSessionOptions object has the execution providers with which the model will be compiled.

ReleaseOrtModelCompilationsOptions must be called to free the OrtModelCompilationOptions after calling CompileModel.

Note: By default, the GraphOptimizationLevel is set to ORT_DISABLE_ALL. Use ModelCompilationOptions_SetGraphOptimizationLevel to enable graph optimizations.

Parameters

[in]	env	OrtEnv object.
[in]	session_options	The OrtSessionOptions instance from which to create the OrtModelCompilationOptions.
[out]	out	The created OrtModelCompilationOptions instance. Returns If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since: Version 1.22.

◆ ModelCompilationOptions_SetEpContextBinaryInformation()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetEpContextBinaryInformation	(	OrtModelCompilationOptions *	model_compile_options,
		const char *	output_directory,
		const char *	model_name
	)

Sets information related to EP context binary file.

EP uses this information to decide the location and context binary file name. Used while compiling model with input and output in memory buffer

Parameters

[in]	model_compile_options	The OrtModelCompilationOptions instance.
[in]	output_directory	Null terminated string of the path (wchar on Windows, char otherwise).
[in]	model_name	Null terminated string of the model name (wchar on Windows, char otherwise).

Returns: If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since: Version 1.23.

◆ ModelCompilationOptions_SetEpContextEmbedMode()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetEpContextEmbedMode	(	OrtModelCompilationOptions *	model_compile_options,
		bool	embed_ep_context_in_model
	)

Enables or disables the embedding of EPContext binary data into the ep_cache_context attribute of EPContext nodes. Defaults to false.

If enabled, the ep_cache_context attribute of EPContext nodes will store the context binary data, which may include weights for compiled subgraphs.

If disabled, the ep_cache_context attribute of EPContext nodes will contain the path to the file containing the context binary data. The path is set by the execution provider creating the EPContext node.

More details relate to EPContext design refers to: EPContext design document.

Parameters

[in]	model_compile_options	The OrtModelCompilationOptions instance.
[in]	embed_ep_context_in_model	True to embed EPContext binary data into the EPContext node `ep_cache_context` attributes.

Returns: If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since: Version 1.22.

◆ ModelCompilationOptions_SetFlags()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetFlags	(	OrtModelCompilationOptions *	model_compile_options,
		uint32_t	flags
	)

Sets flags from OrtCompileApiFlags that represent one or more boolean options to enable.

Parameters

[in]	model_compile_options	The OrtModelCompilationOptions instance.
[in]	flags	bitwise OR of flags in OrtCompileApiFlags to enable.

Returns: If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since: Version 1.23.

◆ ModelCompilationOptions_SetGraphOptimizationLevel()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetGraphOptimizationLevel	(	OrtModelCompilationOptions *	model_compile_options,
		GraphOptimizationLevel	graph_optimization_level
	)

Set the graph optimization level.

Parameters

[in]	model_compile_options	The OrtModelCompilationOptions instance.
[in]	graph_optimization_level	The graph optimization level.

Returns: If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since: Version 1.23.

◆ ModelCompilationOptions_SetInputModelFromBuffer()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetInputModelFromBuffer	(	OrtModelCompilationOptions *	model_compile_options,
		const void *	input_model_data,
		size_t	input_model_data_size
	)

Sets the buffer that stores the bytes of the loaded ONNX model to compile.

The input model's location (e.g., file path or memory buffer) must be set with either ModelCompilationOptions_SetInputModelPath or ModelCompilationOptions_SetInputModelFromBuffer.

Parameters

[in]	model_compile_options	The OrtModelCompilationOptions instance.
[in]	input_model_data	Buffer containing the loaded ONNX model bytes.
[in]	input_model_data_size	The number of bytes in the `input_model_data` buffer.

Returns: If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since: Version 1.22.

◆ ModelCompilationOptions_SetInputModelPath()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetInputModelPath	(	OrtModelCompilationOptions *	model_compile_options,
		const char *	input_model_path
	)

Sets the file path to the input ONNX model to compile.

The input model's location (e.g., file path or memory buffer) must be set with either ModelCompilationOptions_SetInputModelPath or ModelCompilationOptions_SetInputModelFromBuffer.

Parameters

[in]	model_compile_options	The OrtModelCompilationOptions instance.
[in]	input_model_path	Null terminated string of the path (wchar on Windows, char otherwise).

Returns: If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since: Version 1.22.

◆ ModelCompilationOptions_SetOutputModelBuffer()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelBuffer	(	OrtModelCompilationOptions *	model_compile_options,
		OrtAllocator *	allocator,
		void **	output_model_buffer_ptr,
		size_t *	output_model_buffer_size_ptr
	)

Configures model compilation to store the output compiled ONNX model in a buffer.

The caller passes an OrtAllocator that ONNX Runtime uses to allocate memory for the buffer.

The output model's location (e.g., file path or memory buffer) can be set with either ModelCompilationOptions_SetOutputModelPath or ModelCompilationOptions_SetOutputModelBuffer.

If the output model's location is not set, ONNX Runtime will generate an output file with a path based on the input model's file path. Examples: /Path/my_model.onnx -> /Path/my_model_ctx.onnx /Path/my_model -> /Path/my_model_ctx.onnx

Parameters

[in]	model_compile_options	The OrtModelCompilationOptions instance.
[in]	allocator	The allocator used to allocate the buffer for the compiled model.
[out]	output_model_buffer_ptr	Pointer to the buffer that stores the compiled model.
[out]	output_model_buffer_size_ptr	Pointer set to the size of output model in bytes.

Returns: If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since: Version 1.22.

◆ ModelCompilationOptions_SetOutputModelExternalInitializersFile()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelExternalInitializersFile	(	OrtModelCompilationOptions *	model_compile_options,
		const char *	external_initializers_file_path,
		size_t	external_initializers_size_threshold
	)

Optionally sets the file that should store external initializers for the compiled ONNX model. If not set, initializers are stored within the model.

Only initializers for nodes that were not compiled are stored in the external initializers file. Compiled nodes contain their initializer data within the ep_cache_context attribute of EPContext nodes. Refer to ModelCompilationOptions_SetEpContextEmbedMode.

Parameters

[in]	model_compile_options	The OrtModelCompilationOptions instance.
[in]	external_initializers_file_path	Null terminated string of the path to the file.
[in]	external_initializers_size_threshold	Initializers larger than this threshold are stored in the file.

Returns: If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since: Version 1.22.

◆ ModelCompilationOptions_SetOutputModelGetInitializerLocationFunc()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelGetInitializerLocationFunc	(	OrtModelCompilationOptions *	model_compile_options,
		OrtGetInitializerLocationFunc	get_initializer_location_func,
		void *	state
	)

Sets a OrtGetInitializerLocationFunc function that is called by ORT for every initializer in the generated model. Allows implementer to specify whether initializers should be stored within the model or externally.

Parameters

[in]	model_compile_options	The OrtModelCompilationOptions instance.
[in]	get_initializer_location_func	The OrtGetInitializerLocationFunc function called by ORT when to determine the location of the initializer.
[in]	state	Opaque state passed as the first argument to OrtGetInitializerLocationFunc. Can be NULL.

Returns: If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since: Version 1.23.

◆ ModelCompilationOptions_SetOutputModelPath()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelPath	(	OrtModelCompilationOptions *	model_compile_options,
		const char *	output_model_path
	)

Sets the file path for the output ONNX model generated by CompileModel.

The output model's location (e.g., file path or memory buffer) can be set with either ModelCompilationOptions_SetOutputModelPath or ModelCompilationOptions_SetOutputModelBuffer.

If the output model's location is not set, ONNX Runtime will generate an output file with a path based on the input model's file path. Examples: /Path/my_model.onnx -> /Path/my_model_ctx.onnx /Path/my_model -> /Path/my_model_ctx.onnx

Parameters

[in]	model_compile_options	The OrtModelCompilationOptions instance.
[in]	output_model_path	Null terminated string of the path (wchar on Windows, char otherwise).

Returns: If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since: Version 1.22.

◆ ModelCompilationOptions_SetOutputModelWriteFunc()

OrtStatus * OrtCompileApi::ModelCompilationOptions_SetOutputModelWriteFunc	(	OrtModelCompilationOptions *	model_compile_options,
		OrtWriteBufferFunc	write_func,
		void *	state
	)

Sets a OrtWriteBufferFunc function that is called by ORT to write out the output model's serialized ONNX bytes.

The provided write function may be called repeatedly until then entire output model has been written out. Each call to the write function is expected to consume the entire input buffer.

The output model's destination (e.g., file path, memory buffer, or stream) can be set with any of the functions that begin with ModelCompilationOptions_SetOutputModel____.

Parameters

[in]	model_compile_options	The OrtModelCompilationOptions instance.
[in]	write_func	The OrtWriteBufferFunc function called by ORT when writing out the model.
[in]	state	Opaque state passed as the first argument to OrtWriteBufferFunc. Can be NULL.

Returns: If no error, nullptr will be returned. If there is an error, a pointer to an OrtStatus that contains error details will be returned. Use OrtApi::ReleaseStatus to free this pointer.

Since: Version 1.23.

◆ ReleaseModelCompilationOptions()

void OrtCompileApi::ReleaseModelCompilationOptions ( OrtModelCompilationOptions * input )

Public Member Functions

Detailed Description

Member Function Documentation

◆ CompileModel()

◆ CreateModelCompilationOptionsFromSessionOptions()

◆ ModelCompilationOptions_SetEpContextBinaryInformation()

◆ ModelCompilationOptions_SetEpContextEmbedMode()

◆ ModelCompilationOptions_SetFlags()

◆ ModelCompilationOptions_SetGraphOptimizationLevel()

◆ ModelCompilationOptions_SetInputModelFromBuffer()

◆ ModelCompilationOptions_SetInputModelPath()

◆ ModelCompilationOptions_SetOutputModelBuffer()

◆ ModelCompilationOptions_SetOutputModelExternalInitializersFile()

◆ ModelCompilationOptions_SetOutputModelGetInitializerLocationFunc()

◆ ModelCompilationOptions_SetOutputModelPath()

◆ ModelCompilationOptions_SetOutputModelWriteFunc()

◆ ReleaseModelCompilationOptions()