Developing a Plugin Execution Provider Library

This page provides a reference for how to develop plugin EP libraries with ONNX Runtime.

Contents

Creating a plugin EP library

A plugin EP is built as a dynamic/shared library that exports the C functions CreateEpFactories() and ReleaseEpFactory(). ONNX Runtime calls CreateEpFactories() to obtain one or more instances of OrtEpFactory. An OrtEpFactory creates OrtEp instances and specifies the hardware devices supported by the EPs it creates.

The ONNX Runtime repository includes a few sample plugin EP libraries, which are referenced in the following sections.

Defining an OrtEp

An OrtEp represents an instance of an EP that is used by an ONNX Runtime session to identify and execute the model operations supported by the EP.

The OrtEp API struct is defined in onnxruntime_ep_c_api.h.

The following table lists the required variables and functions that an implementer must define for an OrtEp.

Field Summary Example implementation
ort_version_supported The ONNX Runtime version with which the EP was compiled. Implementation should set to ORT_API_VERSION. ExampleEp()
GetName Get the execution provider name.

The recommended convention for an execution provider name is to have the name end with the suffix "ExecutionProvider". E.g., "ContosoAiExecutionProvider".		 ExampleEp::GetNameImpl()
GetCapability Get information about the nodes/subgraphs supported by the OrtEp instance. ExampleEp::GetCapabilityImpl()

The following table lists the required functions that an implementer must define for an OrtEp that compiles nodes. In ORT 1.23, these were required functions because only compiling EPs were supported.

Field Summary Example implementation
Compile Compile OrtGraph instances assigned to the OrtEp. Implementation must set a OrtNodeComputeInfo instance for each OrtGraph in order to define its computation function.

If the session is configured to generate a pre-compiled model, the execution provider must return count number of EPContext nodes. 		ExampleEp::CompileImpl()
ReleaseNodeComputeInfos Release OrtNodeComputeInfo instances. ExampleEp::ReleaseNodeComputeInfosImpl()

The following table lists the required functions that an implementer must define for an OrtEp that provides operator kernels with a kernel registry.

Field Summary Example implementation
GetKernelRegistry Gets the execution provider's kernel registry, if any.

A kernel registry contains kernel creation information for operator kernels supported by an EP. 		ExampleKernelEp::GetKernelRegistryImpl()

The following table lists the optional functions that an implementer may define for an OrtEp. If an optional OrtEp function is not defined, ONNX Runtime uses a default implementation.

Field Summary Example implementation
GetPreferredDataLayout Get the EP's preferred data layout.

If this function is not implemented, ORT assumes that the EP prefers the data layout OrtEpDataLayout::NCHW.
ShouldConvertDataLayoutForOp Given an op with domain domain and type op_type, determine whether an associated node's data layout should be converted to a target_data_layout. If the EP prefers a non-default data layout, this function will be called during layout transformation with target_data_layout set to the EP's preferred data layout

Implementation of this function is optional. If an EP prefers a non-default data layout, it may implement this to customize the specific op data layout preferences at a finer granularity.
SetDynamicOptions Set dynamic options on this EP. Dynamic options can be set by the application at any time after session creation with OrtApi::SetEpDynamicOptions().

Implementation of this function is optional. An EP should only implement this function if it needs to handle any dynamic options.
OnRunStart Called by ORT to notify the EP of the start of a run.

Implementation of this function is optional. An EP should only implement this function if it needs to handle application-provided options at the start of a run.
OnRunEnd Called by ORT to notify the EP of the end of a run.

Implementation of this function is optional. An EP should only implement this function if it needs to handle application-provided options at the end of a run.
CreateAllocator Create an OrtAllocator for the given OrtMemoryInfo for an OrtSession.

The OrtMemoryInfo instance will match one of the values set in the OrtEpDevice using EpDevice_AddAllocatorInfo. Any allocator specific options should be read from the session options.

Implementation of this function is optional. If not provided, ORT will use `OrtEpFactory::CreateAllocator()`.
CreateSyncStreamForDevice Create a synchronization stream for the given memory device for an OrtSession.

This is used to create a synchronization stream for the execution provider and is used to synchronize operations on the device during model execution. Any stream specific options should be read from the session options.

Implementation of this function is optional. If not provided, ORT will use `OrtEpFactory::CreateSyncStreamForDevice()`.
GetCompiledModelCompatibilityInfo Get a string with details about the EP stack used to produce a compiled model.

The compatibility information string can be used with OrtEpFactory::ValidateCompiledModelCompatibilityInfo to determine if a compiled model is compatible with the EP.
IsConcurrentRunSupported Gets whether the execution provider supports concurrent run calls made on the session.

If not implemented, ORT assumes that concurrent runs are supported.

Defining an OrtEpFactory

An OrtEpFactory represents an instance of an EP factory that is used by an ONNX Runtime session to query device support, create allocators, create data transfer objects, and create instances of an EP (i.e., an OrtEp).

The OrtEpFactory API struct is defined in onnxruntime_ep_c_api.h.

The following table lists the required variables and functions that an implementer must define for an OrtEpFactory.

Field Summary Example implementation
ort_version_supported The ONNX Runtime version with which the EP was compiled. Implementation should set this to ORT_API_VERSION. ExampleEpFactory()
GetName Get the name of the EP that the factory creates. Must match OrtEp::GetName(). ExampleEpFactory::GetNameImpl()
GetVendor Get the name of the name of the vendor that owns the EP that the factory creates. ExampleEpFactory::GetVendor()
GetVendorId Get the vendor ID of the vendor that owns the EP that the factory creates. This is typically the PCI vendor ID. ExampleEpFactory::GetVendorId()
GetVersion Get the version of the EP that the factory creates. The version string should adhere to the Semantic Versioning 2.0 specification. ExampleEpFactory::GetVersionImpl()
GetSupportedDevices Get information about the OrtHardwareDevice instances supported by an EP created by the factory. ExampleEpFactory::GetSupportedDevicesImpl()
CreateEp Creates an OrtEp instance for use in an ONNX Runtime session. ORT calls OrtEpFactory::ReleaseEp() to release the instance. ExampleEpFactory::CreateEpImpl()

The following table lists the optional functions that an implementer may define for an OrtEpFactory.

Field Summary Example implementation
ValidateCompiledModelCompatibilityInfo Validate the compatibility of a compiled model with the EP.

This function validates if a model produced with the supllied compatibility information string is supported by the underlying EP. The implementation should check if a compiled model is compatible with the EP and return the appropriate OrtCompiledModelCompatibility value.
CreateAllocator Create an OrtAllocator that can be shared across sessions for the given OrtMemoryInfo.

The factory that creates the EP is responsible for providing the allocators required by the EP. The OrtMemoryInfo instance will match one of the values set in the OrtEpDevice using EpDevice_AddAllocatorInfo.		 ExampleEpFactory::CreateAllocatorImpl()
ReleaseAllocator Releases an OrtAllocator instance created by the factory. ExampleEpFactory::ReleaseAllocatorImpl()
CreateDataTransfer Creates an OrtDataTransferImpl instance for the factory.

An OrtDataTransferImpl can be used to copy data between devices that the EP supports.		 ExampleEpFactory::CreateDataTransferImpl()
IsStreamAware Returns true if the EPs created by the factory are stream-aware. ExampleEpFactory::IsStreamAwareImpl()
CreateSyncStreamForDevice Creates a synchronization stream for the given OrtMemoryDevice.

This is use to create a synchronization stream for the OrtMemoryDevice that can be used for operations outside of a session.		 ExampleEpFactory::CreateSyncStreamForDeviceImpl()
GetHardwareDeviceIncompatibilityDetails Check for known incompatibility reasons between a hardware device and this execution provider.

This function allows an execution provider to check if a specific hardware device is compatible with the execution provider. The EP can set specific incompatibility reasons via the OrtDeviceEpIncompatibilityDetails parameter using OrtEpApi::DeviceEpIncompatibilityDetails_SetDetails. 		ExampleEpFactory::GetHardwareDeviceIncompatibilityDetailsImpl()
CreateExternalResourceImporterForDevice Create an OrtExternalResourceImporterImpl for external resource import.

This is used to create an external resource importer that enables zero-copy import of external GPU memory (e.g., D3D12 shared resources) and synchronization primitives (e.g., D3D12 timeline fences).

EPs that support external resource import (via CUDA, HIP, Vulkan, or D3D12 APIs) can implement this to allow applications to share GPU resources without copies. 		ExampleEpFactory::CreateExternalResourceImporterForDeviceImpl()
GetNumCustomOpDomains Gets the number of EP-specific OrtCustomOpDomains provided by the factory. ExampleEpFactory::GetNumCustomOpDomainsImpl()
GetCustomOpDomains Gets the EP-specific OrtCustomOpDomains provided by the factory. ExampleEpFactory::GetCustomOpDomainsImpl()

Exporting functions to create and release factories

ONNX Runtime expects a plugin EP library to export certain functions/symbols. The following table lists the functions that have to be exported from the plugin EP library.

Function Description Example implementation
CreateEpFactories ONNX Runtime calls this function to create OrtEpFactory instances. ExampleEp: CreateEpFactories
ReleaseEpFactory ONNX Runtime calls this function to release an OrtEpFactory instance. ExampleEp: ReleaseEpFactory

API reference

API header files:

  • onnxruntime_ep_c_api.h
    • Defines interfaces implemented by plugin EP and EP factory instances.
    • Provides APIs utilized by plugin EP and EP factory instances.
  • onnxruntime_c_api.h
    • Provides APIs used to traverse an input model graph.

Data Types

Type Description
OrtHardwareDeviceType Enumerates classes of hardware devices:
  • OrtHardwareDeviceType_CPU
  • OrtHardwareDeviceType_GPU
  • OrtHardwareDeviceType_NPU
OrtHardwareDevice Opaque type that represents a physical hardware device.
OrtExecutionProviderDevicePolicy Enumerates the default EP selection policies available to users of ORT's automatic EP selection.
OrtEpDevice Opaque type that represents a pairing of an EP and hardware device that can run a model or model subgraph.
OrtNodeFusionOptions Struct that contains options for fusing nodes supported by an EP.
OrtNodeComputeContext Opaque type that contains a compiled/fused node's name and host memory allocation functions. ONNX Runtime provides an instance of OrtNodeComputeContext as an argument to OrtNodeComputeInfo::CreateState().
OrtNodeComputeInfo Struct that contains the computation function for a compiled OrtGraph instance. Initialized by an OrtEp instance.
OrtEpGraphSupportInfo Opaque type that contains information on the nodes supported by an EP. An instance of OrtEpGraphSupportInfo is passed to OrtEp::GetCapability() and the EP populates the OrtEpGraphSupportInfo instance with information on the nodes that it supports.
OrtEpDataLayout Enumerates the operator data layouts that could be preferred by an EP. By default, ONNX models use a "channel-first" layout (e.g., NCHW) but some EPs may prefer a "channel-last" layout (e.g., NHWC).
OrtMemoryDevice Opaque type that represents a combination of a physical device and memory type. A memory allocation and allocator are associated with a specific OrtMemoryDevice, and this information is used to determine when data transfer is required.
OrtDataTransferImpl Struct of functions that an EP implements to copy data between the devices that the EP uses and CPU.
OrtSyncNotificationImpl Struct of functions that an EP implements for Stream notifications.
OrtSyncStreamImpl Struct of functions that an EP implements if it needs to support Streams.
OrtEpFactory A plugin EP library provides ORT with one or more instances of OrtEpFactory. An OrtEpFactory implements functions that are used by ORT to query device support, create allocators, create data transfer objects, and create instances of an EP (i.e., an OrtEp instance).
An OrtEpFactory may support more than one hardware device (OrtHardwareDevice). If more than one hardware device is supported by the factory, an EP instance created by the factory is expected to internally partition any graph nodes assigned to the EP among its supported hardware devices.
Alternatively, if an EP library author needs ONNX Runtime to partition the graph nodes among different hardware devices supported by the EP library, then the EP library must provide multiple OrtEpFactory instances. Each OrtEpFactory instance must support one hardware device and must create an EP instance with a unique name (e.g., MyEP_CPU, MyEP_GPU, MyEP_NPU).
OrtEp An instance of an Ep that can execute model nodes on one or more hardware devices (OrtHardwareDevice). An OrtEp implements functions that are used by ORT to query graph node support, compile supported nodes, query preferred data layout, set run options, etc. An OrtEpFactory creates an OrtEp instance via the OrtEpFactory::CreateEp() function.
OrtRunOptions Opaque object containing options passed to the OrtApi::Run() function, which runs a model.
OrtGraph Opaque type that represents a graph. Provided to OrtEp instances in calls to OrtEp::GetCapability() and OrtEp::Compile().
OrtValueInfo Opaque type that contains information for a value in a graph. A graph value can be a graph input, graph output, graph initializer, node input, or node output. An OrtValueInfo instance has the following information.
  • Type and shape (e.g., OrtTypeInfo)
  • OrtNode consumers
  • OrtNode producer
  • Information that classifies the value as a graph input, graph output, initializer, etc.
OrtExternalInitializerInfo Opaque type that contains information for an initializer stored in an external file. An OrtExternalInitializerInfo instance contains the file path, file offset, and byte size for the initializer. Can be obtained from an OrtValueInfo via the function ValueInfo_GetExternalInitializerInfo().
OrtTypeInfo Opaque type that contains the element type and shape information for ONNX tensors, sequences, maps, sparse tensors, etc.
OrtTensorTypeAndShapeInfo Opaque type that contains the element type and shape information for an ONNX tensor.
OrtNode Opaque type that represents a node in a graph.
OrtOpAttrType Enumerates attribute types.
OrtOpAttr Opaque type that represents an ONNX operator attribute.