Class OrtIoBinding

This class enables binding of inputs and/or outputs to pre-allocated memory. This enables interesting scenarios. For example, if your input already resides in some pre-allocated memory like GPU, you can bind that piece of memory to an input name and shape and onnxruntime will use that as input. Other traditional inputs can also be bound that already exists as Tensors.

Note, that this arrangement is designed to minimize data copies and to that effect your memory allocations must match what is expected by the model, whether you run on CPU or GPU. Data copy will still be made, if your pre-allocated memory location does not match the one expected by the model. However, copies with OrtIoBindings are only done once, at the time of the binding, not at run time. This means, that if your input data required a copy, your further input modifications would not be seen by onnxruntime unless you rebind it, even if it is the same buffer. If you require the scenario where data is copied, OrtIOBinding may not be the best match for your use case. The fact that data copy is not made during runtime also has performance implications.

Making OrtValue first class citizen in ORT C# API practically obsoletes all of the existing overloads because OrtValue can be created on top of the all other types of memory. No need to designate it as external or Ort allocation or wrap it in FixedBufferOnnxValue. The latter does not support rebinding or memory other than CPU anyway.

In fact, one can now create OrtValues over arbitrary pieces of memory, managed, native, stack and device(gpu) and feed them to the model and achieve the same effect without using IOBinding class.

Inheritance

Object

CriticalFinalizerObject

SafeHandle

OrtIoBinding

Implements

IDisposable

Inherited Members

SafeHandle.handle

SafeHandle.Close()

SafeHandle.DangerousAddRef(Boolean)

SafeHandle.DangerousGetHandle()

SafeHandle.DangerousRelease()

SafeHandle.Dispose()

SafeHandle.Dispose(Boolean)

SafeHandle.SetHandle(IntPtr)

SafeHandle.SetHandleAsInvalid()

SafeHandle.IsClosed

Object.Equals(Object)

Object.Equals(Object, Object)

Object.GetHashCode()

Object.GetType()

Object.MemberwiseClone()

Object.ReferenceEquals(Object, Object)

Object.ToString()

Namespace: Microsoft.ML.OnnxRuntime

Assembly: Microsoft.ML.OnnxRuntime.dll

Syntax

public class OrtIoBinding : SafeHandle, IDisposable

Properties

| Improve this Doc View Source

IsInvalid

Overrides SafeHandle.IsInvalid

Declaration

public override bool IsInvalid { get; }

Property Value

Type	Description
Boolean	returns true if handle is equal to Zero

Overrides

System.Runtime.InteropServices.SafeHandle.IsInvalid

Methods

| Improve this Doc View Source

BindInput(String, FixedBufferOnnxValue)

Bind the input with the given name as an OrtValue Tensor allocated in pinned managed memory. Instance of FixedBufferOnnxValue owns the memory and should be alive until the end of execution.

Declaration

[Obsolete("This BindInput overload is deprecated. Use of OrtValue based overload is recommended.")]
public void BindInput(string name, FixedBufferOnnxValue fixedValue)

Parameters

Type	Name	Description
String	name	name of input
FixedBufferOnnxValue	fixedValue

| Improve this Doc View Source

BindInput(String, OrtExternalAllocation)

Bind externally (not from OrtAllocator) allocated memory as input. The model will read the specified input from that memory possibly avoiding the need to copy between devices. The user code continues to own the chunk of externally allocated memory, and the allocation should be alive until the end of execution.

Declaration

[Obsolete("This BindInput overload is deprecated. Create OrtValue over an arbitrary piece of memory.")]
public void BindInput(string name, OrtExternalAllocation allocation)

Parameters

Type	Name	Description
String	name	name
OrtExternalAllocation	allocation	non ort allocated memory

| Improve this Doc View Source

BindInput(String, OrtValue)

This is the preferable and universal way to bind input to OrtValue. This way you retain control over the original value, can modify the data using OrtValue interfaces between the runs.

You can also create OrtValue on all kinds of memory, managed, native, stack and device(gpu).

Declaration

public void BindInput(string name, OrtValue ortValue)

Parameters

Type	Name	Description
String	name	input name
OrtValue	ortValue

| Improve this Doc View Source

BindInput(String, TensorElementType, Int64[], OrtMemoryAllocation)

Bind a piece of pre-allocated native memory as a OrtValue Tensor with a given shape to an input with a given name. The model will read the specified input from that memory possibly avoiding the need to copy between devices. OrtMemoryAllocation continues to own the chunk of native memory, and the allocation should be alive until the end of execution.

Declaration

public void BindInput(string name, TensorElementType elementType, long[] shape, OrtMemoryAllocation allocation)

Parameters

Type	Name	Description
String	name	of the input
TensorElementType	elementType	Tensor element type
Int64[]	shape
OrtMemoryAllocation	allocation	native memory allocation

| Improve this Doc View Source

BindOutput(String, FixedBufferOnnxValue)

Bind model output to a given instance of FixedBufferOnnxValue which owns the underlying pinned managed memory and should be alive for the time of execution.

Declaration

[Obsolete("This BindOutput overload is deprecated. Use of OrtValue based overload is recommended.")]
public void BindOutput(string name, FixedBufferOnnxValue fixedValue)

Parameters

Type	Name	Description
String	name	of the output
FixedBufferOnnxValue	fixedValue

| Improve this Doc View Source

BindOutput(String, OrtExternalAllocation)

Bind externally (not from OrtAllocator) allocated memory as output. The model will read the specified input from that memory possibly avoiding the need to copy between devices. The user code continues to own the chunk of externally allocated memory, and the allocation should be alive until the end of execution.

Declaration

[Obsolete("This BindOutput overload is deprecated. Create OrtValue over an arbitrary piece of memory.")]
public void BindOutput(string name, OrtExternalAllocation allocation)

Parameters

Type	Name	Description
String	name	name
OrtExternalAllocation	allocation	non ort allocated memory

| Improve this Doc View Source

BindOutput(String, OrtValue)

This is the preferable and universal way to bind output via OrtValue. This way you retain control over the original value, can modify the data using OrtValue interfaces between the runs, rebind output to input if you are feeding data circular.

Declaration

public void BindOutput(string name, OrtValue ortValue)

Parameters

Type	Name	Description
String	name	output name
OrtValue	ortValue	OrtValue to bind

| Improve this Doc View Source

BindOutput(String, TensorElementType, Int64[], OrtMemoryAllocation)

Bind model output to an OrtValue as Tensor with a given type and shape. An instance of OrtMemoryAllocaiton owns the memory and should be alive for the time of execution.

Declaration

public void BindOutput(string name, TensorElementType elementType, long[] shape, OrtMemoryAllocation allocation)

Parameters

Type	Name	Description
String	name	of the output
TensorElementType	elementType	tensor element type
Int64[]	shape	tensor shape
OrtMemoryAllocation	allocation	allocated memory

| Improve this Doc View Source

BindOutputToDevice(String, OrtMemoryInfo)

This function will bind model output with the given name to a device specified by the memInfo.

Declaration

public void BindOutputToDevice(string name, OrtMemoryInfo memInfo)

Parameters

Type	Name	Description
String	name	output name
OrtMemoryInfo	memInfo	instance of memory info

| Improve this Doc View Source

ClearBoundInputs()

Clear all bound inputs and start anew

Declaration

public void ClearBoundInputs()

| Improve this Doc View Source

ClearBoundOutputs()

Clear all bound outputs

Declaration

public void ClearBoundOutputs()

| Improve this Doc View Source

GetOutputNames()

Returns an array of output names in the same order they were bound

Declaration

public string[] GetOutputNames()

Returns

Type	Description
String[]	array of output names

| Improve this Doc View Source

GetOutputValues()

This fetches bound outputs after running the model with RunWithBinding()

Declaration

public IDisposableReadOnlyCollection<OrtValue> GetOutputValues()

Returns

Type	Description
IDisposableReadOnlyCollection<OrtValue>	IDisposableReadOnlyCollection{OrtValue}

| Improve this Doc View Source

ReleaseHandle()

Overrides SafeHandle.ReleaseHandle() to properly dispose of the native instance of OrtIoBidning

Declaration

protected override bool ReleaseHandle()

Returns

Type	Description
Boolean	always returns true

Overrides

System.Runtime.InteropServices.SafeHandle.ReleaseHandle()

| Improve this Doc View Source

SynchronizeBoundInputs()

Blocks until device completes all preceding requested tasks. Useful for memory synchronization.

Declaration

public void SynchronizeBoundInputs()

| Improve this Doc View Source

SynchronizeBoundOutputs()

Blocks until device completes all preceding requested tasks. Useful for memory synchronization.

Declaration

public void SynchronizeBoundOutputs()

Implements

System.IDisposable