Class OrtIoBinding
This class enables binding of inputs and/or outputs to pre-allocated memory. This enables interesting scenarios. For example, if your input already resides in some pre-allocated memory like GPU, you can bind that piece of memory to an input name and shape and onnxruntime will use that as input. Other traditional inputs can also be bound that already exists as Tensors.
Note, that this arrangement is designed to minimize data copies and to that effect your memory allocations must match what is expected by the model, whether you run on CPU or GPU. Data copy will still be made, if your pre-allocated memory location does not match the one expected by the model. However, copies with OrtIoBindings are only done once, at the time of the binding, not at run time. This means, that if your input data required a copy, your further input modifications would not be seen by onnxruntime unless you rebind it, even if it is the same buffer. If you require the scenario where data is copied, OrtIOBinding may not be the best match for your use case. The fact that data copy is not made during runtime also has performance implications.
Making OrtValue first class citizen in ORT C# API practically obsoletes all of the existing overloads because OrtValue can be created on top of the all other types of memory. No need to designate it as external or Ort allocation or wrap it in FixedBufferOnnxValue. The latter does not support rebinding or memory other than CPU anyway.
In fact, one can now create OrtValues over arbitrary pieces of memory, managed, native, stack and device(gpu) and feed them to the model and achieve the same effect without using IOBinding class.
Inheritance
Namespace: Microsoft.ML.OnnxRuntime
Assembly: Microsoft.ML.OnnxRuntime.dll
Syntax
public class OrtIoBinding : SafeHandle
Properties
| Improve this Doc View SourceIsInvalid
Overrides SafeHandle.IsInvalid
Declaration
public override bool IsInvalid { get; }
Property Value
Type | Description |
---|---|
System.Boolean | returns true if handle is equal to Zero |
Methods
| Improve this Doc View SourceBindInput(String, FixedBufferOnnxValue)
Bind the input with the given name as an OrtValue Tensor allocated in pinned managed memory. Instance of FixedBufferOnnxValue owns the memory and should be alive until the end of execution.
Declaration
public void BindInput(string name, FixedBufferOnnxValue fixedValue)
Parameters
Type | Name | Description |
---|---|---|
System.String | name | name of input |
FixedBufferOnnxValue | fixedValue |
BindInput(String, OrtExternalAllocation)
Bind externally (not from OrtAllocator) allocated memory as input. The model will read the specified input from that memory possibly avoiding the need to copy between devices. The user code continues to own the chunk of externally allocated memory, and the allocation should be alive until the end of execution.
Declaration
public void BindInput(string name, OrtExternalAllocation allocation)
Parameters
Type | Name | Description |
---|---|---|
System.String | name | name |
OrtExternalAllocation | allocation | non ort allocated memory |
BindInput(String, OrtValue)
This is the preferable and universal way to bind input to OrtValue. This way you retain control over the original value, can modify the data using OrtValue interfaces between the runs.
You can also create OrtValue on all kinds of memory, managed, native, stack and device(gpu).
Declaration
public void BindInput(string name, OrtValue ortValue)
Parameters
Type | Name | Description |
---|---|---|
System.String | name | input name |
OrtValue | ortValue |
BindInput(String, TensorElementType, Int64[], OrtMemoryAllocation)
Bind a piece of pre-allocated native memory as a OrtValue Tensor with a given shape to an input with a given name. The model will read the specified input from that memory possibly avoiding the need to copy between devices. OrtMemoryAllocation continues to own the chunk of native memory, and the allocation should be alive until the end of execution.
Declaration
public void BindInput(string name, TensorElementType elementType, long[] shape, OrtMemoryAllocation allocation)
Parameters
Type | Name | Description |
---|---|---|
System.String | name | of the input |
TensorElementType | elementType | Tensor element type |
System.Int64[] | shape | |
OrtMemoryAllocation | allocation | native memory allocation |
BindOutput(String, FixedBufferOnnxValue)
Bind model output to a given instance of FixedBufferOnnxValue which owns the underlying pinned managed memory and should be alive for the time of execution.
Declaration
public void BindOutput(string name, FixedBufferOnnxValue fixedValue)
Parameters
Type | Name | Description |
---|---|---|
System.String | name | of the output |
FixedBufferOnnxValue | fixedValue |
BindOutput(String, OrtExternalAllocation)
Bind externally (not from OrtAllocator) allocated memory as output. The model will read the specified input from that memory possibly avoiding the need to copy between devices. The user code continues to own the chunk of externally allocated memory, and the allocation should be alive until the end of execution.
Declaration
public void BindOutput(string name, OrtExternalAllocation allocation)
Parameters
Type | Name | Description |
---|---|---|
System.String | name | name |
OrtExternalAllocation | allocation | non ort allocated memory |
BindOutput(String, OrtValue)
This is the preferable and universal way to bind output via OrtValue. This way you retain control over the original value, can modify the data using OrtValue interfaces between the runs, rebind output to input if you are feeding data circular.
Declaration
public void BindOutput(string name, OrtValue ortValue)
Parameters
Type | Name | Description |
---|---|---|
System.String | name | output name |
OrtValue | ortValue | OrtValue to bind |
BindOutput(String, TensorElementType, Int64[], OrtMemoryAllocation)
Bind model output to an OrtValue as Tensor with a given type and shape. An instance of OrtMemoryAllocaiton owns the memory and should be alive for the time of execution.
Declaration
public void BindOutput(string name, TensorElementType elementType, long[] shape, OrtMemoryAllocation allocation)
Parameters
Type | Name | Description |
---|---|---|
System.String | name | of the output |
TensorElementType | elementType | tensor element type |
System.Int64[] | shape | tensor shape |
OrtMemoryAllocation | allocation | allocated memory |
BindOutputToDevice(String, OrtMemoryInfo)
This function will bind model output with the given name to a device specified by the memInfo.
Declaration
public void BindOutputToDevice(string name, OrtMemoryInfo memInfo)
Parameters
Type | Name | Description |
---|---|---|
System.String | name | output name |
OrtMemoryInfo | memInfo | instance of memory info |
ClearBoundInputs()
Clear all bound inputs and start anew
Declaration
public void ClearBoundInputs()
ClearBoundOutputs()
Clear all bound outputs
Declaration
public void ClearBoundOutputs()
GetOutputNames()
Returns an array of output names in the same order they were bound
Declaration
public string[] GetOutputNames()
Returns
Type | Description |
---|---|
System.String[] | array of output names |
GetOutputValues()
This class enables binding of inputs and/or outputs to pre-allocated memory. This enables interesting scenarios. For example, if your input already resides in some pre-allocated memory like GPU, you can bind that piece of memory to an input name and shape and onnxruntime will use that as input. Other traditional inputs can also be bound that already exists as Tensors.
Note, that this arrangement is designed to minimize data copies and to that effect your memory allocations must match what is expected by the model, whether you run on CPU or GPU. Data copy will still be made, if your pre-allocated memory location does not match the one expected by the model. However, copies with OrtIoBindings are only done once, at the time of the binding, not at run time. This means, that if your input data required a copy, your further input modifications would not be seen by onnxruntime unless you rebind it, even if it is the same buffer. If you require the scenario where data is copied, OrtIOBinding may not be the best match for your use case. The fact that data copy is not made during runtime also has performance implications.
Making OrtValue first class citizen in ORT C# API practically obsoletes all of the existing overloads because OrtValue can be created on top of the all other types of memory. No need to designate it as external or Ort allocation or wrap it in FixedBufferOnnxValue. The latter does not support rebinding or memory other than CPU anyway.
In fact, one can now create OrtValues over arbitrary pieces of memory, managed, native, stack and device(gpu) and feed them to the model and achieve the same effect without using IOBinding class.
Declaration
public IDisposableReadOnlyCollection<OrtValue> GetOutputValues()
Returns
Type | Description |
---|---|
IDisposableReadOnlyCollection<OrtValue> |
ReleaseHandle()
Overrides SafeHandle.ReleaseHandle() to properly dispose of the native instance of OrtIoBidning
Declaration
protected override bool ReleaseHandle()
Returns
Type | Description |
---|---|
System.Boolean | always returns true |
SynchronizeBoundInputs()
Blocks until device completes all preceding requested tasks. Useful for memory synchronization.
Declaration
public void SynchronizeBoundInputs()
SynchronizeBoundOutputs()
Blocks until device completes all preceding requested tasks. Useful for memory synchronization.
Declaration
public void SynchronizeBoundOutputs()