Class OrtValue

Represents a disposable OrtValue. This class exposes a native instance of OrtValue. The class implements IDisposable and must be disposed of, otherwise native resources will leak and will eventually cause the application to slow down or crash.

If the OrtValue instance is constructed over a managed memory, and it is not disposed properly, the pinned memory will continue to be pinned and interfere with GC operation.

Inheritance

object

OrtValue

Implements

IDisposable

IReadOnlyOrtValue

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Namespace: Microsoft.ML.OnnxRuntime

Assembly: Microsoft.ML.OnnxRuntime.dll

Syntax

public class OrtValue : IDisposable, IReadOnlyOrtValue

Properties

IsSparseTensor

Returns true if OrtValue contains a sparse tensor

Declaration

public bool IsSparseTensor { get; }

Property Value

Type	Description
bool	true if sparse tensor

IsTensor

Returns true if OrtValue contains a tensor

Declaration

public bool IsTensor { get; }

Property Value

Type	Description
bool	true if tensor

OnnxType

Fetches OrtValue type if it has one.

Declaration

public OnnxValueType OnnxType { get; }

Property Value

Type	Description
OnnxValueType	OnnxValueType

Value

Implement IOrtValueOwner interface

Declaration

public OrtValue Value { get; }

Property Value

Type	Description
OrtValue	returns this

Methods

CreateAllocatedTensorValue(OrtAllocator, TensorElementType, long[])

The factory API creates an OrtValue with memory allocated using the given allocator according to the specified shape and element type. The memory will be released when OrtValue is disposed. Use GetTensorMutableDataAsSpan<T>() API to fill in the data.

Declaration

public static OrtValue CreateAllocatedTensorValue(OrtAllocator allocator, TensorElementType elementType, long[] shape)

Parameters

Type	Name	Description
OrtAllocator	allocator
TensorElementType	elementType
long[]	shape

Returns

Type	Description
OrtValue	A disposable OrtValue

CreateFromStringTensor(Tensor<string>)

Creates an OrtValue that contains a string tensor. String tensors are always allocated on CPU. String data will be converted to UTF-8 and copied to native memory.

Note, this is different from creating an OrtValue from other primitive data types where memory is pinned (if necessary) and the OrtValue points to that chunk of memory.

Declaration

public static OrtValue CreateFromStringTensor(Tensor<string> tensor)

Parameters

Type	Name	Description
Tensor<string>	tensor	Tensor{string}

Returns

Type	Description
OrtValue	A disposable OrtValue instance

Exceptions

Type	Condition
OnnxRuntimeException

CreateMap(ref OrtValue, ref OrtValue)

Creates a map OrtValue with keys and values. On a high level the Onnxruntime representation of the map always consists of two OrtValues, keys and values.

According to ONNX standard map keys can be unmanaged types only (or strings). Those keys are contained in a single tensor within OrtValue keys.

Map values, on the other hand, can be composite types. The values parameter can either contain a single tensor with unmanaged map values with the same number of elements as the keys, or it can be a sequence of OrtValues, each of those can be a composite type (tensor, sequence, map). If it is a sequence, then the number of elements must match the number of elements in keys.

Keys and values must be in the same order.

ORT supports only a subset of types for keys and values, however, this API does not restrict it.

The ortValues that are passed as argument are taken possession of by the newly created OrtValue. The caller should not dispose them, unless this call fails.

Keys and values arguments will be set to null on success.

Declaration

public static OrtValue CreateMap(ref OrtValue keys, ref OrtValue values)

Parameters

Type	Name	Description
OrtValue	keys	Contains keys
OrtValue	values	Contains values

Returns

Type	Description
OrtValue	A disposable OrtValue

Exceptions

Type	Condition
ArgumentNullException

CreateMapWithStringKeys<V>(IReadOnlyCollection<string>, V[])

Creates a map OrtValue with string keys and non-string values. This helps the user not to create OrtValues for keys and values separately. The number of elements in keys and values must be the same and they must be in order. The map would consist of two tensors, one for keys and one for values.

string keys would be converted to UTF-8 encoding and copied to an allocated native memory. The OrtValue for values would be created on top of the managed memory using it directly.

The values type must be unmanaged.

Declaration

public static OrtValue CreateMapWithStringKeys<V>(IReadOnlyCollection<string> keys, V[] values) where V : unmanaged

Parameters

Type	Name	Description
IReadOnlyCollection<string>	keys	Collection of strings
V[]	values

Returns

Type	Description
OrtValue	OrtValue instance

Type Parameters

Name	Description
V

Exceptions

Type	Condition
ArgumentNullException
ArgumentException

CreateMapWithStringValues<K>(K[], IReadOnlyCollection<string>)

Creates a map OrtValue with non-string keys and string values.

This helps the user not to create OrtValues for keys and values separately. The number of elements in keys and values must be the same and they must be in order.

The OrtValue for keys would be created on top of the managed memory using it directly. string values would be converted to UTF-8 encoding and copied to an allocated native memory.

Declaration

public static OrtValue CreateMapWithStringValues<K>(K[] keys, IReadOnlyCollection<string> values) where K : unmanaged

Parameters

Type	Name	Description
K[]	keys
IReadOnlyCollection<string>	values	collection of string values

Returns

Type	Description
OrtValue	Instance of OrtValue

Type Parameters

Name	Description
K	unmanaged type of keys

Exceptions

Type	Condition
ArgumentNullException
ArgumentException

CreateMap<K, V>(K[], V[])

This API helps to quickly creates a map OrtValue with unmanaged (primitive) keys and values specified as arrays. This helps the user not to create OrtValues for keys and values separately and deal only with the final result. The map would consist of two tensors, one for keys and one for values.

The OrtValues would be created on top of the managed memory arrays and use it directly. The number of elements in keys and values must be the same and they must be in order.

The types must be unmanaged.

Declaration

public static OrtValue CreateMap<K, V>(K[] keys, V[] values) where K : unmanaged where V : unmanaged

Parameters

Type	Name	Description
K[]	keys	array of keys of K type
V[]	values	array of values of V type

Returns

Type	Description
OrtValue	OrtValue instance

Type Parameters

Name	Description
K	keys type
V	values type

Exceptions

Type	Condition
ArgumentNullException
ArgumentException

CreateSequence(ICollection<OrtValue>)

Creates a sequence of OrtValues from a collection of OrtValues. All OrtValues in the collection must be of the same Onnx type. I.e. (Tensor, SparseTensor, Map, Sequence, etc.)

The ortValues that are passed as argument are taken possession of by the newly created OrtValue. The caller should not dispose them, unless this call fails.

The ortValues would be empty on successful return.

Declaration

public static OrtValue CreateSequence(ICollection<OrtValue> ortValues)

Parameters

Type	Name	Description
ICollection<OrtValue>	ortValues	a collection of OrtValues. On success the ortValues contained in the list are taken ownership of and the list is cleared.

Returns

Type	Description
OrtValue	A disposable instance of OrtValues

Exceptions

Type	Condition
ArgumentNullException

CreateTensorValueFromMemory<T>(OrtMemoryInfo, Memory<T>, long[])

This is a factory method that creates an OrtValue of Tensor type on top of MemoryT memory. The API pins the memory for the duration of the OrtValue lifetime. It is unpinned at disposal time.

Declaration

public static OrtValue CreateTensorValueFromMemory<T>(OrtMemoryInfo memoryInfo, Memory<T> memory, long[] shape) where T : unmanaged

Parameters

Type	Name	Description
OrtMemoryInfo	memoryInfo	Memory information that describes memory location
Memory<T>	memory	contiguous region of memory
long[]	shape	shape of the tensor to create. The size required by the shape must be less of equal of the memory.Length

Returns

Type	Description
OrtValue	A disposable OrtValue instance

Type Parameters

Name	Description
T	T must be one of the supported types

Exceptions

Type	Condition
OnnxRuntimeException

CreateTensorValueFromMemory<T>(T[], long[])

This is a factory method that creates an OrtValue of Tensor type on top managed data array. The API pins the memory for the duration of the OrtValue lifetime. It is unpinned at disposal time.

Declaration

public static OrtValue CreateTensorValueFromMemory<T>(T[] data, long[] shape) where T : unmanaged

Parameters

Type	Name	Description
T[]	data	managed data buffer
long[]	shape	shape that describes the buffer

Returns

Type	Description
OrtValue	A disposable OrtValue instance

Type Parameters

Name	Description
T

CreateTensorValueFromSystemNumericsTensorObject<T>(Tensor<T>)

This is a factory method creates a native Onnxruntime OrtValue containing a tensor on top of the existing tensor managed memory. The method will attempt to pin managed memory so no copying occurs when data is passed down to native code.

Declaration

[Experimental("SYSLIB5001")]
public static OrtValue CreateTensorValueFromSystemNumericsTensorObject<T>(Tensor<T> tensor) where T : unmanaged

Parameters

Type	Name	Description
Tensor<T>	tensor	Tensor object

Returns

Type	Description
OrtValue	And instance of OrtValue constructed on top of the object

Type Parameters

Name	Description
T

CreateTensorValueWithData(OrtMemoryInfo, TensorElementType, long[], nint, long)

Factory method to construct an OrtValue of Tensor type on top of pre-allocated memory. This can be a piece of arbitrary memory that may be allocated by OrtAllocator (possibly on a device), a chunk of managed memory (must be pinned for the duration of OrtValue lifetime) or a memory that is allocated natively allocated using Marshal.AllocHGlobal(), stackalloc or other means (may be on a device).

The resulting OrtValue does not own the underlying memory buffer and will not attempt to deallocate it. The caller must make sure that the memory remains valid for the duration of OrtValue lifetime.

Declaration

public static OrtValue CreateTensorValueWithData(OrtMemoryInfo memInfo, TensorElementType elementType, long[] shape, nint dataBufferPtr, long bufferLengthInBytes)

Parameters

Type	Name	Description
OrtMemoryInfo	memInfo	Memory Info. For managed memory its default is cpu. For other kinds of memory, one must construct as appropriate.
TensorElementType	elementType	DataType for the Tensor
long[]	shape	shape of the tensor to create. The size required by the shape must be less of equal of the memory.Length
nint	dataBufferPtr	Pointer to a raw memory buffer which may reside on a device
long	bufferLengthInBytes	Buffer length in bytes

Returns

Type	Description
OrtValue	A disposable instance of OrtValue

CreateTensorWithEmptyStrings(OrtAllocator, long[])

Creates an OrtValue that contains a string tensor of specified shape, and containing empty strings. String tensors are always on CPU. Use StringTensorSetElementAt to assign individual elements values.

Declaration

public static OrtValue CreateTensorWithEmptyStrings(OrtAllocator allocator, long[] shape)

Parameters

Type	Name	Description
OrtAllocator	allocator
long[]	shape	tensor shape

Returns

Type	Description
OrtValue	disposable OrtValue

Dispose()

IDisposable implementation

Declaration

public void Dispose()

Dispose(bool)

IDisposable implementation

Declaration

protected virtual void Dispose(bool disposing)

Parameters

Type	Name	Description
bool	disposing	true if invoked from Dispose() method

~OrtValue()

Represents a disposable OrtValue. This class exposes a native instance of OrtValue. The class implements IDisposable and must be disposed of, otherwise native resources will leak and will eventually cause the application to slow down or crash.

If the OrtValue instance is constructed over a managed memory, and it is not disposed properly, the pinned memory will continue to be pinned and interfere with GC operation.

Declaration

protected ~OrtValue()

GetStringElement(int)

Fetch string tensor element buffer pointer at the specified index, copy/convert UTF-8 into a UTF-16 string and return it.

Obtain TensorTypeAndShape to get shape and element count.

Declaration

public string GetStringElement(int index)

Parameters

Type	Name	Description
int	index	flat string tensor element index

Returns

Type	Description
string	UTF-16 string instance

GetStringElementAsMemory(int)

Fetch string tensor element buffer pointer at the specified index, convert/copy to UTF-16 char[] and return a ReadOnlyMemory{char} instance.

Obtain TensorTypeAndShape to get shape and element count.

Declaration

public ReadOnlyMemory<char> GetStringElementAsMemory(int index)

Parameters

Type	Name	Description
int	index	flat string tensor element index

Returns

Type	Description
ReadOnlyMemory<char>	ReadOnlyMemory{char} backed by a managed char[]. Its lifespan is not tied to the native buffer of OrtValue.

GetStringElementAsSpan(int)

Get a span over the native memory of the string tensor element. The span is valid as long as the OrtValue is valid.

This is useful if you want to perform your own UTF-8 decoding or you do not care about decoding. Obtain TensorTypeAndShape to get shape and element count.

Declaration

public ReadOnlySpan<byte> GetStringElementAsSpan(int index)

Parameters

Type	Name	Description
int	index	flat element index

Returns

Type	Description
ReadOnlySpan<byte>	ReadOnlySpan over UTF-8 bytes of the string tensor element

GetStringTensorAsArray()

Convenience method to obtain all string tensor elements as a string array.

Declaration

public string[] GetStringTensorAsArray()

Returns

Type	Description
string[]	string[]

Exceptions

Type	Condition
OnnxRuntimeException

GetTensorDataAsSpan<T>()

Returns a ReadOnlySpanT over tensor native buffer that provides a read-only view.

Note, that the memory may be device allocated and, therefore, not accessible from the CPU. To get memory descriptor use GetTensorMemoryInfo().

OrtValue must contain a non-string tensor. The span is valid as long as the OrtValue instance is alive (not disposed).

Declaration

public ReadOnlySpan<T> GetTensorDataAsSpan<T>() where T : unmanaged

Returns

Type	Description
ReadOnlySpan<T>	ReadOnlySpanT

Type Parameters

Name	Description
T

Exceptions

Type	Condition
OnnxRuntimeException

GetTensorDataAsTensorSpan<T>()

Returns a ReadOnlyTensorSpanT over tensor native buffer that provides a read-only view.

Note, that the memory may be device allocated and, therefore, not accessible from the CPU. To get memory descriptor use GetTensorMemoryInfo().

OrtValue must contain a non-string tensor. The span is valid as long as the OrtValue instance is alive (not disposed).

Declaration

[Experimental("SYSLIB5001")]
public ReadOnlyTensorSpan<T> GetTensorDataAsTensorSpan<T>() where T : unmanaged

Returns

Type	Description
ReadOnlyTensorSpan<T>	ReadOnlySpanT

Type Parameters

Name	Description
T

Exceptions

Type	Condition
OnnxRuntimeException

GetTensorMemoryInfo()

Returns OrtMemoryInfo iff this OrtValue contains a tensor or a sparse tensor.

Declaration

public OrtMemoryInfo GetTensorMemoryInfo()

Returns

Type	Description
OrtMemoryInfo	OrtMemoryInfo that describes the underlying memory allocation

Exceptions

Type	Condition
OnnxRuntimeException

GetTensorMutableDataAsSpan<T>()

Returns a SpanT over tensor native buffer. This enables you to safely and efficiently modify the underlying native buffer in a type-safe manner. This is useful for example in IOBinding scenarios where you want to modify results of the inference and feed it back as input.

Note, that the memory may be device allocated. To get memory descriptor use GetTensorMemoryInfo().

OrtValue must contain a non-string tensor. The span is valid as long as the OrtValue instance is alive (not disposed).

Declaration

public Span<T> GetTensorMutableDataAsSpan<T>() where T : unmanaged

Returns

Type	Description
Span<T>	Typed Span over the native buffer

Type Parameters

Name	Description
T

GetTensorMutableDataAsTensorSpan<T>()

Returns a TensorSpanT over tensor native buffer.

Note, that the memory may be device allocated and, therefore, not accessible from the CPU. To get memory descriptor use GetTensorMemoryInfo().

OrtValue must contain a non-string tensor. The span is valid as long as the OrtValue instance is alive (not disposed).

Declaration

[Experimental("SYSLIB5001")]
public TensorSpan<T> GetTensorMutableDataAsTensorSpan<T>() where T : unmanaged

Returns

Type	Description
TensorSpan<T>	ReadOnlySpanT

Type Parameters

Name	Description
T

Exceptions

Type	Condition
OnnxRuntimeException

GetTensorMutableRawData()

Provides mutable raw native buffer access.

Declaration

public Span<byte> GetTensorMutableRawData()

Returns

Type	Description
Span<byte>	Span over the native buffer bytes

GetTensorSizeInBytes()

This API computes and returns the size of the tensor data in bytes.

Declaration

public long GetTensorSizeInBytes()

Returns

Type	Description
long	size of the tensor data in bytes

GetTensorSpanMutableRawData<T>()

Provides mutable raw native buffer access.

Declaration

[Experimental("SYSLIB5001")]
public TensorSpan<byte> GetTensorSpanMutableRawData<T>() where T : unmanaged

Returns

Type	Description
TensorSpan<byte>	TensorSpan over the native buffer bytes

Type Parameters

Name	Description
T

GetTensorTypeAndShape()

Obtains Tensor And Type Information from the OrtValue iff it contains a tensor. Valid only for OrtValues that contain a tensor.

Declaration

public OrtTensorTypeAndShapeInfo GetTensorTypeAndShape()

Returns

Type	Description
OrtTensorTypeAndShapeInfo	A disposable instance of OrtTensorTypeAndShapeInfo

GetTypeInfo()

Creates and fetches Type information about the contained OnnxValue.

Declaration

public OrtTypeInfo GetTypeInfo()

Returns

Type	Description
OrtTypeInfo	a disposable instance of OrtTypeInfo

GetValue(int, OrtAllocator)

For non tensors return OrtValue element at the specified index. For maps only indices 0 and 1 are valid. For sequences, [0..N) are valid. See GetValueCount() to determine the valid range.

Declaration

public OrtValue GetValue(int index, OrtAllocator allocator)

Parameters

Type	Name	Description
int	index
OrtAllocator	allocator	allocator to use

Returns

Type	Description
OrtValue	OrtValue disposable instance that points to the corresponding element of the composite type

GetValueCount()

Valid for composite ML types like map, sequence. Returns 2 for map (keys, values) and N for sequence, where N is the number of elements in the sequence.

Declaration

public int GetValueCount()

Returns

Type	Description
int	Element count

ProcessMap(MapVisitor, OrtAllocator)

This API helps the user to process a map OrtValue without having to deal with the lifespan of intermediate OrtValues.

each API value is fed to the visitor functor.

Declaration

public void ProcessMap(OrtValue.MapVisitor visitor, OrtAllocator allocator)

Parameters

Type	Name	Description
OrtValue.MapVisitor	visitor	visitor function
OrtAllocator	allocator	Allocator to use for intermediate values

Exceptions

Type	Condition
OnnxRuntimeException

ProcessSequence(SequenceElementVisitor, OrtAllocator)

Feeds each OrtValue in a sequence to the visitor delegate. This helps users to avoid dealing each value life-span

Declaration

public void ProcessSequence(OrtValue.SequenceElementVisitor visitor, OrtAllocator allocator)

Parameters

Type	Name	Description
OrtValue.SequenceElementVisitor	visitor	visitor delegate
OrtAllocator	allocator	allocator to use for intermediate ort values

Exceptions

Type	Condition
OnnxRuntimeException

StringTensorSetElementAt(ReadOnlyMemory<char>, int)

Converts the string argument represented by ReadOnlyMemory to UTF-8, allocates space in the native tensor and copies it into the native tensor memory. Typically, this is used to populate a new empty string tensor element.

The number of elements is according to the shape supplied to CreateTensorWithEmptyStrings(). However, this API can also be used to overwrite any existing element within the string tensor.

In general, to obtain the number of elements for any tensor, use GetTensorTypeAndShape() which would return a disposable instance of TensorTypeAndShapeInfo. Then call GetElementCount() or GetShape().

Declaration

public void StringTensorSetElementAt(ReadOnlyMemory<char> rom, int index)

Parameters

Type	Name	Description
ReadOnlyMemory<char>	rom	ReadOnlyMemory instance over an array of chars
int	index	index of the string element within the tensor must be within bounds of [0, N)

StringTensorSetElementAt(ReadOnlySpan<byte>, int)

This API resizes String Tensor element to the requested amount of bytes (UTF-8) and copies the bytes from the supplied ReadOnlySpan into the native tensor memory (resized buffer).

The API is useful for quick loading of utf8 data into the native tensor memory.

Declaration

public void StringTensorSetElementAt(ReadOnlySpan<byte> utf8Bytes, int index)

Parameters

Type	Name	Description
ReadOnlySpan<byte>	utf8Bytes	read only span of bytes
int	index	flat index of the element in the string tensor

StringTensorSetElementAt(ReadOnlySpan<char>, int)

Converts the string argument represented by ReadOnlySpan to UTF-8, allocates space in the native tensor and copies it into the native tensor memory. Typically, this is used to populate a new empty string tensor element.

The number of elements is according to the shape supplied to CreateTensorWithEmptyStrings(). However, this API can also be used to overwrite any existing element within the string tensor.

In general, to obtain the number of elements for any tensor, use GetTensorTypeAndShape() which would return a disposable instance of TensorTypeAndShapeInfo. Then call GetElementCount() or GetShape().

Declaration

public void StringTensorSetElementAt(ReadOnlySpan<char> str, int index)

Parameters

Type	Name	Description
ReadOnlySpan<char>	str	ReadOnlySpan over chars
int	index	index of the string element within the tensor must be within bounds of [0, N)

Implements

IDisposable

IReadOnlyOrtValue