Package ai.onnxruntime

Class OrtSession.SessionOptions

  • All Implemented Interfaces:
    java.lang.AutoCloseable
    Enclosing class:
    OrtSession
    public static class OrtSession.SessionOptions
extends java.lang.Object
implements java.lang.AutoCloseable
    Represents the options used to construct this session.

    Used to set the number of threads, optimisation level, computation backend and other options.

    Modifying this after the session has been constructed will have no effect.

    The SessionOptions object must not be closed until all sessions which use it are closed, as otherwise it could release resources that are in use.

    • Constructor Summary

      Constructors 
      Constructor Description
      SessionOptions()
      Create an empty session options.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void addACL​(boolean enableFastMath)
      Adds the ARM Compute Library as an execution backend.
      void addArmNN​(boolean useArena)
      Adds the ARM Neural Net library as an execution backend.
      void addConfigEntry​(java.lang.String configKey, java.lang.String configValue)
      Adds a single session configuration entry as a pair of strings.
      void addCoreML()
      Adds Apple's CoreML as an execution backend.
      void addCoreML​(java.util.EnumSet<CoreMLFlags> flags)
      Adds Apple's CoreML as an execution backend.
      void addCoreML​(java.util.Map<java.lang.String,​java.lang.String> providerOptions)
      Adds CoreML as an execution backend.
      void addCPU​(boolean useArena)
      Adds the CPU as an execution backend, using the arena allocator if desired.
      void addCUDA()
      Add CUDA as an execution backend, using device 0.
      void addCUDA​(int deviceNum)
      Add CUDA as an execution backend, using the specified CUDA device id.
      void addCUDA​(OrtCUDAProviderOptions cudaOpts)
      Adds CUDA as an execution backend, using the specified CUDA options.
      void addDirectML​(int deviceId)
      Adds DirectML as an execution backend.
      void addDnnl​(boolean useArena)
      Adds Intel's Deep Neural Network Library as an execution backend.
      void addExternalInitializers​(java.util.Map<java.lang.String,​OnnxTensorLike> initializers)
      Adds in the supplied externally loaded initializers.
      void addInitializer​(java.lang.String name, OnnxTensorLike initializer)
      Adds an initializer to override one from the ONNX model.
      void addNnapi()
      Adds Android's NNAPI as an execution backend.
      void addNnapi​(java.util.EnumSet<NNAPIFlags> flags)
      Adds Android's NNAPI as an execution backend.
      void addOpenVINO​(java.lang.String deviceId)
      Adds OpenVINO as an execution backend.
      void addQnn​(java.util.Map<java.lang.String,​java.lang.String> providerOptions)
      Adds QNN as an execution backend.
      void addROCM()
      Add ROCM as an execution backend, using device 0.
      void addROCM​(int deviceNum)
      Add ROCM as an execution backend, using the specified ROCM device id.
      void addTensorrt​(int deviceNum)
      Adds Nvidia's TensorRT as an execution backend.
      void addTensorrt​(OrtTensorRTProviderOptions tensorRTOpts)
      Adds Nvidia's TensorRT as an execution backend.
      void addTvm​(java.lang.String settings)
      Adds TVM as an execution backend.
      void addXnnpack​(java.util.Map<java.lang.String,​java.lang.String> providerOptions)
      Adds XNNPACK as an execution backend.
      void close()
      Closes the session options, releasing any memory acquired.
      void disablePerSessionThreads()
      Disables the per session thread pools.
      void disableProfiling()
      Disables profiling in sessions using this SessionOptions.
      void enableProfiling​(java.lang.String filePath)
      Enables profiling in sessions using this SessionOptions.
      java.util.Map<java.lang.String,​java.lang.String> getConfigEntries()
      Returns an unmodifiable view of the map contains all session configuration entries.
      void registerCustomOpLibrary​(java.lang.String path)
      Registers a library of custom ops for use with OrtSessions using this SessionOptions.
      void registerCustomOpsUsingFunction​(java.lang.String registrationFuncName)
      Registers custom ops for use with OrtSessions using this SessionOptions by calling the specified native function name.
      void setCPUArenaAllocator​(boolean useArena)
      Sets the CPU to use an arena memory allocator.
      void setDeterministicCompute​(boolean value)
      Set whether to use deterministic compute.
      void setExecutionMode​(OrtSession.SessionOptions.ExecutionMode mode)
      Sets the execution mode of this options object, overriding the old setting.
      void setInterOpNumThreads​(int numThreads)
      Sets the size of the CPU thread pool used for executing multiple request concurrently, if executing on a CPU.
      void setIntraOpNumThreads​(int numThreads)
      Sets the size of the CPU thread pool used for executing a single graph, if executing on a CPU.
      void setLoggerId​(java.lang.String loggerId)
      Sets the logger id to use.
      void setMemoryPatternOptimization​(boolean memoryPatternOptimization)
      Turns on memory pattern optimizations, where memory is preallocated if all shapes are known.
      void setOptimizationLevel​(OrtSession.SessionOptions.OptLevel level)
      Sets the optimization level of this options object, overriding the old setting.
      void setOptimizedModelFilePath​(java.lang.String outputPath)
      Sets the output path for the optimized model.
      void setSessionLogLevel​(OrtLoggingLevel logLevel)
      Sets the Session's logging level.
      void setSessionLogVerbosityLevel​(int logLevel)
      Sets the Session's logging verbosity level.
      void setSymbolicDimensionValue​(java.lang.String dimensionName, long dimensionValue)
      Sets the value of a symbolic dimension.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • SessionOptions

        public SessionOptions()
        Create an empty session options.

    • Method Detail

      • close

        public void close()
        Closes the session options, releasing any memory acquired.
        Specified by:
        close in interface java.lang.AutoCloseable

      • setOptimizationLevel

        public void setOptimizationLevel​(OrtSession.SessionOptions.OptLevel level)
                          throws OrtException
        Sets the optimization level of this options object, overriding the old setting.
        Parameters:
        level - The optimization level to use.
        Throws:
        OrtException - If there was an error in native code.

      • setInterOpNumThreads

        public void setInterOpNumThreads​(int numThreads)
                          throws OrtException
        Sets the size of the CPU thread pool used for executing multiple request concurrently, if executing on a CPU.
        Parameters:
        numThreads - The number of threads to use.
        Throws:
        OrtException - If there was an error in native code.

      • setIntraOpNumThreads

        public void setIntraOpNumThreads​(int numThreads)
                          throws OrtException
        Sets the size of the CPU thread pool used for executing a single graph, if executing on a CPU.
        Parameters:
        numThreads - The number of threads to use.
        Throws:
        OrtException - If there was an error in native code.

      • setOptimizedModelFilePath

        public void setOptimizedModelFilePath​(java.lang.String outputPath)
                               throws OrtException
        Sets the output path for the optimized model.
        Parameters:
        outputPath - The output path to write the model to.
        Throws:
        OrtException - If there was an error in native code.

      • setLoggerId

        public void setLoggerId​(java.lang.String loggerId)
                 throws OrtException
        Sets the logger id to use.
        Parameters:
        loggerId - The logger id string.
        Throws:
        OrtException - If there was an error in native code.

      • enableProfiling

        public void enableProfiling​(java.lang.String filePath)
                     throws OrtException
        Enables profiling in sessions using this SessionOptions.
        Parameters:
        filePath - The file to write profile information to.
        Throws:
        OrtException - If there was an error in native code.

      • disableProfiling

        public void disableProfiling()
                      throws OrtException
        Disables profiling in sessions using this SessionOptions.
        Throws:
        OrtException - If there was an error in native code.

      • setMemoryPatternOptimization

        public void setMemoryPatternOptimization​(boolean memoryPatternOptimization)
                                  throws OrtException
        Turns on memory pattern optimizations, where memory is preallocated if all shapes are known.
        Parameters:
        memoryPatternOptimization - If true enable memory pattern optimizations.
        Throws:
        OrtException - If there was an error in native code.

      • setCPUArenaAllocator

        public void setCPUArenaAllocator​(boolean useArena)
                          throws OrtException
        Sets the CPU to use an arena memory allocator.
        Parameters:
        useArena - If true use an arena memory allocator for the CPU execution provider.
        Throws:
        OrtException - If there was an error in native code.

      • setSessionLogLevel

        public void setSessionLogLevel​(OrtLoggingLevel logLevel)
                        throws OrtException
        Sets the Session's logging level.
        Parameters:
        logLevel - The log level to use.
        Throws:
        OrtException - If there was an error in native code.

      • setSessionLogVerbosityLevel

        public void setSessionLogVerbosityLevel​(int logLevel)
                                 throws OrtException
        Sets the Session's logging verbosity level.
        Parameters:
        logLevel - The logging verbosity to use.
        Throws:
        OrtException - If there was an error in native code.

      • registerCustomOpLibrary

        public void registerCustomOpLibrary​(java.lang.String path)
                             throws OrtException
        Registers a library of custom ops for use with OrtSessions using this SessionOptions.
        Parameters:
        path - The path to the library on disk.
        Throws:
        OrtException - If there was an error loading the library.

      • registerCustomOpsUsingFunction

        public void registerCustomOpsUsingFunction​(java.lang.String registrationFuncName)
                                    throws OrtException
        Registers custom ops for use with OrtSessions using this SessionOptions by calling the specified native function name. The custom ops library must either be linked against, or have previously been loaded by the user.

        The registration function must have the signature:

         OrtStatus* (*fn)(OrtSessionOptions* options, const OrtApiBase* api);

        See https://onnxruntime.ai/docs/reference/operators/add-custom-op.html for more information on custom ops. See https://github.com/microsoft/onnxruntime/blob/342a5bf2b756d1a1fc6fdc582cfeac15182632fe/onnxruntime/test/testdata/custom_op_library/custom_op_library.cc#L115 for an example of a custom op library registration function.

        Parameters:
        registrationFuncName - The name of the registration function to call.
        Throws:
        OrtException - If there was an error finding or calling the registration function.

      • setSymbolicDimensionValue

        public void setSymbolicDimensionValue​(java.lang.String dimensionName,
                                      long dimensionValue)
                               throws OrtException
        Sets the value of a symbolic dimension. Fixed dimension computations may have more optimizations applied to them.
        Parameters:
        dimensionName - The name of the symbolic dimension.
        dimensionValue - The value to set that dimension to.
        Throws:
        OrtException - If there was an error in native code.

      • setDeterministicCompute

        public void setDeterministicCompute​(boolean value)
                             throws OrtException
        Set whether to use deterministic compute.

        Default is false. If set to true, this will enable deterministic compute for GPU kernels where possible. Note that this most likely will have a performance cost.

        Parameters:
        value - Should the compute be deterministic?
        Throws:
        OrtException - If there was an error in native code.

      • disablePerSessionThreads

        public void disablePerSessionThreads()
                              throws OrtException
        Disables the per session thread pools. Must be used in conjunction with an environment containing global thread pools.
        Throws:
        OrtException - If there was an error in native code.

      • addConfigEntry

        public void addConfigEntry​(java.lang.String configKey,
                           java.lang.String configValue)
                    throws OrtException
        Adds a single session configuration entry as a pair of strings.
        Parameters:
        configKey - The config key string.
        configValue - The config value string.
        Throws:
        OrtException - If there was an error in native code.

      • getConfigEntries

        public java.util.Map<java.lang.String,​java.lang.String> getConfigEntries()
        Returns an unmodifiable view of the map contains all session configuration entries.
        Returns:
        All session configuration entries

      • addExternalInitializers

        public void addExternalInitializers​(java.util.Map<java.lang.String,​OnnxTensorLike> initializers)
                             throws OrtException
        Adds in the supplied externally loaded initializers.

        Note the initializers are copied into the session once it has been created, and the native references are removed from this SessionOptions. Once the session has been created those initializers can be closed. This is a different lifetime to initializers added via addInitializer(String, OnnxTensorLike). The initializers must be created from Buffer objects.

        Parameters:
        initializers - The map of names to initializers.
        Throws:
        OrtException - If the initializers could not be loaded.

      • addInitializer

        public void addInitializer​(java.lang.String name,
                           OnnxTensorLike initializer)
                    throws OrtException
        Adds an initializer to override one from the ONNX model.

        Note the initializer lifetime must outlive the session and session options. This is a different lifetime to initializers added via addExternalInitializers(Map). The initializers must be created from Buffer objects.

        Parameters:
        name - The initializer name.
        initializer - The initializer value.
        Throws:
        OrtException - If the initializer could not be loaded into the session options.

      • addCUDA

        public void addCUDA()
             throws OrtException
        Add CUDA as an execution backend, using device 0.
        Throws:
        OrtException - If there was an error in native code.

      • addCUDA

        public void addCUDA​(int deviceNum)
             throws OrtException
        Add CUDA as an execution backend, using the specified CUDA device id.
        Parameters:
        deviceNum - The CUDA device id.
        Throws:
        OrtException - If there was an error in native code.

      • addCUDA

        public void addCUDA​(OrtCUDAProviderOptions cudaOpts)
             throws OrtException
        Adds CUDA as an execution backend, using the specified CUDA options.
        Parameters:
        cudaOpts - The CUDA execution provider options.
        Throws:
        OrtException - If there was an error in the native code.

      • addROCM

        public void addROCM()
             throws OrtException
        Add ROCM as an execution backend, using device 0.
        Throws:
        OrtException - If there was an error in native code.

      • addROCM

        public void addROCM​(int deviceNum)
             throws OrtException
        Add ROCM as an execution backend, using the specified ROCM device id.
        Parameters:
        deviceNum - The ROCM device id.
        Throws:
        OrtException - If there was an error in native code.

      • addCPU

        public void addCPU​(boolean useArena)
            throws OrtException
        Adds the CPU as an execution backend, using the arena allocator if desired.

        By default this backend is used, but if other backends are requested, it should be requested last.

        Parameters:
        useArena - If true use the arena memory allocator.
        Throws:
        OrtException - If there was an error in native code.

      • addDnnl

        public void addDnnl​(boolean useArena)
             throws OrtException
        Adds Intel's Deep Neural Network Library as an execution backend.
        Parameters:
        useArena - If true use the arena memory allocator.
        Throws:
        OrtException - If there was an error in native code.

      • addOpenVINO

        public void addOpenVINO​(java.lang.String deviceId)
                 throws OrtException
        Adds OpenVINO as an execution backend.
        Parameters:
        deviceId - The id of the OpenVINO execution device.
        Throws:
        OrtException - If there was an error in native code.

      • addTensorrt

        public void addTensorrt​(int deviceNum)
                 throws OrtException
        Adds Nvidia's TensorRT as an execution backend.
        Parameters:
        deviceNum - The id of the CUDA device.
        Throws:
        OrtException - If there was an error in native code.

      • addTensorrt

        public void addTensorrt​(OrtTensorRTProviderOptions tensorRTOpts)
                 throws OrtException
        Adds Nvidia's TensorRT as an execution backend.
        Parameters:
        tensorRTOpts - The configuration parameters for TensorRT.
        Throws:
        OrtException - If there was an error in native code.

      • addNnapi

        public void addNnapi()
              throws OrtException
        Adds Android's NNAPI as an execution backend. Uses the default empty flag.
        Throws:
        OrtException - If there was an error in native code.

      • addNnapi

        public void addNnapi​(java.util.EnumSet<NNAPIFlags> flags)
              throws OrtException
        Adds Android's NNAPI as an execution backend.
        Parameters:
        flags - The flags which control the NNAPI configuration.
        Throws:
        OrtException - If there was an error in native code.

      • addTvm

        public void addTvm​(java.lang.String settings)
            throws OrtException
        Adds TVM as an execution backend.
        Parameters:
        settings - See the documentation for valid settings strings.
        Throws:
        OrtException - If there was an error in native code.

      • addDirectML

        public void addDirectML​(int deviceId)
                 throws OrtException
        Adds DirectML as an execution backend.
        Parameters:
        deviceId - The id of the DirectML device.
        Throws:
        OrtException - If there was an error in native code.

      • addACL

        public void addACL​(boolean enableFastMath)
            throws OrtException
        Adds the ARM Compute Library as an execution backend.
        Parameters:
        enableFastMath - Enable fast math mode in ACL.
        Throws:
        OrtException - If there was an error in native code.

      • addArmNN

        public void addArmNN​(boolean useArena)
              throws OrtException
        Adds the ARM Neural Net library as an execution backend.
        Parameters:
        useArena - If true use the arena memory allocator.
        Throws:
        OrtException - If there was an error in native code.

      • addCoreML

        public void addCoreML()
               throws OrtException
        Adds Apple's CoreML as an execution backend. Uses the default empty flag.
        Throws:
        OrtException - If there was an error in native code.

      • addCoreML

        public void addCoreML​(java.util.EnumSet<CoreMLFlags> flags)
               throws OrtException
        Adds Apple's CoreML as an execution backend.
        Parameters:
        flags - The flags which control the CoreML configuration.
        Throws:
        OrtException - If there was an error in native code.

      • addXnnpack

        public void addXnnpack​(java.util.Map<java.lang.String,​java.lang.String> providerOptions)
                throws OrtException
        Adds XNNPACK as an execution backend.
        Parameters:
        providerOptions - Configuration options for the XNNPACK backend. Refer to the XNNPACK execution provider's documentation.
        Throws:
        OrtException - If there was an error in native code.

      • addQnn

        public void addQnn​(java.util.Map<java.lang.String,​java.lang.String> providerOptions)
            throws OrtException
        Adds QNN as an execution backend.
        Parameters:
        providerOptions - Configuration options for the QNN backend. Refer to the QNN execution provider's documentation.
        Throws:
        OrtException - If there was an error in native code.

      • addCoreML

        public void addCoreML​(java.util.Map<java.lang.String,​java.lang.String> providerOptions)
               throws OrtException
        Adds CoreML as an execution backend.
        Parameters:
        providerOptions - Configuration options for the CoreML backend. Refer to the CoreML execution provider's documentation.
        Throws:
        OrtException - If there was an error in native code.