NX Delegate

Overview

The NX Delegate is a TensorFlow™ Lite (TFLite) delegate that offloads supported operators to a Vulkan®-based execution path via Arm®'s tosa_for_spirv_codegen (tfsc) toolchain.

It converts segments of a TFLite graph into a TOSA graph, lowers that graph to SPIR-V™, and dispatches compute workloads on Vulkan-capable Hardware

Unsupported operators (unless fallback is disabled) run using standard TFLite kernels.

Features

• nxdelegate
  • the shared delegate library that implements operator conversion and Vulkan dispatch.
• delegateExecutor
  • an executable that runs a TFLite model using the delegate and can emit VGF artefacts.
• DelegateUnitTests
  • test binary that verifies conversion, probing, and end-to-end correctness for supported ops/models.

Quantisation support uses helper functions (see Utils.hpp) including ComputeMultiplierAndShiftTosaScale16 to derive 16‑bit rescale multiplier/shift pairs aligned with TOSA semantics.

Rescale stages are inserted around ops as needed depending on operator code and network data types.

Build & Install

The project uses CMake™ (>= 3.25) with FetchContent for TensorFlow Lite and tosa_for_spirv_codegen (tfsc).

ML Emulation Layer for Vulkan

Before you begin, you must clone and install the "ML Emulation Layer for Vulkan" - the latest steps for this can here found here.

Cloning the NX Delegate repository

After building and installing the ML Emulation Layer for Vulkan, you can clone our repository using the following structure:

git clone <nx-delegate-repository-url>

Once cloned, cd into the newly-created repository directory and choose the preset that meets your requirements, making sure to incorporate the ML Emulation Layer for Vulkan paths:

CMake Presets

Debug

cmake --preset debug
cmake --build --preset debug -j

Release (shared TFLite):

cmake --preset release
cmake --build --preset release -j

Release (static TFLite):

cmake --preset release-static
cmake --build --preset release-static -j

Add -G Ninja during configuration after each --preset call to generate Ninja build scripts, for example

cmake --preset debug -G Ninja

If using the ML Emulation Layer you can point to the Vulkan headers provided during cmake configuration, for example:

cmake --preset debug -G Ninja \
  -DVULKANML_HEADERS_PATH=/path/to/emulation_layer/dependencies/Vulkan-Headers/include/

Unit tests are available in the test directory, and can be run with the following command:

./build/debug/test/DelegateUnitTests

Delegate Public API (C++)

Header: include/NXDelegate.hpp & include/DelegateOptions.hpp

• include/NXDelegate.hpp: C-style TfLite delegate entrypoints and thin C++ wrappers used to create/delete and configure the delegate.
• include/DelegateOptions.hpp: Configuration surface for logging, VGF output path, model name and fallback control.

// Construct options
nxdelegate::DelegateOptions opts;

opts.SetLoggingLevel(tflite::TFLITE_LOG_INFO); // or TFLITE_LOG_SILENT
opts.SetVgfPath("/tmp/my_model_delegate.vgf"); // optional serialisation target file
opts.SetModelName("my_model");                 // optional metadata

// Disable fallback if you want strict delegation
opts.SetDisableFallback(true);

// Create a TFLite delegate instance
TfLiteDelegate* delegate = nxdelegate::NXDelegateCreate(opts);

// Attach to interpreter
interpreter->ModifyGraphWithDelegate(delegate);

// ... use interpreter normally

// Cleanup
nxdelegate::NXDelegateDelete(delegate);

Utility probe (non-mutating full support test) is available internally in tests (IsFullySupportedByNXDelegate) demonstrating identification without graph modification.

DelegateOptions Summary

Option	Method	Description
Logging Level	SetLoggingLevel(tflite::LogSeverity)	Controls delegate logging verbosity.
VGF Path	SetVgfPath(std::string)	If set, serialised VGF artefact will be written during Prepare.
Model Name	SetModelName(std::string)	Embedded in serialisation / logging contexts.
Disable Fallback	SetDisableFallback(bool)	Prevent execution with standard kernels if some ops unsupported.

Dependencies

Dependency	Licence	Version
ML Emulation Layer for Vulkan	Apache-2.0	Emulation Layer v0.7.0 Latest
TensorFlow Lite	Apache-2.0	v2.19.0
tfsc (tosa_for_spirv_codegen)	Apache-2.0	release/v1.0.1
CMake	BSD 3-Clause	v3.25.0

Build Flags

NX Delegate uses the following security related build flags:

Build flags
-Wall
-Wextra
-Wold-style-cast
-Wno-missing-braces
-Wconversion
-Wsign-conversion
-Werror

Usage Example

The test harness provides a command-line interface to run a model and optionally generate a VGF artefact.

# Run with delegate without producing a VGF file
./build/debug/test/delegateExecutor model.tflite --no-vgf

# Run with delegate and produce default VGF naming
./build/debug/test/delegateExecutor model.tflite

# Specify output VGF directory and custom name
./build/debug/test/delegateExecutor model.tflite \
  --out_dir /tmp/vgf_out --out_name compiled.vgf

# Compute RMSE and generate JSON report
./build/debug/test/delegateExecutor model.tflite \
  --report-json --report-dir /tmp/json_out

# Disable delegation (reference only)
./build/debug/test/delegateExecutor model.tflite --no-delegate

# Enforce no CPU fallback for unsupported operators
./build/debug/test/delegateExecutor model.tflite --no-fallback

Behaviour:

• Random inputs initialised via PrepareInputsRandom().
• Delegate run compared against a reference interpreter (RunReference + CompareWithReference).
• VGF/JSON files written alongside the model or to specified path when delegate active.

Integrating Into Your Application

Minimal integration pattern:

#include <NXDelegate.hpp>

#include <tensorflow/lite/interpreter.h>
#include <tensorflow/lite/kernels/register.h>

auto model = tflite::FlatBufferModel::BuildFromFile("model.tflite");
tflite::ops::builtin::BuiltinOpResolver resolver;
std::unique_ptr<tflite::Interpreter> interpreter;

if (tflite::InterpreterBuilder(*model, resolver)(&interpreter) != kTfLiteOk)
{
    /* handle error */
}

nxdelegate::DelegateOptions opts;
opts.SetLoggingLevel(tflite::TFLITE_LOG_INFO);
opts.SetVgfPath("/tmp/model_delegate.vgf");

TfLiteDelegate* delegate = nxdelegate::NXDelegateCreate(opts);

if (interpreter->ModifyGraphWithDelegate(delegate) != kTfLiteOk)
{
    /* fallback or abort */
}

if (interpreter->AllocateTensors() != kTfLiteOk)
{
    /* handle */
}

// Populate inputs, invoke
if (interpreter->Invoke() != kTfLiteOk)
{
    /* handle */
}

nxdelegate::NXDelegateDelete(delegate);

Logging & Diagnostics

Set logging level via DelegateOptions::SetLoggingLevel.

Available severities include:

• TFLITE_LOG_SILENT
• TFLITE_LOG_ERROR
• TFLITE_LOG_WARNING
• TFLITE_LOG_INFO

During configuration, CMake prints status lines for fetches, patch application, and tfsc helper scripts.

Failures in patch or script execution terminate configuration early with actionable error messages.

Extending Operator Support

To add an operator:

1. Implement a mapper subclass via DEFINE_OPERATOR_MAPPER(NewOp) in OperatorMapper.hpp.
2. Provide Validate, ConfigureRescales, and Convert logic, where appropriate.
3. Update any tests with new operator coverage.
4. Rebuild; the factory OperatorMapper::getInstance will expose it based on internal operator code mapping.

License

The NXDelegate project is licensed under Apache-2.0. Copyright (c) 2023-2025 Arm Limited.

Attributions

CMake is a trademark of Kitware inc. TensorFlow is a registered trademark of Google LLC. Vulkan is a registered trademark of the Khronos Group Inc. SPIR-V is a trademark of the Khronos Group Inc.

Changelog

For release notes and recent changes, see CHANGELOG.md.