Comment on page
MLEdgeModel
class NatML.MLEdgeModel : MLModel
The
MLEdgeModel
represents an ML model that makes predictions on the local device. As such, it forms the basis for implementing edge predictors in code.The edge model can be created from a NatML Hub predictor, from a file, or from model data:
/// <summary>
/// Create an edge ML model.
/// </summary>
/// <param name="tagOrPath">Predictor tag or path to model file.</param>
/// <param name="configuration">Optional model configuration.</param>
/// <param name="accessKey">NatML access key.</param>
static Task<MLEdgeModel> Create (string tagOrPath, Configuration configuration = null, string accessKey = null);
The model can be created from a predictor on NatML Hub:
Explore - NatML
NatML Hub predictor catalog.
The predictor MUST have an active graph for your current platform.
When loading Edge predictors, NatML caches the model graph on device. This means that a user only has to download the model graph once.
The model can be created from an ML model file. Simply call the
Create
method and pass in a file path:// Given the path to an ML graph
var modelPath = "/Users/developer/Desktop/yolov5.onnx";
// Create an edge model
var model = await MLEdgeModel.Create(modelPath);
There are restrictions on what platforms support a given ML model format:
- CoreML models can only be used on iOS and macOS.
- ONNX models can only be used on Windows and Web.
- TensorFlow Lite models can only be used on Android.
Edge models created from model files do not contain any supplementary data, like class labels, normalization, and others.
/// <summary>
/// Create an edge ML model.
/// </summary>
/// <param name="modelData">ML model data.</param>
/// <param name="configuration">Optional model configuration.</param>
static Task<MLEdgeModel> Create (MLModelData modelData, Configuration configuration = null);
The model can be created from
MLModelData
instances, which are opaque representations of an ML model file in your Unity project.Edge models provide information about their expected input and output feature types. This type information is crucial for writing Edge predictors.
/// <summary>
/// Model input feature types.
/// </summary>
MLFeatureType[] inputs { get; }
/// <summary>
/// Model output feature types.
/// </summary>
MLFeatureType[] outputs { get; }
/// <summary>
/// Get the model metadata dictionary.
/// </summary>
IReadOnlyDictionary<string, string> metadata { get; }
/// <summary>
/// Model classification labels.
/// This is `null` if the predictor does not have use classification labels.
/// </summary>
string[] labels { get; }
For classification and detection models, this field contains the list of class labels associated with each class in the output distribution. If class labels don't apply to the model, it will return
null
.Edge models created from files will never have
labels
. Use NatML Hub instead./// <summary>
/// Expected feature normalization for predictions with this model.
/// </summary>
Normalization normalization { get; }
Vision models often require that images be normalized to a specific mean and standard deviation. As such,
MLEdgeModel
defines a Normalization
struct:struct Normalization {
/// <summary>
/// Per-channel normalization means.
/// </summary>
float[] mean { get; }
/// <summary>
/// Per-channel normalization standard deviations.
/// </summary>
float[] std { get; }
}
When working with image features, the
Normalization
struct can be easily deconstructed:// Get the model's preferred image normalization
Vector4 mean, std;
(mean, std) = modelData.normalization;
/// <summary>
/// Expected image aspect mode for predictions with this model.
/// </summary>
MLImageFeature.AspectMode aspectMode { get; }
Vision models might require that input image features be scaled a certain way when they are resized to fit the model's input size. The
aspectMode
can be passed directly to an MLImageFeature
./// <summary>
/// Expected audio format for predictions with this model.
/// </summary>
AudioFormat audioFormat { get; }
Audio and speech models often require or produce audio data with a specific sample rate and channel count. As such,
MLEdgeModel
defines an AudioFormat
struct:struct AudioFormat {
/// <summary>
/// Sample rate.
/// </summary>
int sampleRate { get; }
/// <summary>
/// Channel count.
/// </summary>
int channelCount { get; }
}
When working with audio features, the
AudioFormat
struct can be easily deconstructed like so:// Get the model's audio format
int sampleRate, channelCount;
(sampleRate, channelCount) = modelData.audioFormat;
/// <summary>
/// Make a prediction on one or more Edge ML features.
/// </summary>
/// <param name="inputs">Input edge ML features.</param>
/// <returns>Output edge ML features.</returns>
MLFeatureCollection<MLEdgeFeature> Predict (params MLEdgeFeature[] inputs);
The
MLEdgeModel
exposes a Predict
method which makes predictions on one or more MLEdgeFeature
instances.Instead of using
MLEdgeFeature
instances directly, we highly recommend using the managed feature classes instead (MLArrayFeature
, MLAudioFeature
, and so on).The input and output features MUST be disposed when they are no longer needed. Call
Dispose
on the individual features, or on the returned feature collection to do so.All calls to
Predict
MUST be serialized on a single thread. Calling Predict
across several threads is undefined behaviour and can result in a crash. As such, you must not make predictions within a ThreadPool
. Use MLPredictorExtensions.ToAsync
to make predictions on a dedicated worker thread instead./// <summary>
/// Dispose the model and release resources.
/// </summary>
void Dispose ();
/// <summary>
/// Embed the edge model into the app at build time.
/// </summary>
/// <param name="tag">Predictor tag.</param>
/// <param name="accessKey">NatML access key. If `null` the project access key will be used.</param>
class EmbedAttribute (string tag, string accessKey = null) : Attribute;
When fetching edge models from NatML Hub, the model graph must first be downloaded to the device before it is cached and loaded. For larger ML models or for users with poor internet, this download can take a long time. As such, the
MLEdgeModel
class defines the EmbedAttribute
to embed the ML graph into the app binary at build time.Note that the build size of the application will increase as a result of the embedded model data.
The attribute can be placed on any
class
or struct
definition. At build time, NatML will find all such attributes, and embed the corresponding model data in the build. The attribute can be used like so:ObjectDetector.cs
// An example script that embeds the `ssd-lite` model data from NatML Hub
[MLEdgeModel.Embed("@natsuite/ssd-lite")]
class ObjectDetector : MonoBehaviour {
async void Start () {
// Fetch the edge model from memory
// In a build, this will complete immediately
var model = await MLEdgeModel.Create("@natsuite/ssd-lite");
// Use the edge model
...
}
}
Last modified 9mo ago