Vision Predictors

Computer vision is perhaps the most common use case for machine learning. ML allows for solving some interesting vision tasks, like image classification, detection, segmentation, and so on. Across all of these tasks, the NatML workflow is relatively uniform:
Creating a Vision Predictor
First, we fetch a vision model:
1
// Fetch model data from NatML
2
var modelData = await MLModelData.FromHub("@natsuite/ssd-lite");
3
// Deserialize the model
4
var model = modelData.Deserialize();
Copied!
Vision models typically consume an image, and produce one or more outputs depending on what the model does:
1
// Inspect the model
2
Debug.Log(model);
3
// Prints:
4
// MLEdgeModel
5
// Input: input.1: (1, 3, 300, 300) System.Single <-- INPUT IMAGE
6
// Output: 836: (1, 3000, 21) System.Single
7
// Output: 863: (1, 3000, 4) System.Single
Copied!
Then we instantiate the predictor class for the model:
1
// Create the SSD Lite predictor
2
var predictor = new SSDLitePredictor(model, modelData.labels);
Copied!
When using models from NatML Hub, you can download the package to get the predictor class implementation.
You will always want to fetch the model data, deserialize the model, and create the predictor once, usually in your Start method.
Making Vision Predictions
To make a prediction, we first need to create an image feature which the model will make the prediction on:
1
// Create an image feature
2
Texture2D image = ...;
3
var imageFeature = new MLImageFeature(image);
4
// Apply normalization
5
(imageFeature.mean, imageFeature.std) = modelData.normalization;
6
// Apply aspect mode
7
imageFeatutre.aspectMode = modelData.aspectMode;
Copied!
See MLImageFeature for all the different ways to create an image feature from existing images or image data.
It is crucial to apply the proper normalization and aspect mode to the image feature, as many vision models will produce wrong outputs without the proper settings.
We can now make a prediction on the image feature:
1
// Make a prediction on the image feature
2
var results = predictor.Predict(imageFeature);
Copied!
Some vision predictors might require multiple inputs. See the README of your predictor for what features to pass in.
Predictions on Videos
INCOMPLETE.

Workflows - Previous

Fetching Models

Next - Workflows

Audio Predictors

Last modified 1mo ago

Copy link

Contents

Creating a Vision Predictor

Making Vision Predictions

Predictions on Videos