MLImageFeature

Search…
MLImageFeature
class NatSuite.ML.Features.MLImageFeature : MLFeature, IMLEdgeFeature, IMLCloudFeature
This feature contains a pixel buffer. Because computer vision models have similar pre-processing requirements, the image feature is able to perform these operations when predictions are made with it.
Creating the Feature
The image feature can be created from several common image inputs:
From a Texture2D
1
/// <summary>
2
/// Create an image feature.
3
/// </summary>
4
/// <param name="texture"></param>
5
MLImageFeature (Texture2D texture);
Copied!
The image feature can be created from a Texture2D.
The input texture MUST be readable.
This constructor allocates a pixel buffer every time it is used, so prefer using one of the other constructors that accepts a pixel buffer instead.
From a Color Buffer
1
/// <summary>
2
/// Create an image feature.
3
/// </summary>
4
/// <param name="pixelBuffer">Pixel buffer to create image feature from.</param>
5
/// <param name="width">Pixel buffer width.</param>
6
/// <param name="height">Pixel buffer height.</param>
7
MLImageFeature (Color32[] pixelBuffer, int width, int height);
Copied!
The image feature can be created from a color buffer.
From a Pixel Buffer
1
/// <summary>
2
/// Create an image feature.
3
/// </summary>
4
/// <param name="pixelBuffer">Pixel buffer to create image feature from. MUST have an RGBA8888 layout.</param>
5
/// <param name="width">Pixel buffer width.</param>
6
/// <param name="height">Pixel buffer height.</param>
7
MLImageFeature (byte[] pixelBuffer, int width, int height);
Copied!
The image feature can be created from a raw pixel buffer.
The pixel buffer must have an RGBA8888 layout.
From a Native Array
1
/// <summary>
2
/// Create an image feature from a pixel buffer.
3
/// </summary>
4
/// <param name="pixelBuffer">Pixel buffer.</param>
5
/// <param name="width">Pixel buffer width.</param>
6
/// <param name="height">Pixel buffer height.</param>
7
MLImageFeature (NativeArray<byte> pixelBuffer, int width, int height);
Copied!
The image feature can be created from a NativeArray<byte>. This is useful when making predictions with pixel data from Unity's Texture2D API's.
The native array must remain valid for the lifetime of the image feature.
From a Native Buffer
1
/// <summary>
2
/// Create an image feature from a pixel buffer.
3
/// </summary>
4
/// <param name="pixelBuffer">Pixel buffer.</param>
5
/// <param name="width">Pixel buffer width.</param>
6
/// <param name="height">Pixel buffer height.</param>
7
MLImageFeature (void* pixelBuffer, int width, int height);
Copied!
The image feature can be created from a native pixel buffer. This is useful when making predictions with data from native plugins or external libraries like OpenCV.
The pixel buffer must have an RGBA8888 layout.
The pixel buffer must remain valid for the lifetime of the image feature.
From a Cloud ML Feature
1
/// <summary>
2
/// Create an image feature from a Cloud ML feature.
3
/// </summary>
4
/// <param name="feature">Cloud ML feature. This MUST be an `IMAGE` feature.</param>
5
MLImageFeature (MLCloudFeature feature);
Copied!
An image feature can be created from an MLCloudFeature. This constructor is useful for working with output prediction data from an MLCloudModel when authoring a predictor:
1
// Make a Hub prediction
2
MLCloudModel model = ...;
3
MLCloudFeature[] outputFeatures = model.Predict(...);
4
// Create an image feature from a Cloud ML image feature
5
var feature = new MLImageFeature(outputFeatures[0]);
Copied!
The Cloud ML feature MUST be an IMAGE feature.
Inspecting the Feature
The image feature exposes its underlying type, along with convenience properties for inspecting the aforementioned type.
Feature Type
1
/// <summary>
2
/// Feature type.
3
/// </summary>
4
MLFeatureType type { get; }
Copied!
Refer to the Inspecting the Feature section of the MLFeature class for more information.
The type is always an MLImageType.
Image Width
1
/// <summary>
2
/// Image width.
3
/// </summary>
4
int width { get; }
Copied!
The image feature provides this convenience property for accessing the width of the feature type.
Image Height
1
/// <summary>
2
/// Image height.
3
/// </summary>
4
int height { get; }
Copied!
The image feature provides this convenience property for accessing the height of the feature type.
Image Preprocessing
The image feature supports preprocessing when creating an MLEdgeFeature for edge predictions.
Normalization
When making Edge predictions on image features, some models might require that input data is normalized to some be within some range. The image feature provides these properties as an easy way to perform any required normalization.
The default range for image features is [0.0, 1.0].
When using NatML Hub, the normalization coefficients can be specified when creating a predictor:
Specifying normalization coefficients on NatML Hub.
The specified normalization coefficients can then be used like so:
1
// Fetch model data from NatML Hub
2
var modelData = await MLModelData.FromHub("@author/some-model");
3
// Create image feature
4
var imageFeature = new MLImageFeature(...);
5
// Apply normalization
6
(imageFeature.mean, imageFeature.std) = modelData.normalization;
Copied!
Mean
1
/// <summary>
2
/// Normalization mean.
3
/// </summary>
4
Vector4 mean { get; set; }
Copied!
The image feature supports specifying a per-channel normalization mean when creating an MLEdgeFeature.
Standard Deviation
1
/// <summary>
2
/// Normalization standard deviation.
3
/// </summary>
4
Vector4 std { get; set; }
Copied!
The image feature supports specifying a per-channel normalization standard deviation when creating an MLEdgeFeature.
Aspect Mode
1
/// <summary>
2
/// Aspect mode.
3
/// </summary>
4
AspectMode aspectMode { get; set; }
Copied!
The image feature supports specifying an aspect mode when creating an MLEdgeFeature with a different aspect ratio than the image feature. The aspectMode specifies how the difference in aspect ratio should be handled:
Aspect Mode
Example
AspectMode.ScaleToFit
​
 
AspectMode.AspectFill
​
 
AspectMode.AspectFit
​
 
When the aspectMode is AspectMode.AspectFit, the edge feature will be padded with transparent pixels, (0.0,0.0,0.0,0.0)(0.0, 0.0, 0.0, 0.0)(0.0,0.0,0.0,0.0)
.
Accessing Feature Data
The image feature provides methods for copying image data:
Copying
1
/// <summary>
2
/// Copy the image data in this feature into a pixel buffer.
3
/// </summary>
4
/// <param name="pixelBuffer">Destination pixel buffer.</param>
5
void CopyTo<T> (T[] pixelBuffer) where T : unmanaged;
Copied!
INCOMPLETE.
1
/// <summary>
2
/// Copy the image data in this feature into a pixel buffer.
3
/// </summary>
4
/// <param name="pixelBuffer">Destination pixel buffer.</param>
5
void CopyTo<T> (NativeArray<T> pixelBuffer) where T : unmanaged;
6
​
7
/// <summary>
8
/// Copy the image data in this feature into the provided pixel buffer.
9
/// </summary>
10
/// <param name="pixelBuffer">Destination pixel buffer.</param>
11
void CopyTo (void* pixelBuffer);
Copied!
Converting to Texture
1
/// <summary>
2
/// Convert the image feature to a texture.
3
/// This method MUST only be used from the Unity main thread.
4
/// </summary>
5
/// <param name="result">Optional. Result texture to copy data into.</param>
6
/// <returns>Result texture.</returns>
7
Texture2D ToTexture (Texture2D result = default);
Copied!
INCOMPLETE.
This method MUST only be used from the Unity main thread.
Region of Interest
1
/// <summary>
2
/// Get a region-of-interest in the image feature.
3
/// .
4
/// </summary>
5
/// <param name="rect">ROI rectangle in normalized coordinates.</param>
6
/// <returns>Region-of-interest image feature.</returns>
7
MLImageFeature RegionOfInterest (Rect rect);
Copied!
INCOMPLETE.
1
/// <summary>
2
/// Get a region-of-interest in the image feature.
3
/// </summary>
4
/// <param name="rect">ROI rectangle in pixel coordinates.</param>
5
/// <returns>Region-of-interest image feature.</returns>
6
MLImageFeature RegionOfInterest (RectInt rect);
Copied!
If the rectangle extends beyond the bounds of the image, it will be clamped to fit within the image.
Creating Edge Features
1
/// <summary>
2
/// Create an Edge ML feature that is ready for prediction with Edge ML models.
3
/// </summary>
4
/// <param name="featureType">Feature type used to create the Edge ML feature.</param>
5
/// <returns>Edge ML feature.</returns>
6
MLEdgeFeature IMLEdgeFeature.Create (in MLFeatureType type);
Copied!
INCOMPLETE.
Creating Cloud Features
1
/// <summary>
2
/// Create a Cloud ML feature that is ready for prediction with Cloud ML models.
3
/// </summary>
4
/// <param name="featureType">Optional feature type used to create the Cloud ML feature.</param>
5
/// <returns>Cloud ML feature.</returns>
6
MLCloudFeature IMLCloudFeature.Create (in MLFeatureType featureType = default);
Copied!
INCOMPLETE.
MLDepthFeature
MLVideoFeature
Last modified 1mo ago