MLImageFeature

class NatML.Features.MLImageFeature : MLFeature, IMLEdgeFeature, IMLCloudFeature

This feature contains a pixel buffer. Because computer vision models have similar pre-processing requirements, the image feature is able to perform these operations when predictions are made with it.

Creating the Feature

The image feature can be created from several common image inputs:

From an Image Size

/// <summary>
/// Create an empty image feature.
/// </summary>
/// <param name="width">Image feature width.</param>
/// <param name="height">Image feature height.</param>
MLImageFeature (int width, int height);

The image feature can be created from an image size. This constructor initializes the image feature with empty data (i.e. transparent pixels).

This constructor allocates pixel buffer memory.

From a Texture2D

/// <summary>
/// Create an image feature.
/// </summary>
/// <param name="texture"></param>
MLImageFeature (Texture2D texture);

The image feature can be created from a Texture2D.

The input texture MUST be readable.

This constructor allocates pixel buffer memory when the texture.format is not TextureFormat.RGBA32.

From a Color Buffer

/// <summary>
/// Create an image feature.
/// </summary>
/// <param name="pixelBuffer">Pixel buffer to create image feature from.</param>
/// <param name="width">Pixel buffer width.</param>
/// <param name="height">Pixel buffer height.</param>
MLImageFeature (Color32[] pixelBuffer, int width, int height);

The image feature can be created from a color buffer.

This constructor copies the contents of pixelBuffer, and as such it allocates pixel buffer memory.

From a Pixel Buffer

/// <summary>
/// Create an image feature.
/// </summary>
/// <param name="pixelBuffer">Pixel buffer to create image feature from.</param>
/// <param name="width">Pixel buffer width.</param>
/// <param name="height">Pixel buffer height.</param>
MLImageFeature (byte[] pixelBuffer, int width, int height);

The image feature can be created from a raw pixel buffer.

This constructor creates a view of the given pixelBuffer.

The pixel buffer MUST have an RGBA8888 layout.

From a Native Array

/// <summary>
/// Create an image feature from a pixel buffer.
/// </summary>
/// <param name="pixelBuffer">Pixel buffer.</param>
/// <param name="width">Pixel buffer width.</param>
/// <param name="height">Pixel buffer height.</param>
MLImageFeature (NativeArray<byte> pixelBuffer, int width, int height);

The image feature can be created from a NativeArray<byte>. This is useful when making predictions with pixel data from Unity's Texture2D API's.

This constructor creates a view of the given pixelBuffer.

The native array MUST have an RGBA8888 layout.

The native array MUST remain valid for the lifetime of the image feature.

From a Native Buffer

/// <summary>
/// Create an image feature from a pixel buffer.
/// </summary>
/// <param name="pixelBuffer">Pixel buffer.</param>
/// <param name="width">Pixel buffer width.</param>
/// <param name="height">Pixel buffer height.</param>
MLImageFeature (void* pixelBuffer, int width, int height);

The image feature can be created from a native pixel buffer. This is useful when making predictions with data from native plugins or external libraries like OpenCV.

This constructor creates a view of the given pixelBuffer.

The pixel buffer MUST have an RGBA8888 layout.

The pixel buffer MUST remain valid for the lifetime of the image feature.

From a Cloud Feature

/// <summary>
/// Create an image feature from a cloud feature.
/// </summary>
/// <param name="feature">Cloud feature.</param>
MLImageFeature (MLCloudFeature feature);

The image feature can be created from an MLCloudFeature. This is useful for making predictions with an MLCloudModel .

The feature MUST have an image type.

This constructor can only be used on the Unity main thread.

Inspecting the Feature

The image feature exposes its underlying type, along with convenience properties for inspecting the aforementioned type.

Feature Type

/// <summary>
/// Feature type.
/// </summary>
MLFeatureType type { get; }

Refer to the Inspecting the Feature section of the MLFeature class for more information.

The type is always an MLImageType.

Image Width

/// <summary>
/// Image width.
/// </summary>
int width { get; }

The image feature provides this convenience property for accessing the width of the feature type.

Image Height

/// <summary>
/// Image height.
/// </summary>
int height { get; }

The image feature provides this convenience property for accessing the height of the feature type.

Preprocessing the Feature

The image feature supports preprocessing when creating an MLEdgeFeature for edge predictions.

Normalization

When making Edge predictions on image features, some models might require that input data is normalized to some be within some range. The image feature provides these properties as an easy way to perform any required normalization.

The default range for image features is [0.0, 1.0].

When using NatML Hub, the normalization coefficients can be specified when creating a predictor:

Specifying normalization coefficients on NatML Hub.

The specified normalization coefficients can then be used like so:

// Create edge model
var model = await MLEdgeModel.Create("@author/some-model");
// Create image feature
var imageFeature = new MLImageFeature(...);
// Apply normalization
(imageFeature.mean, imageFeature.std) = model.normalization;

Mean

/// <summary>
/// Normalization mean.
/// </summary>
Vector4 mean { get; set; }

The image feature supports specifying a per-channel normalization mean when creating an MLEdgeFeature.

Standard Deviation

/// <summary>
/// Normalization standard deviation.
/// </summary>
Vector4 std { get; set; }

The image feature supports specifying a per-channel normalization standard deviation when creating an MLEdgeFeature.

Aspect Mode

/// <summary>
/// Aspect mode.
/// </summary>
AspectMode aspectMode { get; set; }

The image feature supports specifying an aspect mode when creating an MLEdgeFeature with a different aspect ratio than the image feature. The aspectMode specifies how the difference in aspect ratio should be handled:

Aspect Mode	Example
AspectMode.ScaleToFit
AspectMode.AspectFill
AspectMode.AspectFit

When the aspectMode is AspectMode.AspectFit, the edge feature will be padded with transparent pixels, (0, 0, 0, 0).

Accessing Feature Data

The image feature provides several accessors for reading and writing feature data:

Copying

/// <summary>
/// Copy the image feature into another feature.
/// </summary>
/// <param name="destination">Feature to copy data into.</param>
void CopyTo (MLImageFeature destination);

The image feature can copy its pixel data into another image feature.

The destination feature size (width and height) MUST match that of the image feature.

Extracting an ROI

/// <summary>
/// Copy an image feature region of interest into another feature.
/// </summary>
/// <param name="destination">Feature to copy data into.</param>
/// <param name="rect">ROI rectangle in normalized coordinates.</param>
/// <param name="rotation">Rectangle clockwise rotation in degrees.</param>
/// <param name="background">Background color for unmapped pixels.</param>
void CopyTo (MLImageFeature destination, Rect rect, float rotation = 0f, Color32 background = default);

The image feature can copy a normalized region of interest rectangle into another image feature. The rotation defines the clockwise rotation about the center of the rect that will be applied before copying. The background color defines the color of unmapped pixels. The ROI rectangle can also be defined in pixel coordinates:

/// <summary>
/// Copy an image feature region of interest into another feature.
/// </summary>
/// <param name="destination">Feature to copy data into.</param>
/// <param name="rect">ROI rectangle in pixel coordinates.</param>
/// <param name="rotation">Rectangle clockwise rotation in degrees.</param>
/// <param name="background">Background color for unmapped pixels.</param>
void CopyTo (MLImageFeature destination, RectInt rect, float rotation = 0f, Color32 background = default);

Copying to Texture

/// <summary>
/// Copy the image feature data into a texture.
/// </summary>
/// <param name="destination">Texture to copy data into.</param>
/// <param name="upload">Whether to upload the pixel data to the GPU after copying.</param>
void CopyTo (Texture2D destination, bool upload = true);

The image feature can copy its pixel data into a Texture2D.

The destination texture size (width and height) MUST match that of the image feature.

This method MUST only be used from the Unity main thread.

Converting to Texture

/// <summary>
/// Create a texture from the image feature.
/// </summary>
/// <returns>Texture containing image feature data.</returns>
Texture2D ToTexture ();

The image feature can be converted into a Texture2D.

This method creates a new texture every time it is called. As such, you must remember to release the texture when it is no longer needed.

This method MUST only be used from the Unity main thread.

Copying From an AR Image

/// <summary>
/// Copy image data from an ARFoundation image.
/// </summary>
/// <param name="feature">Image feature to copy data into.</param>
/// <param name="image">AR image.</param>
/// <param name="world">Whether AR image is from world-facing camera.</param>
/// <param name="orientation">Image orientation. If `Unknown`, this will default to the screen orientation.</param>
static void CopyFrom (this MLImageFeature feature, XRCpuImage image, bool world = true, ScreenOrientation orientation = 0);

The MLXRExtensions.CopyFrom extension method copies image data from an ARFoundation XRCpuImage into an image feature. The size of the feature MUST match the feature size of the AR image. For this, the MLXRExtensions.GetFeatureType method can be used:

/// <summary>
/// Get the ML feature type for a given AR image.
/// </summary>
/// <param name="image">AR image.</param>
/// <param name="orientation">Image orientation. If `Unknown`, this will default to the screen orientation.</param>
/// <returns>Feature type for image.</returns>
static MLImageType GetFeatureType (this XRCpuImage image, ScreenOrientation orientation = 0);

Both of these methods can be used like so:

// Given an ARCameraManager
ARCameraManager cameraManager = ...;
// Acquire the latest CPU image
if (cameraManager.TryAcquireLatestCpuImage(out var image)) {
    // Get the AR image ML feature type
    var featureType = image.GetFeatureType();
    // Create a destination image feature
    var feature = new MLImageFeature(featureType.width, featureType.height);
    // Copy pixel data from the AR image to the image feature
    feature.CopyFrom(image);
}

These extensions is distributed in the ai.natml.natml.arfoundation integration library.

Pinning

/// <summary>
/// Pin the image feature.
/// </summary>
ref T GetPinnableReference ();

The image feature supports pinning, allowing direct access to the underlying pixel data while bypassing all checks. This can be used in unsafe context with the fixed statement:

// Given an image feature
MLImageFeature feature = ...;
// Manually set the 12th element in the pixel data by pinning
fixed (byte* featureData = feature)
    featureData[11] = 0xFF;

Pinning is mostly used for bulk read or write operations.

Do not use pinning unless you know exactly what you are doing. Mistakes are almost guaranteed to result in segmentation faults and hard crashes.

Coordinate Transformations

Image features expose methods for converting points and rectangles from arbitrary feature space into the image space.

These methods are useful for correcting for aspect ratio differences during prediction.

Transforming Points

/// <summary>
/// Transform a normalized point from feature space into image space.
/// </summary>
/// <param name="rect">Input point.</param>
/// <param name="featureType">Feature type that defines the input space.</param>
/// <returns>Normalized point in image space.</returns>
Vector2 TransformPoint (Vector2 point, MLImageType featureType);

This method transforms a normalized point in the frame of the given featureType back into the image feature frame. It works by reverting any aspect ratio corrections that might have been made when creating an edge feature with the given featureType.

Transforming Rectangles

/// <summary>
/// Transform a normalized region-of-interest rectangle from feature space into image space.
/// This method is used by detection models to correct for aspect ratio padding when making predictions.
/// </summary>
/// <param name="rect">Input rectangle.</param>
/// <param name="featureType">Feature type that defines the input space.</param>
/// <returns>Normalized rectangle in image space.</returns>
Rect TransformRect (Rect rect, MLImageType featureType);

This method transforms a normalized rectangle in the frame of the given featureType back into the image feature frame. Internally, this method uses TransformPoint on the vertices of the rectangle.

Vision Operations

The image feature class defines routines for common vision operations:

Non Maximum Suppression

/// <summary>
/// Perform non-max suppression on a set of candidate boxes.
/// </summary>
/// <param name="rects">Candidate boxes.</param>
/// <param name="scores">Candidate scores.</param>
/// <param name="maxIoU">Maximum IoU for preserving overlapping boxes.</param>
/// <returns>Indices of boxes to keep.</returns>
static int[] NonMaxSuppression (IReadOnlyList<Rect> rects, IReadOnlyList<float> scores, float maxIoU);

This method performs non-max suppression on a set of candidate boxes, returning the indices of boxes to keep.

Intersection-over-Union

/// <summary>
/// Calculate the intersection-over-union (IoU) of two rectangles.
/// </summary>
static float IntersectionOverUnion (Rect a, Rect b);

This method computes the IoU between two rectangles.

Creating an Edge Feature

/// <summary>
/// Create an edge feature that is ready for prediction with edge models.
/// </summary>
/// <param name="featureType">Feature type used to create the edge feature.</param>
/// <returns>Edge feature.</returns>
MLEdgeFeature IMLEdgeFeature.Create (MLFeatureType type);

INCOMPLETE.

Creating a Cloud Feature

/// <summary>
/// Create a cloud feature that is ready for prediction with cloud models.
/// </summary>
/// <param name="featureType">Feature type used to create the cloud feature.</param>
/// <returns>Cloud feature.</returns>
MLCloudFeature IMLCloudFeature.Create (MLFeatureType type);

INCOMPLETE.