MLImageFeature
class NatML.Features.MLImageFeature : MLFeature, IMLEdgeFeature, IMLCloudFeature
This feature contains a pixel buffer. Because computer vision models have similar pre-processing requirements, the image feature is able to perform these operations when predictions are made with it.
The image feature can be created from several common image inputs:
/// <summary>
/// Create an empty image feature.
/// </summary>
/// <param name="width">Image feature width.</param>
/// <param name="height">Image feature height.</param>
MLImageFeature (int width, int height);
The image feature can be created from an image size. This constructor initializes the image feature with empty data (i.e. transparent pixels).
This constructor allocates pixel buffer memory.
/// <summary>
/// Create an image feature.
/// </summary>
/// <param name="texture"></param>
MLImageFeature (Texture2D texture);
The image feature can be created from a
Texture2D
./// <summary>
/// Create an image feature.
/// </summary>
/// <param name="pixelBuffer">Pixel buffer to create image feature from.</param>
/// <param name="width">Pixel buffer width.</param>
/// <param name="height">Pixel buffer height.</param>
MLImageFeature (Color32[] pixelBuffer, int width, int height);
The image feature can be created from a color buffer.
This constructor copies the contents of
pixelBuffer
, and as such it allocates pixel buffer memory./// <summary>
/// Create an image feature.
/// </summary>
/// <param name="pixelBuffer">Pixel buffer to create image feature from.</param>
/// <param name="width">Pixel buffer width.</param>
/// <param name="height">Pixel buffer height.</param>
MLImageFeature (byte[] pixelBuffer, int width, int height);
The image feature can be created from a raw pixel buffer.
This constructor creates a view of the given
pixelBuffer
.The pixel buffer MUST have an
RGBA8888
layout./// <summary>
/// Create an image feature from a pixel buffer.
/// </summary>
/// <param name="pixelBuffer">Pixel buffer.</param>
/// <param name="width">Pixel buffer width.</param>
/// <param name="height">Pixel buffer height.</param>
MLImageFeature (NativeArray<byte> pixelBuffer, int width, int height);
The image feature can be created from a
NativeArray<byte>
. This is useful when making predictions with pixel data from Unity's Texture2D
API's.This constructor creates a view of the given
pixelBuffer
.The native array MUST have an
RGBA8888
layout.The native array MUST remain valid for the lifetime of the image feature.
/// <summary>
/// Create an image feature from a pixel buffer.
/// </summary>
/// <param name="pixelBuffer">Pixel buffer.</param>
/// <param name="width">Pixel buffer width.</param>
/// <param name="height">Pixel buffer height.</param>
MLImageFeature (void* pixelBuffer, int width, int height);
The image feature can be created from a native pixel buffer. This is useful when making predictions with data from native plugins or external libraries like OpenCV.
This constructor creates a view of the given
pixelBuffer
.The pixel buffer MUST have an
RGBA8888
layout.The pixel buffer MUST remain valid for the lifetime of the image feature.
/// <summary>
/// Create an image feature from a cloud feature.
/// </summary>
/// <param name="feature">Cloud feature.</param>
MLImageFeature (MLCloudFeature feature);
The image feature can be created from an
MLCloudFeature
. This is useful for making predictions with an MLCloudModel
.The
feature
MUST have an image
type.This constructor can only be used on the Unity main thread.
The image feature exposes its underlying
type
, along with convenience properties for inspecting the aforementioned type
./// <summary>
/// Feature type.
/// </summary>
MLFeatureType type { get; }
/// <summary>
/// Image width.
/// </summary>
int width { get; }
The image feature provides this convenience property for accessing the
width
of the feature type
./// <summary>
/// Image height.
/// </summary>
int height { get; }
The image feature provides this convenience property for accessing the
height
of the feature type
.When making Edge predictions on image features, some models might require that input data is normalized to some be within some range. The image feature provides these properties as an easy way to perform any required normalization.
The default range for image features is
[0.0, 1.0]
.
Specifying normalization coefficients on NatML Hub.
The specified normalization coefficients can then be used like so:
// Create edge model
var model = await MLEdgeModel.Create("@author/some-model");
// Create image feature
var imageFeature = new MLImageFeature(...);
// Apply normalization
(imageFeature.mean, imageFeature.std) = model.normalization;
/// <summary>
/// Normalization mean.
/// </summary>
Vector4 mean { get; set; }
The image feature supports specifying a per-channel normalization mean when creating an
MLEdgeFeature
./// <summary>
/// Normalization standard deviation.
/// </summary>
Vector4 std { get; set; }
The image feature supports specifying a per-channel normalization standard deviation when creating an
MLEdgeFeature
./// <summary>
/// Aspect mode.
/// </summary>
AspectMode aspectMode { get; set; }
The image feature supports specifying an aspect mode when creating an
MLEdgeFeature
with a different aspect ratio than the image feature. The aspectMode
specifies how the difference in aspect ratio should be handled:Aspect Mode | Example |
AspectMode.ScaleToFit | ![]() |
AspectMode.AspectFill | ![]() |
AspectMode.AspectFit | ![]() |
When the
aspectMode
is AspectMode.AspectFit
, the edge feature will be padded with transparent pixels, (0, 0, 0, 0)
.The image feature provides several accessors for reading and writing feature data:
/// <summary>
/// Copy the image feature into another feature.
/// </summary>
/// <param name="destination">Feature to copy data into.</param>
void CopyTo (MLImageFeature destination);
The image feature can copy its pixel data into another image feature.
The
destination
feature size (width and height) MUST match that of the image feature./// <summary>
/// Copy an image feature region of interest into another feature.
/// </summary>
/// <param name="destination">Feature to copy data into.</param>
/// <param name="rect">ROI rectangle in normalized coordinates.</param>
/// <param name="rotation">Rectangle clockwise rotation in degrees.</param>
/// <param name="background">Background color for unmapped pixels.</param>
void CopyTo (MLImageFeature destination, Rect rect, float rotation = 0f, Color32 background = default);
The image feature can copy a normalized region of interest rectangle into another image feature. The
rotation
defines the clockwise rotation about the center of the rect that will be applied before copying. The background
color defines the color of unmapped pixels. The ROI rectangle can also be defined in pixel coordinates:/// <summary>
/// Copy an image feature region of interest into another feature.
/// </summary>
/// <param name="destination">Feature to copy data into.</param>
/// <param name="rect">ROI rectangle in pixel coordinates.</param>
/// <param name="rotation">Rectangle clockwise rotation in degrees.</param>
/// <param name="background">Background color for unmapped pixels.</param>
void CopyTo (MLImageFeature destination, RectInt rect, float rotation = 0f, Color32 background = default);
/// <summary>
/// Copy the image feature data into a texture.
/// </summary>
/// <param name="destination">Texture to copy data into.</param>
/// <param name="upload">Whether to upload the pixel data to the GPU after copying.</param>
void CopyTo (Texture2D destination, bool upload = true);
The image feature can copy its pixel data into a
Texture2D
.The
destination
texture size (width and height) MUST match that of the image feature.This method MUST only be used from the Unity main thread.
/// <summary>
/// Create a texture from the image feature.
/// </summary>
/// <returns>Texture containing image feature data.</returns>
Texture2D ToTexture ();
The image feature can be converted into a
Texture2D
.This method creates a new texture every time it is called. As such, you must remember to release the texture when it is no longer needed.
This method MUST only be used from the Unity main thread.
/// <summary>
/// Copy image data from an ARFoundation image.
/// </summary>
/// <param name="feature">Image feature to copy data into.</param>
/// <param name="image">AR image.</param>
/// <param name="world">Whether AR image is from world-facing camera.</param>
/// <param name="orientation">Image orientation. If `Unknown`, this will default to the screen orientation.</param>
static void CopyFrom (this MLImageFeature feature, XRCpuImage image, bool world = true, ScreenOrientation orientation = 0);
The
MLXRExtensions.CopyFrom
extension method copies image data from an ARFoundation XRCpuImage
into an image feature. The size of the feature
MUST match the feature size of the AR image. For this, the MLXRExtensions.GetFeatureType
method can be used:/// <summary>
/// Get the ML feature type for a given AR image.
/// </summary>
/// <param name="image">AR image.</param>
/// <param name="orientation">Image orientation. If `Unknown`, this will default to the screen orientation.</param>
/// <returns>Feature type for image.</returns>
static MLImageType GetFeatureType (this XRCpuImage image, ScreenOrientation orientation = 0);
Both of these methods can be used like so:
// Given an ARCameraManager
ARCameraManager cameraManager = ...;
// Acquire the latest CPU image
if (cameraManager.TryAcquireLatestCpuImage(out var image)) {
// Get the AR image ML feature type
var featureType = image.GetFeatureType();
// Create a destination image feature
var feature = new MLImageFeature(featureType.width, featureType.height);
// Copy pixel data from the AR image to the image feature
feature.CopyFrom(image);
}
/// <summary>
/// Pin the image feature.
/// </summary>
ref T GetPinnableReference ();
The image feature supports pinning, allowing direct access to the underlying pixel data while bypassing all checks. This can be used in
unsafe
context with the fixed
statement:// Given an image feature
MLImageFeature feature = ...;
// Manually set the 12th element in the pixel data by pinning
fixed (byte* featureData = feature)
featureData[11] = 0xFF;
Pinning is mostly used for bulk read or write operations.
Do not use pinning unless you know exactly what you are doing. Mistakes are almost guaranteed to result in segmentation faults and hard crashes.
Image features expose methods for converting points and rectangles from arbitrary feature space into the image space.
These methods are useful for correcting for aspect ratio differences during prediction.
/// <summary>
/// Transform a normalized point from feature space into image space.
/// </summary>
/// <param name="rect">Input point.</param>
/// <param name="featureType">Feature type that defines the input space.</param>
/// <returns>Normalized point in image space.</returns>
Vector2 TransformPoint (Vector2 point, MLImageType featureType);
This method transforms a normalized point in the frame of the given
featureType
back into the image feature frame. It works by reverting any aspect ratio corrections that might have been made when creating an edge feature with the given featureType
./// <summary>
/// Transform a normalized region-of-interest rectangle from feature space into image space.
/// This method is used by detection models to correct for aspect ratio padding when making predictions.
/// </summary>
/// <param name="rect">Input rectangle.</param>
/// <param name="featureType">Feature type that defines the input space.</param>
/// <returns>Normalized rectangle in image space.</returns>
Rect TransformRect (Rect rect, MLImageType featureType);
This method transforms a normalized rectangle in the frame of the given
featureType
back into the image feature frame. Internally, this method uses TransformPoint
on the vertices of the rectangle.The image feature class defines routines for common vision operations:
/// <summary>
/// Perform non-max suppression on a set of candidate boxes.
/// </summary>
/// <param name="rects">Candidate boxes.</param>
/// <param name="scores">Candidate scores.</param>
/// <param name="maxIoU">Maximum IoU for preserving overlapping boxes.</param>
/// <returns>Indices of boxes to keep.</returns>
static int[] NonMaxSuppression (IReadOnlyList<Rect> rects, IReadOnlyList<float> scores, float maxIoU);
This method performs non-max suppression on a set of candidate boxes, returning the indices of boxes to keep.
/// <summary>
/// Calculate the intersection-over-union (IoU) of two rectangles.
/// </summary>
static float IntersectionOverUnion (Rect a, Rect b);
This method computes the IoU between two rectangles.
/// <summary>
/// Create an edge feature that is ready for prediction with edge models.
/// </summary>
/// <param name="featureType">Feature type used to create the edge feature.</param>
/// <returns>Edge feature.</returns>
MLEdgeFeature IMLEdgeFeature.Create (MLFeatureType type);
INCOMPLETE.
/// <summary>
/// Create a cloud feature that is ready for prediction with cloud models.
/// </summary>
/// <param name="featureType">Feature type used to create the cloud feature.</param>
/// <returns>Cloud feature.</returns>
MLCloudFeature IMLCloudFeature.Create (MLFeatureType type);
INCOMPLETE.
Last modified 6d ago