MLImageFeature
class NatML.Features.MLImageFeature : MLFeature, IMLEdgeFeature, IMLCloudFeature
This feature contains a pixel buffer. Because computer vision models have similar pre-processing requirements, the image feature is able to perform these operations when predictions are made with it.
Creating the Feature
The image feature can be created from several common image inputs:
From an Image Size
The image feature can be created from an image size. This constructor initializes the image feature with empty data (i.e. transparent pixels).
This constructor allocates pixel buffer memory.
From a Texture2D
The image feature can be created from a Texture2D
.
The input texture
MUST be readable.
This constructor allocates pixel buffer memory when the texture.format
is not TextureFormat.RGBA32
.
From a Color Buffer
The image feature can be created from a color buffer.
This constructor copies the contents of pixelBuffer
, and as such it allocates pixel buffer memory.
From a Pixel Buffer
The image feature can be created from a raw pixel buffer.
This constructor creates a view of the given pixelBuffer
.
The pixel buffer MUST have an RGBA8888
layout.
From a Native Array
The image feature can be created from a NativeArray<byte>
. This is useful when making predictions with pixel data from Unity's Texture2D
API's.
This constructor creates a view of the given pixelBuffer
.
The native array MUST have an RGBA8888
layout.
The native array MUST remain valid for the lifetime of the image feature.
From a Native Buffer
The image feature can be created from a native pixel buffer. This is useful when making predictions with data from native plugins or external libraries like OpenCV.
This constructor creates a view of the given pixelBuffer
.
The pixel buffer MUST have an RGBA8888
layout.
The pixel buffer MUST remain valid for the lifetime of the image feature.
From a Cloud Feature
The image feature can be created from an MLCloudFeature
. This is useful for making predictions with an MLCloudModel
.
The feature
MUST have an image
type.
This constructor can only be used on the Unity main thread.
Inspecting the Feature
The image feature exposes its underlying type
, along with convenience properties for inspecting the aforementioned type
.
Feature Type
Refer to the Inspecting the Feature section of the MLFeature
class for more information.
The type
is always an MLImageType
.
Image Width
The image feature provides this convenience property for accessing the width
of the feature type
.
Image Height
The image feature provides this convenience property for accessing the height
of the feature type
.
Preprocessing the Feature
The image feature supports preprocessing when creating an MLEdgeFeature
for edge predictions.
Normalization
When making Edge predictions on image features, some models might require that input data is normalized to some be within some range. The image feature provides these properties as an easy way to perform any required normalization.
The default range for image features is [0.0, 1.0]
.
When using NatML Hub, the normalization coefficients can be specified when creating a predictor:
The specified normalization coefficients can then be used like so:
Mean
The image feature supports specifying a per-channel normalization mean when creating an MLEdgeFeature
.
Standard Deviation
The image feature supports specifying a per-channel normalization standard deviation when creating an MLEdgeFeature
.
Aspect Mode
The image feature supports specifying an aspect mode when creating an MLEdgeFeature
with a different aspect ratio than the image feature. The aspectMode
specifies how the difference in aspect ratio should be handled:
When the aspectMode
is AspectMode.AspectFit
, the edge feature will be padded with transparent pixels, (0, 0, 0, 0)
.
Accessing Feature Data
The image feature provides several accessors for reading and writing feature data:
Copying
The image feature can copy its pixel data into another image feature.
The destination
feature size (width and height) MUST match that of the image feature.
Extracting an ROI
The image feature can copy a normalized region of interest rectangle into another image feature. The rotation
defines the clockwise rotation about the center of the rect that will be applied before copying. The background
color defines the color of unmapped pixels. The ROI rectangle can also be defined in pixel coordinates:
Copying to Texture
The image feature can copy its pixel data into a Texture2D
.
The destination
texture size (width and height) MUST match that of the image feature.
This method MUST only be used from the Unity main thread.
Converting to Texture
The image feature can be converted into a Texture2D
.
This method creates a new texture every time it is called. As such, you must remember to release the texture when it is no longer needed.
This method MUST only be used from the Unity main thread.
Copying From an AR Image
The MLXRExtensions.CopyFrom
extension method copies image data from an ARFoundation XRCpuImage
into an image feature. The size of the feature
MUST match the feature size of the AR image. For this, the MLXRExtensions.GetFeatureType
method can be used:
Both of these methods can be used like so:
These extensions is distributed in the ai.natml.natml.arfoundation
integration library.
Pinning
The image feature supports pinning, allowing direct access to the underlying pixel data while bypassing all checks. This can be used in unsafe
context with the fixed
statement:
Pinning is mostly used for bulk read or write operations.
Do not use pinning unless you know exactly what you are doing. Mistakes are almost guaranteed to result in segmentation faults and hard crashes.
Coordinate Transformations
Image features expose methods for converting points and rectangles from arbitrary feature space into the image space.
These methods are useful for correcting for aspect ratio differences during prediction.
Transforming Points
This method transforms a normalized point in the frame of the given featureType
back into the image feature frame. It works by reverting any aspect ratio corrections that might have been made when creating an edge feature with the given featureType
.
Transforming Rectangles
This method transforms a normalized rectangle in the frame of the given featureType
back into the image feature frame. Internally, this method uses TransformPoint
on the vertices of the rectangle.
Vision Operations
The image feature class defines routines for common vision operations:
Non Maximum Suppression
This method performs non-max suppression on a set of candidate boxes, returning the indices of boxes to keep.
Intersection-over-Union
This method computes the IoU between two rectangles.
Creating an Edge Feature
INCOMPLETE.
Creating a Cloud Feature
INCOMPLETE.
Last updated