Machine Learning

Enhancing User Experience

NatDevice is built to tightly integrate with our cross-platform machine learning runtime for Unity Engine, NatML. NatML provides hardware accelerated machine learning to bring ML features to interactive media developers.

GitHub - natmlx/NatML: High performance, cross-platform machine learning for Unity Engine. Register at https://hub.natml.aiGitHub

NatML on GitHub.

Building Vision Pipelines

Computer vision models are becoming extremely popular for building interactive experiences, especially in user-generated content (think TikTok). The first step in building these experiences is streaming the camera preview.

Detecting objects from the camera stream.

The typical workflow begins by fetching model data from NatML and creating a predictor:

// Fetch model data from NatML
var modelData = await MLModelData.FromHub("@natsuite/mobilenet-v2");
// Deserialize the model
var model = modelData.Deserialize();
// Create the MobileNet predictor
var predictor = new MobileNetv2Predictor(model, modelData.labels);

Next, we start the camera preview with NatDevice:

// Discover a camera device
var query = new MediaDeviceQuery(MediaDeviceCriteria.CameraDevice);
var device = query.current as CameraDevice;
// Create a texture output
var textureOutput = new TextureOutput();
// Start the camera preview
device.StartRunning(textureOutput);

Then in the Update function, we can then create an MLImageFeature from the camera image and make ML predictions:

void Update () {
    // Check that the preview has started
    var previewTexture = textureOutput.texture;
    if (!previewTexture)
        return;
    // Create an image feature from the preview texture
    var imageFeature = new MLImageFeature(previewTexture);
    // Set the image feature pre-processing config
    (imageFeature.mean, imageFeature.std) = modelData.normalization;
    imageFeature.aspectMode = modelData.aspectMode;
    // Make a prediction
    var (label, score) = predictor.Predict(imageFeature);
    ...
}