Comment on page
MLAudioFeature
class NatML.Features.MLAudioFeature : MLFeature, IMLEdgeFeature, IEnumerable<(MLAudioFeature feature, long timestamp)>
This feature contains raw audio data. Currently, NatML only supports floating-point linear PCM audio data.
The audio feature can be created from several different audio inputs:
/// <summary>
/// Create an audio feature from an audio clip.
/// </summary>
/// <param name="clip">Audio clip.</param>
/// <param name="duration">Optional duration to extract in seconds.</param>
MLAudioFeature (AudioClip clip, float duration = ...);
The audio feature can be created from an
AudioClip
, with the optional ability to specify the duration of the clip to extract./// <summary>
/// Create an audio feature from a sample buffer.
/// </summary>
/// <param name="sampleBuffer">Linear PCM sample buffer.</param>
/// <param name="sampleRate">Sample rate.</param>
/// <param name="channelCount">Channel count.</param>
MLAudioFeature (float[] sampleBuffer, int sampleRate, int channelCount);
The audio feature can be created from a sample buffer in managed memory, along with audio format information.
The sample buffer must be linear PCM and interleaved by channel.
/// <summary>
/// Create an audio feature from a sample buffer.
/// </summary>
/// <param name="sampleBuffer">Linear PCM sample buffer.</param>
/// <param name="sampleRate">Sample rate.</param>
/// <param name="channelCount">Channel count.</param>
MLAudioFeature (NativeArray<float> sampleBuffer, int sampleRate, int channelCount);
The audio feature can be created from a
NativeArray<float>
sample buffer, along with audio format information.The
sampleBuffer
MUST remain valid for the lifetime of the audio feature./// <summary>
/// Create an audio feature from a sample buffer.
/// </summary>
/// <param name="sampleBuffer">Linear PCM sample buffer.</param>
/// <param name="sampleRate">Sample rate.</param>
/// <param name="channelCount">Channel count.</param>
/// <param name="sampleCount">Total sample count.</param>
MLAudioFeature (float* sampleBuffer, int sampleRate, int channelCount, int sampleCount);
The audio feature can be created from a sample buffer, along with audio format information.
The
sampleBuffer
MUST remain valid for the lifetime of the audio feature./// <summary>
/// Create an audio feature from a sample buffer list.
/// </summary>
/// <param name="sampleBuffer">List of linear PCM sample buffers.</param>
/// <param name="sampleRate">Sample rate.</param>
/// <param name="channelCount">Channel count.</param>
MLAudioFeature (IEnumerable<float[]> bufferList, int sampleRate, int channelCount);
The audio feature can be created from an audio buffer list. This is useful for audio-based predictors that make predictions on longer segments of audio data, like speech-to-text models.
This constructor will combine each buffer in the list into one contiguous sample buffer. As such, this constructor allocates memory.
/// <summary>
/// Feature type.
/// </summary>
MLFeatureType type { get; }
The audio feature supports preprocessing when creating an
MLEdgeFeature
for edge predictions that use raw waveform data:/// <summary>
/// Desired sample rate for Edge predictions.
/// </summary>
int sampleRate { get; set; }
For Edge predictors that make predictions on raw audio waveform data, the audio feature can resample audio data to the specified
sampleRate
.The
sampleRate
is initialized to that of the audio data used to create the feature./// <summary>
/// Desired channel count for Edge predictions.
/// </summary>
int channelCount { get; set; }
For Edge predictors that make predictions on raw audio waveform data, the audio feature can multiplex or demultiplex audio data to the specified
channelCount
.The
channelCount
is initialized to that of the audio data used to create the feature.When making Edge predictions on audio features, some models might require that input data is normalized to some be within some range. The audio feature provides these properties as an easy way to perform any required normalization.
The default range for audio features is
[-1.0, 1.0]
.
Specifying normalization coefficients on NatML Hub.
The specified normalization coefficients can then be used like so:
// Fetch model data from NatML Hub
var modelData = await MLModelData.FromHub("@author/some-model");
// Create audio feature
var audioFeature = new MLAudioFeature(...);
// Apply normalization
audioFeature.mean = modelData.normalization.mean[0];
audioFeature.std = modelData.normalization.std[0];
/// <summary>
/// Normalization mean.
/// </summary>
float mean { get; set; } = 0;
/// <summary>
/// Normalization standard deviation.
/// </summary>
float std { get; set; } = 1f;
The audio feature supports specifying a normalization standard deviation when creating an
MLEdgeFeature
./// <summary>
/// Create an Edge ML feature that is ready for prediction with Edge ML models.
/// </summary>
/// <param name="featureType">Feature type used to create the Edge ML feature.</param>
/// <returns>Edge ML feature.</returns>
MLEdgeFeature IMLEdgeFeature.Create (in MLFeatureType type);
INCOMPLETE.
Last modified 9mo ago