Media Devices

Record More than Just the Screen

NatCorder seamlessly integrates with our cross-platform media device API for Unity Engine, NatDevice. NatDevice provides a lightweight API for accessing and using hardware cameras and microphones. It has the same cross-platform support as NatCorder, and has the same minimum requirements.

GitHub - natmlx/NatDevice: High performance, cross-platform camera and microphone streaming for Unity Engine. Register at https://hub.natml.aiGitHub

NatDevice on GitHub.

Recording the Camera

Unity's API for using the hardware camera, WebCamTexture, has several limitations that prevent it from being used in a production environment.

First and foremost, the camera preview is not corrected for orientation on mobile platforms. This means that when the camera is started and displayed, it will be rotated wrongly. Correcting this orientation for display is a non-trivial process. But correcting the orientation for recording is orders of magnitude more difficult, as you will have to manually rotate each pixel buffer in memory. NatDevice handles all these gory details automatically.

Beyond the above point, NatDevice provides an expansive API for camera control. NatDevice allows for controlling the camera exposure, focus, flash, zoom, torch, white balance mode, and so on.

The typical NatDevice camera workflow is to first find a camera device to stream the camera preview from:

// Get a camera device
var query = new MediaDeviceQuery(MediaDeviceCriteria.CameraDevice);
var device = query.current as CameraDevice;

Then we can start streaming the camera preview:

// Set the preview resolution to full HD
device.previewResolution = (1920, 1080);
// Then start streaming the preview
var textureOutput = new TextureOutput();
device.StartRunning(textureOutput);
var previewTexture = await textureOutput;

Back on the NatCorder side, we will first create a recorder. Since we are recording the camera preview, we need to make sure that the frame size of the recorder is set to that of the preview texture.

// Create a recorder
var recorder = new MP4Recorder(previewTexture.width, previewTexture.height, 30);

We can now commit a bunch of frames from the camera preview into our recording.

// Record a bunch of frames
var clock = new RealtimeClock();
for (var i = 0; i < 300; ++i) {
    recorder.CommitFrame(previewTexture.GetPixels32(), clock.timestamp);
    await Task.Yield();
}

And once we are done recording camera preview frames, we can stop recording.

// Finish recording
var path = await recorder.FinishWriting();

Recording the Microphone

If you plan to record user audio through the microphone, you will want to use NatDevice. Unity's Microphone API is known to often produce glitchy audio.

We created our NatMic API because we couldn't get around Unity's glitchy microphone audio. NatMic was later merged with NatCam to create NatDevice.

NatDevice on the other hand will stream audio sample buffers directly from the microphone to your code in .NET. To start, get a microphone:

// Get a microphone
var query = new MediaDeviceQuery(MediaDeviceCriteria.AudioDevice);
var device = query.current as AudioDevice;

Then when you start recording, stream audio samples from the NatDevice AudioDevice to the recorder:

// Create a recorder
var recorder = new WAVRecorder(device.sampleRate, device.channelCount);
// Start the microphone
device.StartRunning(audioBuffer => {
    // And pass sample buffers directly to the recorder
    recorder.CommitSamples(audioBuffer.sampleBuffer);
});

When finishing recording, stop the audio device before stopping the recording:

// Stop streaming audio samples to the recorder
device.StopRunning();
// Then stop recording
var path = await recorder.FinishWriting();

And that's it!