Recording Textures

For some more advanced use cases, you might want to record video frames from a Texture in Unity. Typically, recording from a Texture requires you to first read the pixel data from the GPU before committing the data to the recorder.
Unity provides two ways to readback pixel data from a Texture into system memory. They differ based on whether the client is willing to accept a deferred completion of the readback.
Synchronous Readbacks
A synchronous readback will request pixel data from the GPU and block until the request has been completed.
1
// Say we have some `RenderTexture`
2
RenderTexture renderTexture = ...;
3
var width = renderTexture.width;
4
var height = renderTexture.height;
5
// We can perform a synchronous readback using a `Texture2D`
6
var readbackTexture = new Texture2D(width, height);
7
RenderTexture.active = renderTexture;
8
readbackTexture.ReadPixels(new Rect(0, 0, width, height), 0, 0);
9
RenderTexture.active = null;
Copied!
Once the above code executes, the pixel data will be accessible through the Texture2D.
1
// Commit the pixel buffer
2
recorder.CommitFrame(readbackTexture.GetPixels32());
Copied!
Due to the concurrent nature of GPU's, such a readback will typically result in a pipeline stall. This means that both the CPU and GPU cannot do any other work until the request is complete. Pipeline stalls are typically very expensive, and will cause a very noticeable frame rate hit in most cases.
Though expensive, synchronous readbacks exhibit very predictable memory consumption patterns.
Asynchronous Readbacks
Unity provides the AsyncGPUReadback API for performing an asynchronous readback.
1
// Issue a readback request
2
AsyncGPUReadback.Request(renderTexture, 0, request => {
3
    // Once complete, access the data container
4
    var nativeArray = request.GetData<byte>()
5
    // And commit the pixel buffer
6
    recorder.CommitFrame(nativeArray.ToArray());
7
});
Copied!
This readback will request the data from the GPU but will not wait for the GPU to complete the request. When the transfer is complete, Unity will invoke the provided callback. We can then commit the pixel buffer to the recorder within this callback.
The advantage with this approach is that it provides much better performance over synchronous readbacks. There are a few disadvantages with this approach:
It adds latency on the order of a few frames
It has less predictable memory consumption patterns than synchronous readbacks.
It is not supported on all platforms and devices, so you must check at runtime.
​CameraInput will default to asynchronous readbacks on devices that support it.
If your app is GPU-bound, async readbacks can cause memory consumption to explode, resulting in a hard crash.
Texture Inputs
NatCorder provides primitives that implement both synchronous and asynchronous readbacks, so that you don't have to. The TextureInput class handles synchronous readbacks:
1
// Create a texture input
2
var recorder = ...;
3
var clock = ...;
4
var textureInput = new TextureInput(recorder);
5
// Commit video frames from a texture
6
var renderTexture = RenderTexture.GetTemporary(...);
7
textureInput.CommitFrame(renderTexture, clock.timestamp);
Copied!
Similarly for asynchronous readbacks, NatCorder provides AsyncTextureInput:
1
// Create a texture input
2
var textureInput = new AsyncTextureInput(recorder);
3
// Commit video frames from a texture
4
var renderTexture = RenderTexture.GetTemporary(...);
5
textureInput.CommitFrame(renderTexture, clock.timestamp);
Copied!

Workflows - Previous

Recorder Inputs

Next - Workflows

Offline Recording

Last modified 4mo ago

Copy link

Contents

Synchronous Readbacks

Asynchronous Readbacks

Texture Inputs