EGLStream

EGLStream is a mechanism that efficiently transfers a sequence of image frames from one API to another, e.g., from OpenGL^® to NVIDIA^® CUDA^®.

In EGLStream architecture a producer and a consumer are attached to each end of a stream object. A producer adds image frames to the stream. A consumer retrieves image frames from the stream.

EGLStream Producers

An EGLStream producer posts EGL™ image frames to an EGLStream. The producer is responsible for inserting each image frame into the EGLStream at the correct time so that the consumer can display the image frame for the appropriate period of time.

There are two types of producers:

CUDA producers post a CUDA array or CUDA pointer to the EGLStream as EGL image frames.
OpenGL producers post graphic surfaces to the EGLStream as EGL image frames.

EGLStream Consumers

An EGLStream consumer retrieves EGL image frames from the EGLStream. The consumer is responsible for noticing that an image frame is available and displaying it (or otherwise consuming it). The consumer is responsible for the latency of the acquired frame (the time elapsed between image frame retrieval and display).

There are three types of consumers:

CUDA consumers retrieve EGL image frames and deliver the frame information as a CUDA array or a CUDA pointer. The CUDA frames can then be processed in the CUDA domain.
OpenGL consumers retrieve EGL image frames that can then be bound as OpenGL textures for graphics rendering.
EGLOutput consumers (EGLDevice window system only) retrieve EGL image frames and render them directly to EGLOutput. An EGLOutput consumer is applicable when EGLOutput is used on the EGLDevice window system.

EGLStream Operation Modes

EGLStream operates in one of two modes: mailbox mode and FIFO mode.

Mailbox Mode

In mailbox mode, EGLStream conceptually operates as a mailbox. When the producer has a new image frame it empties the mailbox (discards the old contents) and inserts the new image frame into the mailbox. The consumer retrieves the image frame from the mailbox and examines it. When the consumer is finished examining the image frame, it either puts the frame back in the mailbox (if the mailbox is empty) or discards it (if the mailbox is not empty).

Timing is mainly controlled by the producer. The consumer operates with a fixed latency that it indicates to the producer through the EGLStream attribute EGL_CONSUMER_LATENCY_USEC_KHR. The consumer must notice when a new image frame is available in the EGLStream, retrieve it, and display it in the time indicated by EGL_CONSUMER_LATENCY_USEC_KHR. The producer controls when the image frame is displayed by inserting it into the stream at time t − EGL_CONSUMER_LATENCY_USEC_KHR, where t is the time when the image frame is to be displayed.

FIFO Mode

In FIFO mode images are not discarded. When a producer adds image frames to the stream they go in a FIFO queue for subsequent retrieval by the consumer. EGLStream sets the queue size when it creates the queue. If the producer attempts to insert a frame and the queue is full, the producer blocks until there is room in the queue. When the consumer retrieves an image frame from the EGLStream, it gets the image frame that immediately follows the image frame last retrieved (unless no such frame has been inserted yet, in which case it retrieves the same image frame as last time).

Timing of an EGLStream in FIFO mode is the responsibility of the consumer. Each image frame in the queue has a timestamp set by the producer. The consumer uses this timestamp to determine when the image frame is intended to be displayed.

EGLStream Pipeline

EGL provides functions to create and destroy EGLStreams, to query and set attributes of EGLStreams, and to connect EGLStreams to producers and consumers.

An EGLStream cannot be used until it is connected to a consumer and then to a producer. It must be connected to the consumer first.

An EGLStream may be connected to only one producer and one consumer. Once it is connected to a producer or a consumer, it remains connected to that entity until it is destroyed.

To build a simple EGLStream pipeline

Create the EGLDisplay object which the EGLStream is to bind to.
Call eglInitialize() to initialize EGL on the display.

To use the OpenGL consumer or EGLOutput consumer, you must initialize the rendering window before you create the EGLStream pipeline.

To use the CUDA producer-consumer, window system initialization is not required.

Create an EGLStream.

This example of creating the EGLStream is from eglstreamcube.c:

client->stream = eglCreateStreamKHR(demoState.display, streamAttr);
if (client->stream == EGL_NO_STREAM_KHR) {
    NvGlDemoLog("Couldn't create EGL stream.\n");
    goto fail;
}

Set streamAttr according to whether you are using mailbox mode or FIFO mode. For FIFO mode, the attribute EGL_STREAM_FIFO_LENGTH_KHR is initialized:

if (demoOptions.nFifo > 0) {
    streamAttr[numAttrs++] = EGL_STREAM_FIFO_LENGTH_KHR;
    streamAttr[numAttrs++] = demoOptions.nFifo;
}

Create a consumer of the appropriate type and connect it to the EGLStream.

This example of connecting an OpenGL consumer to the EGLStream is from eglstreamcube.c:
```
glBindTexture(GL_TEXTURE_EXTERNAL_OES, texture);
if (!eglStreamConsumerGLTextureExternalKHR(demoState.display, client->stream)) {
    NvGlDemoLog("Couldn't bind texture.\n");
    goto fail;
}
```
Remember that once an EGLStream is connected to a consumer, it remains connected to that consumer until the EGLStream is destroyed.
Create a producer of the appropriate type and connect it to the EGLStream.

This example of connecting an OpenGL producer to the EGLStream is from nvgldemo_main.c:
```
eglCreateStreamProducerSurfaceKHR(demoState.display,
                                  demoState.config,
                                  demoState.stream,
                                  srfAttrs);
```
Remember that once an EGLStream is connected to a producer, it remains connected to that producer until the EGLStream is destroyed.
The producer posts the image frames (the stream) to the consumer in order, depending on the type of stream (FIFO or mailbox).

In the OpenGL producer case, eglSwapBuffers() posts the buffer to the consumer.
The consumer acquires the image frame posted by the producer, uses it, and then releases the frame back to the stream. Methods for acquiring frames from a stream and releasing them back to a stream are dependent on the type of consumer.

This example of acquiring and releasing the frame in the OpenGL consumer case is from eglstreamcube.c:
```
eglStreamConsumerAcquireKHR(demoState.display, clientList[i].stream);
eglStreamConsumerReleaseKHR(demoState.display, client->stream);
```
In the OpenGL consumer case, if the application calls eglStreamConsumerAcquireKHR() twice on the same EGLStream without an intervening call to eglStreamConsumerReleaseKHR(), then eglStreamConsumerReleaseKHR() is implicitly called at the start of eglStreamConsumerAcquireKHR().

To destroy the EGLStream pipeline

Destroy the producer.
Destroy the consumer.
Destroy the EGLStream.
Destroy the window system resources.

The application must destroy resources in this order to produce a correct result.

EGLStream State

At any time after an EGLStream is created, it is in one of these states:

EGL_STREAM_STATE_CREATED_KHR: Created but not yet connected to a producer or a consumer.
EGL_STREAM_STATE_CONNECTING_KHR: Connected to a consumer but not yet connected to a producer.
EGL_STREAM_STATE_EMPTY_KHR: Connected to a consumer and a producer, but the producer has not yet posted any image frames.
EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR: The producer has posted at least one image frame that the consumer has not yet acquired.
EGL_STREAM_STATE_OLD_FRAME_AVAILABLE_KHR: The producer has posted at least one image frame, and the consumer has acquired the most recently posted image frame.
EGL_STREAM_STATE_DISCONNECTED_KHR: The producer, the consumer, or both are no longer connected to the EGLStream, i.e., they have been destroyed. Once the EGLStream is in this state it remains in this state until it is destroyed. In this state the only valid operations on the EGStream are eglQueryStreamKHR() and eglDestroyStreamKHR().

You query the EGLStream’s state like this:

EGLBoolean eglQueryStreamKHR(
           EGLDisplay dpy,
           EGLStreamKHR stream,
           EGLenum attribute,
           EGLuint64KHR *value);

These state transitions can occur:

EGL_STREAM_STATE_CREATED_KHR: The initial state, of an EGLStream that has just been created.
EGL_STREAM_STATE_CREATED_KHR to EGL_STREAM_STATE_CONNECTING_KHR: Occurs when a consumer is connected to the EGLStream.
EGL_STREAM_STATE_CONNECTING_KHR to EGL_STREAM_STATE_EMPTY_KHR: Occurs when a producer is connected to the EGLStream.
EGL_STREAM_STATE_EMPTY_KHR to EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR: Occurs the first time the producer inserts an EGL image frame.
EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR to EGL_STREAM_STATE_OLD_FRAME_AVAILABLE_KHR: Occurs when the consumer begins acquiring a newly posted EGL image frame.
EGL_STREAM_STATE_OLD_FRAME_AVAILABLE_KHR to EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR: Occurs when the producer posts a new EGL image frame.
Any state to EGL_STREAM_STATE_DISCONNECTED_KHR: Occurs when the producer or consumer is destroyed.

Building a Cross-Process EGLStream Pipeline

A cross-process EGLStream is one whose producer and consumer are in different processes.

To build a cross-process EGLStream pipeline

The consumer process creates an EGLStream.
The consumer process gets the file descriptor for the EGLStream and sends it through the socket to the producer process.
In the producer process, the producer receives the file descriptor through the socket and creates a corresponding EGLStream from the file descriptor. After the EGLStreams are created in both consumer and producer processes, the consumer and the producer are created and are connected to the EGLStreams, and through them, to each other.

Cross-Process EGLStream Example

Jetson Linux provides a pair of sample applications that constitute an example of communication through a cross-process EGLStream pipeline:

samples/opengles2/eglstreamcube is the consumer.
samples/opengles2/bubble is the producer.

You can demonstrate communication through a cross-process EGLStream pipeline by starting the consumer, then the producer.

The consumer application performs these operations:

Creates the stream and get the file descriptor of the stream (see eglstreamcube.c):

client->stream = eglCreateStreamKHR(demoState.display, streamAttr);
client->fd = eglGetStreamFileDescriptorKHR(demoState.display, client->stream);

Shares the file descriptor with the producer through the socket.

Binds the consumer end of the EGLStream with the OPENGL texture (see eglstreamcube.c):

glBindTexture(GL_TEXTURE_EXTERNAL_OES, texture);
eglStreamConsumerGLTextureExternalKHR(demoState.display, client->stream);

Latches the most recent image frame to the texture with eglStreamConsumerAcquireKHR():
```
eglStreamConsumerAcquireKHR(demoState.display,stream);
```
Renders the texture as one face of the cube.

The producer application performs these operations:

Receives the stream file descriptor through the socket (specified by a -eglstreamsocket option) and creates the stream (see nvgldemo_main.c):
```
eglCreateStreamFromFileDescriptorKHR(demoState.display, fd);
```

Creates the EGLStream surface (see nvgldemo_main.c):

eglCreateStreamProducerSurfaceKHR(
                    demoState.display,
                    demoState.config,
                    demoState.stream,
                    srfAttrs);

Creates an instance EGLContext and binds it to the thread by calling eglMakeCurrent() on the EGLStream surface created in step 2.
Renders the frames and calls eglSwapBuffers().

To run the cross-process EGLStream pipeline example

Enter this command to start the consumer application:

$ ./eglstreamcube -dispno 1 -layer 1 -windowoffset 1000 0 -socket /tmp/test &

Enter this command to start the producer application:
```
$ ./bubble -eglstreamsocket /tmp/test &
```

The bubble application content appears on one face of the cube.