EGLStream

EGLStream is a mechanism that efficiently transfers a sequence of image frames from one API to another. APIs can be OpenGL, CUDA, or NvMedia. A producer and a consumer are attached to two ends of a stream object:
A producer adds image frames into the stream.
A consumer retrieves image frames from the stream.

EGLStream Producer

The EGLStream producer is the entity that posts EGL image frames into the EGLStream. The producer is responsible for inserting each image frame into the EGLStream at the correct time so that the consumer can display the image frame for the appropriate period of time.
Different types of producers:
CUDA producer: Posts a CUDA array or CUDA pointer as EGL image frames to the EGLStream.
GL producer: Posts graphic surfaces as EGL image frames to the EGLStream.

EGLStream Consumer

The EGLStream consumer is the entity that retrieves EGL image frames from the EGLStream. The consumer is responsible for noticing that an image frame is available and displaying it (or otherwise consuming it). The consumer is also responsible for indicating the latency when that is possible (the latency is the time that elapses between the time it is retrieved from the EGLStream until the time it is displayed).
Different types of consumers:
CUDA consumer: Retrieves EGL image frames and fills the frame information as a CUDA array or CUDA pointer. The CUDA frames can then be processed in the CUDA domain.
GL consumer: Retrieves EGL image frames that can then be bound as OpenGL textures for graphics rendering.
EGLOutput consumer (egldevice window system only): Retrieves EGL image frames and renders them directly to EGLOutput. This consumer is valid when EGLOutput is used on the egldevice window system.
 

EGLStream Operation Modes

There are two types of EGLStream operation modes: mailbox mode and FIFO mode.
Mailbox Mode
In mailbox mode, EGLStream conceptually operates as a mailbox. When the producer has a new image frame it empties the mailbox (discards the old contents) and inserts the new image frame into the mailbox. The consumer retrieves the image frame from the mailbox and examines it. When the consumer is finished examining the image frame, it is either placed back in the mailbox (if the mailbox is empty) or discarded (if the mailbox is not empty).
Timing is mainly controlled by the producer. The consumer operates with a fixed latency that it indicates to the producer through the EGLStream attribute EGL_CONSUMER_LATENCY_USEC_KHR. The consumer is expected to notice when a new image frame is available in the EGLStream, retrieve it, and display it in the time indicated by EGL_CONSUMER_LATENCY_USEC_KHR. The producer controls when the image frame is displayed by inserting it into the stream at time T - EGL_CONSUMER_LATENCY_USEC_KHR, where T is the time the image frame appears.
FIFO Mode
In FIFO mode no images are discarded. When a producer adds image frames to the stream, they are placed in a queue for subsequent retrieval by the consumer. EGLStream sets the queue size when it creates the queue. If the producer attempts to insert a frame and the FIFO queue is full, then the producer stalls until there is room in the FIFO queue. When the consumer retrieves an image frame from the EGLStream, it sees the image frame that immediately follows the image frame that it last retrieved (unless no such frame has been inserted yet, in which case it retrieves the same image frame that it retrieved last time).
Timing of an EGLStream in FIFO mode is the responsibility of the consumer. Each image frame in the FIFO has an associated timestamp set by the producer. The consumer uses this timestamp to determine when the image frame is intended to be displayed.

EGLStream Pipeline

EGL provides functions to create and destroy EGLStreams, to query and set attributes of EGLStreams, and to connect EGLStreams to producers and consumers.
Each EGLStream must be connected to only one producer and one consumer. Once an EGLStream is connected to a consumer, it is connected to that consumer until the EGLStream is destroyed. Likewise, once an EGLStream is connected to a producer, it is connected to that producer until the EGLStream is destroyed.
The EGLStream cannot be used until it has been connected to a consumer and then to a producer. It must be connected to a consumer before it is connected to a producer.
Building a Simple EGLStream Pipeline
Before you build a simple EGLStream pipeline, you must initialize the EGL interface on the platform and create the EGLDisplay.
1. Create the EGLDisplay object for the EGLStream to bind to it.
2. Call eglInitialize() to initialize EGL on the created display or on the default display.
If the GL consumer or EGLOutput consumer is used, you must initialize the rendering window before creating the EGLStream pipeline.
If the CUDA producer-consumer is used, window system initialization is not required.
3. Create an EGLStream.
Example for creating the EGLStream (from eglstreamcube.c):
client->stream = eglCreateStreamKHR(demoState.display, streamAttr);
if (client->stream == EGL_NO_STREAM_KHR) {
NvGlDemoLog("Couldn't create EGL stream.\n");
goto fail;
}
Depending on whether mailbox mode or FIFO mode is used, streamAttr is set accordingly. The attribute EGL_STREAM_FIFO_LENGTH_KHR is initialized for FIFO mode.
if (demoOptions.nFifo > 0) {
streamAttr[numAttrs++] = EGL_STREAM_FIFO_LENGTH_KHR;
streamAttr[numAttrs++] = demoOptions.nFifo;
}
4. Create a consumer and connect it to the EGLStream. It is specific to the consumer type.
Example for connecting a GL consumer to EGLStream (from eglstreamcube.c):
glBindTexture(GL_TEXTURE_EXTERNAL_OES, texture);
if (!eglStreamConsumerGLTextureExternalKHR(demoState.display, client->stream)) {
NvGlDemoLog("Couldn't bind texture.\n");
goto fail;
}
Once an EGLStream is connected to a consumer, it remains connected to the same consumer until the EGLStream is destroyed.
5. Create a producer and connect it to the EGLStream. It is specific to the producer type.
Example for connecting a GL producer to EGLStream (from nvgldemo_main.c):
eglCreateStreamProducerSurfaceKHR(demoState.display,
demoState.config,
demoState.stream,
srfAttrs);
Once an EGLStream is connected to a producer, it must remain connected to the same producer until the EGLStream is destroyed.
6. The producer posts the image frame to the consumer, depending on the producer type.
In the GL producer case, eglSwapBuffers posts the buffer to the consumer.
7. The consumer acquires the image frame posted by the producer, uses it, and then releases the frame back to the stream. Methods for acquiring frames from a stream and releasing them back to a stream are dependent on the type of consumer.
Example for acquiring and releasing the frame in the GL consumer case (from eglstreamcube.c):
eglStreamConsumerAcquireKHR(demoState.display,
clientList[i].stream)
eglStreamConsumerReleaseKHR(demoState.display,
client->stream)
In the GL consumer case, If eglStreamConsumerAcquireKHR() is called twice on the same EGLStream without an intervening call to eglStreamConsumerReleaseKHR(), then eglStreamConsumerReleaseKHR() is implicitly called at the start of eglStreamConsumerAcquireKHR().
Destroying the EGLStream Pipeline
Destroy the EGLStream pipeline in the following order:
1. Destroy the producer.
2. Destroy the consumer.
3. Destroy the EGLStream.
4. Destroy the window system resources.
EGLStream State
After an EGLStream is created, at any given time, the EGLStream is in one of the following defined states:
EGL_STREAM_STATE_CREATED_KHR: The EGLStream has been created but not yet connected to a producer or a consumer.
EGL_STREAM_STATE_CONNECTING_KHR: The EGLStream has been connected to a consumer but not yet connected to a producer.
EGL_STREAM_STATE_EMPTY_KHR: The EGLStream has been connected to a consumer and a producer, but the producer has not yet posted any image frames.
EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR: The producer has posted at least one image frame that the consumer has not yet acquired.
EGL_STREAM_STATE_OLD_FRAME_AVAILABLE_KHR: The producer has posted at least one image frame, and the consumer has acquired the most recently posted image frame.
EGL_STREAM_STATE_DISCONNECTED_KHR: The producer, the consumer, or both, are no longer connected to the EGLStream (e.g., they have been destroyed). Once the EGLStream is in this state it remains in this state until the EGLStream is destroyed. In this state only eglQueryStreamKHR() and eglDestroyStreamKHR() are valid operations.
 
The EGLStream state is queried as follows:
EGLBoolean eglQueryStreamKHR(
EGLDisplay dpy,
EGLStreamKHR stream,
EGLenum attribute,
EGLuint64KHR *value);
The following state transitions may occur:
1. EGL_STREAM_STATE_CREATED_KHR: A new EGLStream is created in this state.
2. EGL_STREAM_STATE_CREATED_KHR to EGL_STREAM_STATE_CONNECTING_KHR: Occurs when a consumer is connected to the EGLStream.
3. EGL_STREAM_STATE_CONNECTING_KHR to EGL_STREAM_STATE_EMPTY_KHR: Occurs when a producer is connected to the EGLStream.
4. EGL_STREAM_STATE_EMPTY_KHR to EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR: Occurs the first time the producer inserts an EGL image frame.
5. EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR to EGL_STREAM_STATE_OLD_FRAME_AVAILABLE_KHR: Occurs when the consumer begins acquiring a newly posted EGL image frame.
6. EGL_STREAM_STATE_OLD_FRAME_AVAILABLE_KHR to EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR: Occurs when the producer posts a new EGL image frame.
7. Any state to EGL_STREAM_STATE_DISCONNECTED_KHR: Occurs when the producer or consumer is destroyed.

Building a Cross-Process EGLStream Pipeline

A cross-process EGLStream is one where the producer and consumer are in different processes.
NVIDIA provides two options for establishing a cross-process stream. One uses the EGL_KHR_stream_cross_process_fd extension, and the other uses the EGL_NV_stream_remote extension. In both cases, producer and consumer processes are responsible for establishing a socket connection with each other before beginning to create the stream.
With the EGL_KHR_stream_cross_process_fd extension, one of the applications creates the EGLStream and then gets a file descriptor from it. In the sample code, the consumer creates the stream, but it could be either end. The creator then obtains a file descriptor from the stream and passes it over to the socket to the other process. That process receives the file descriptor and uses it to create the other end of the EGLStream. After both endpoints have created their EGLStream object, consumer and producer connection proceeds as in the single process case. The processes maintain ownership of the socket, and can continue to use it for any other communication they need.
With the EGL_NV_stream_remote extension, the processes may use the sockets they create for any initial handshaking and communication, but then pass ownership of them to the stream. Producer and consumer both create EGLStream endpoints from their socket. For cross-process streams within the same partition, NVIDIA requires that the type used for the socket in the EGLStream creation call be UNIX. INET sockets are not supported. (This is in constrast with cross-partition streams described below.)
You can execute the following cross-process examples in two shells or run one in the background in one shell.
Cross-process EGLStream example
Consult the samples/opengles2/eglstreamcube and samples/opengles2/bubble examples. eglstreamcube is the consumer, and bubble app is the producer.
The steps involved on the consumer side:
1. Create the stream and get the file descriptor of the stream (refer to eglstreamcube.c).
client->stream = eglCreateStreamKHR(demoState.display, streamAttr);
client->fd = eglGetStreamFileDescriptorKHR(demoState.display, client->stream);
2. Share the file descriptor with the producer through the socket.
3. Bind the consumer end of the EGLStream with the GL texture (refer to eglstreamcube.c).
glBindTexture(GL_TEXTURE_EXTERNAL_OES, texture);
eglStreamConsumerGLTextureExternalKHR(demoState.display, client->stream)
4. Latch the recent image frame to the texture with eglStreamConsumerAcquireKHR:
eglStreamConsumerAcquireKHR(demoState.display,stream)
5. Render the texture as one face of the cube.
The steps involved on the producer side:
1. Receive the stream file descriptor through the socket (given as an eglstreamsocket argument) and create the stream (refer to nvgldemo_main.c):
eglCreateStreamFromFileDescriptorKHR(demoState.display, fd)
2. Create the EGLStream surface (refer to nvgldemo_main.c).
eglCreateStreamProducerSurfaceKHR(
demoState.display,
demoState.config,
demoState.stream,
srfAttrs)
3. Create EGLContext and eglMakeCurrent on the EGLStream surface created in step 2.
4. Start to render the frames and eglSwapBuffers.
Run the cross process example
./eglstreamcube -dispno 1 -layer 1 -windowoffset 1000 0 -socket /tmp/test &
./bubble -eglstreamsocket /tmp/test &
The bubble app content appears on one face of the cube.

Building a Cross-Partition EGLStream Pipeline

A cross-partition EGLStream is one where the producer and consumer are in two different partitions in a multi-OS environment.
Creating a cross-partition stream is similar to the second method (using EGL_NV_stream_remote) for creating a cross-process extension described above. The two processes must establish socket communication with each other, and then pass ownership of that socket to the stream, with each end creating an EGLStream object from their socket. For cross-partition streams, NVIDIA requires that the type used for the socket be INET. (This is in constrast with cross-process streams described above.)
In our example code, the consumer acts as a server, opening a socket and listening for a connection on a known address. The producer then connects to that address, and both sides obtain the socket to use for the stream. They then proceed to create their EGLStreams. After both endpoints have created their EGLStream object, consumer and producer connection proceeds as in the single process case.
You can execute the following cross-partition examples in two shells, one for each partition.
To run cross-partition samples, hv0 of the consumer and producer partition must be configured to use two different IP addresses. For example, on VM1 (the producer partition), set it to 12.0.0.2 using:
ifconfig hv0 12.0.0.2
Similarly, on the consumer partition, set it to 12.0.0.1.
Cross-partition EGLStream example
Consult the samples/opengles2/eglstreamcrosspart example.
The steps involved on the consumer side:
1. Create the socket on the port number given as an argument (default: 8888) and set the created socket ID to g_ServerID (refer to nvgldemo_main.c).
2. Create the cross-partition EGLStream for the consumer end by setting the attributes (refer to nvgldemo_main.c).
EGL_STREAM_TYPE_NV:EGL_STREAM_CROSS_PARTITION_NV, EGL_STREAM_ENDPOINT_NV:EGL_STREAM_PRODUCER_NV,
EGL_SOCKET_HANDLE_NV: g_SEGLerverID
demoState.stream = eglCreateStreamKHR(demoState.display, attr);
3. Bind the consumer end of the EGLStream with the GL texture (refer to eglstreamcrosspart/consumer.cpp).
glBindTexture(GL_TEXTURE_EXTERNAL_OES, args.videoTexID);
eglStreamConsumerGLTextureExternalKHR(args.display, args.eglStream)
4. Latch the recent image frame to the texture with eglStreamConsumerAcquireKHR, render the frame, and call eglSwapBuffers (refer to eglstreamcrosspart/consumer.cpp):
eglStreamConsumerAcquireKHR(args.display,args.eglStream)
The steps involved on the producer side:
1. Connect to the socket with the given IP address and the port number arguments. Set the created socketID to g_ClientID (refer to nvgldemo_main.c)
2. Create the cross-partition EGLStream for the producer end by setting the attributes (refer to nvgldemo_main.c):
EGL_STREAM_TYPE_NV:EGL_STREAM_CROSS_PARTITION_NV, EGL_STREAM_ENDPOINT_NV:EGL_STREAM_PRODUCER_NV,
EGL_SOCKET_HANDLE_NV: g_ClientID
3. Create the EGLStream surface (refer to nvgldemo_main.c):
eglCreateStreamProducerSurfaceKHR(
demoState.display,
demoState.config,
demoState.stream,
srfAttrs)
4. Create EGLContext and eglMakeCurrent the EGLStream surface created in step 2 (refer to nvgldemo_main.c).
5. Start to render the frames and eglSwapBuffers (refer to eglstreamcrosspart/producer.cpp).
To run cross-partition sample
On Linux, run EGLStream producer:
/samples/opengles2/eglcrosspart/egldevice/eglcrosspart -proctype producer -ip 12.0.0.1 -port 1111 &