EGL is an interface between Khronos rendering APIs such as OpenGL and the underlying native platform window system. It handles:

It also enables high-performance, accelerated, mixed-mode 2D and 3D rendering using other Khronos APIs.

EGLStream is an EGL object that exposes a set of APIs that transfer a sequence of EGL image frames between software components (e.g., video frames from NvMedia to OpenGL ES). Each EGLStream is associated with a producer that generates EGL image frames and inserts them into the EGLStream. The producer is responsible for inserting each EGL image frame into the EGLStream at the correct time so that the consumer can process the frame for the appropriate duration; for example, to display the EGL image frame.

Each EGLStream is associated with a consumer that retrieves EGL image frames from the EGLStream. The consumer is responsible for determining when an image frame is available and displaying it; or otherwise consuming it in some way. The consumer is also responsible for indicating the latency when that is possible. The latency is the time that elapses between when an image frame is retrieved from the EGLStream until the time it is delivered to the user.

An EGL image frame is the entity posted by the producer to the EGLStream and acquired later by the consumer. The sequence can be:

The color format for the image frame can be:

Different types of image frames must be posted to the EGLStream using different types of producers.

The EGLStream producer is the entity that inserts or posts EGL image frames into the EGLStream. In the NvMedia domain, there are two types of producers:

The EGLStream consumer is the entity that retrieves or acquires EGL image frames from the EGLStream. In the NvMedia domain, there are two types of consumers:

There are two types of EGLStream operation modes: mailbox mode and FIFO mode. Both modes are supported by NvMedia producers and consumers.

In mailbox mode, EGLStream conceptually operates as a mailbox. When a new frame is generated, the producer waits for a certain time for the consumer to process the frame. If the consumer does not consume the frame, the frame is discarded when the next frame is posted into the stream by the producer. This means that if the consumer consumes frames slower than the producer posts frames, some frames may be lost and never seen by the consumer. The producer is never stalled. Therefore, the timing of EGLStream in mailbox mode is controlled by the producer.

In FIFO mode, EGLStream operates as a FIFO rather than a mailbox. If the previous frame is not used by the consumer yet, the newly generated frame is added into an available slot in the queue. The FIFO size is set when it is created by the EGLStream. If the FIFO is full, the producer is stalled until there is room in the FIFO. This implies that the consumer has consumed some previous frames. In this mode, frames are not discarded.

The timing of an EGLStream in FIFO mode is the responsibility of the consumer. Each frame in the FIFO must be associated with a timestamp set by the producer. The consumer uses this timestamp to determine when the frame is intended to be processed.

EGL provides functions to create and destroy EGLStreams, for querying and setting attributes of EGLStreams, and for connecting EGLStreams to producers and consumers. To use the EGLStream, it must be connected to a consumer before being connected to a producer.

Each EGLStream must be connected to only one producer and one consumer. Once an EGLStream is connected to a consumer, it is connected to that consumer until the EGLStream is destroyed. Likewise, once an EGLStream is connected to a producer it is connected to that producer until the EGLStream is destroyed.

Before building simple EGLStream pipeline, you must initialize the EGL interface on the platform, and create the EGLDisplay.

A consumer acquires one frame, processes it, and releases it before acquiring the next frame. Consequently a consumer holds, at most, one frame at any time in both mailbox and FIFO mode.

Single Frame communication between producer and consumer in the EGLStream pipeline are as follows:

In the NvMedia EGLStream sample application, two threads are created:

In each of the thread process function, ProducerProc() and ConsumerProc(), the following diagram shows which functions are called in each thread for EGLStream frame communication.

Before releasing for the first time, the consumer acquires multiple frames. This means the consumer can hold more than one frame during the active processing time. Use FIFO mode for acquiring multiple frames. For NvMedia consumer type, the maximum frame number a consumer can hold is 6. The frame communication is the same as for acquiring a single case; except in ConsumerProc(), the ConsumerAcquireSurface is called more than once before ConsumerReleaseSurface gets called. Consumer must verify that the surface is released in the same order as they are acquired, i.e., first in first out manner.

To prevent the producer from overwriting the frame buffer when the consumer still processes the frame, it is recommended that you create a buffer pool at the producer side. The buffer pool size can be 2 or more, depending on the application. If buffer pool size equals 2, the producer can generate and post one frame to the EGLStream, then the producer can generate the second frame using the second buffer in the pool without waiting for the consumer to consume the first frame. When the consumer finishes and releases the first frame back to the producer, the producer can then reuse the first frame buffer for the third frame.

When acquiring multiple frames, the buffer pool size must be at least n+1.

To destroy the EGLStream pipeline, you must:

EGLStream sample functions that destroy producers are:

EGLStream sample functions that destroy consumers are:

For example:

The EGLStream sample function that destroys the EGLStream is:

The EGLStream sample code for that shuts down the window interface is:

GrUtilShutdown() is in the folder:

It calls GrUtilWindowTerm() and GrUtilDisplayTerm(), each has different implementation for different window interface.

After an EGLStream is created, at any given time, the EGLStream is in one of the following defined states:

The EGLStream state is queried as follows:

The following state transitions may occur:

To build a cross process EGLStream pipeline, the EGLStream is created by the consumer process. Then, the consumer gets the file descriptor for the EGLStream and send the file descriptor through socket.

In the producer process, the producer receives the file descriptor through socket, then creates the EGLStream from the file descriptor. After the EGLStream is created in both consumer and producer processes, the consumer and the producer is created and connected to the EGLStream.

The EGLStream can also be used across multiple partitions in a multi-OS environment. For more details about setting up and configuring multiple OSes, refer to the Foundation documentation.

To build a cross partition EGLStream pipeline, the EGLStream is created by the consumer process on consumer partition. The consumer creates a socket and waits for the producer to connect to that socket.

The producer uses the IP of the partition on which the consumer is running and the specific port on which EGLStream was created by the consumer to connect to it. After a successful connection, the producer creates an EGLStream using the same handle. After the EGLStream is created in both consumer and producer processes, the consumer and the producer are created and connected to the same EGLStream.

The NvMedia EGLStream sample application demonstrates how to:

Two types of NvMedia producers and two types of NvMedia consumers are supported:

To demonstrate the NvMedia/GL/CUDA/Screen EGLStream interoperability, the application includes sample code for creating and using GL producer, GL consumer, CUDA producer, and CUDA consumer.

The following shows the command syntax:

The above steps are not required for EGL device interface.

The following table describes the required command line options.

The following table describes the optional command line options.

This topic explains how to perform single process tasks.

Once properly setup, you can view the save output using yuvviewer. The output is saved to cuda.yuv.

Once properly setup, you can view the save output using yuvviewer. The output is saved to cuda.yuv.

Once properly setup, you can view the save output using yuvviewer. The output is saved to cuda.yuv.

This example is valid for Linux and EGLOutput.

This example is valid for Linux with EGLOutput.

The following cross process examples can be executed in two shells or run in the background in one shell.

The examples provided are similar to those described for single process samples except for an additional argument -consumer.

To run cross process samples, add standalone 1 for producer, and standalone 2 for consumer. Mention the consumer type in the producer thread using the -consumer argument.

The following cross partition examples can be executed in two shells, one for each partition.

The examples provided are similar to those described for cross process samples except for a few additional arguments: -consumervm, -ip, and -socketport.

To run cross partition samples, hv0 of both the consumer and producer partition must be configured to use two different IP address. For example, on VM1 (producer), set it to 12.0.0.2 using:

Similarly, on consumer partition, hv0 can be set to 12.0.0.1.

Add standalone 3 for producer, and standalone 4 for consumer.

In the producer process, mention the VM ID of the consumer partition using the -consumervm argument. Set -ip as the IP of consumer partition, i.e., 12.0.0.1, as in the above example. Set -socketport to the same port number used while creating EGLStream in the consumer process.

If the producer is running on a partition that does not own the display, specify the -defaultdisplay option in the command line while executing producer to use the default EGL display for EGL initialization.

Set the DISPLAY_VM environment variable on the consumer partition. If EGLOutput consumer is being used, set the DISPLAY_VM environment variable on the producer partition as well before executing the producer thread.

On VM1, where producer will be running:

On VM0, where consumer will be running:

On VM1, where producer will be running:

On VM0, where consumer will be running: