NVIDIA Tegra
NVIDIA DRIVE OS 5.1 Linux SDK

Developer Guide
5.1.12.0 Release


 
EGLStream
 
EGLStream Producer
EGLStream Consumer
EGLStream Operation Modes
EGLStream Pipeline
Building a Cross-Process EGLStream Pipeline
Building a Cross-Partition EGLStream Pipeline
Building a Cross-System EGLStream Pipeline
EGLStream is a mechanism that efficiently transfers a sequence of image frames from one API to another. APIs can be OpenGL, CUDA, or NvMedia. A producer and a consumer are attached to two ends of a stream object:
A producer adds image frames into the stream.
A consumer retrieves image frames from the stream.
EGLStream Producer
The EGLStream producer is the entity that posts EGL image frames into the EGLStream. The producer is responsible for inserting each image frame into the EGLStream at the correct time so that the consumer can display the image frame for the appropriate period of time.
Different types of producers:
CUDA producer: Posts a CUDA array or CUDA pointer as EGL image frames to the EGLStream.
GL producer: Posts graphic surfaces as EGL image frames to the EGLStream.
EGLStream Consumer
The EGLStream consumer is the entity that retrieves EGL image frames from the EGLStream. The consumer is responsible for noticing that an image frame is available and displaying it (or otherwise consuming it). The consumer is also responsible for indicating the latency when that is possible (the latency is the time that elapses between the time it is retrieved from the EGLStream until the time it is displayed).
Different types of consumers:
CUDA consumer: Retrieves EGL image frames and fills the frame information as a CUDA array or CUDA pointer. The CUDA frames can then be processed in the CUDA domain.
GL consumer: Retrieves EGL image frames that can then be bound as OpenGL textures for graphics rendering.
EGLOutput consumer (egldevice window system only): Retrieves EGL image frames and renders them directly to EGLOutput. This consumer is valid when EGLOutput is used on the egldevice window system.
 
EGLStream Operation Modes
There are two types of EGLStream operation modes: mailbox mode and FIFO mode.
Mailbox Mode
In mailbox mode, EGLStream conceptually operates as a mailbox. When the producer has a new image frame it empties the mailbox (discards the old contents) and inserts the new image frame into the mailbox. The consumer retrieves the image frame from the mailbox and examines it. When the consumer is finished examining the image frame, it is either placed back in the mailbox (if the mailbox is empty) or discarded (if the mailbox is not empty).
Timing is mainly controlled by the producer. The consumer operates with a fixed latency that it indicates to the producer through the EGLStream attribute EGL_CONSUMER_LATENCY_USEC_KHR. The consumer is expected to notice when a new image frame is available in the EGLStream, retrieve it, and display it in the time indicated by EGL_CONSUMER_LATENCY_USEC_KHR. The producer controls when the image frame is displayed by inserting it into the stream at time T - EGL_CONSUMER_LATENCY_USEC_KHR, where T is the time the image frame appears.
FIFO Mode
In FIFO mode no images are discarded. When a producer adds image frames to the stream, they are placed in a queue for subsequent retrieval by the consumer. EGLStream sets the queue size when it creates the queue. If the producer attempts to insert a frame and the FIFO queue is full, then the producer stalls until there is room in the FIFO queue. When the consumer retrieves an image frame from the EGLStream, it sees the image frame that immediately follows the image frame that it last retrieved (unless no such frame has been inserted yet, in which case it retrieves the same image frame that it retrieved last time).
Timing of an EGLStream in FIFO mode is the responsibility of the consumer. Each image frame in the FIFO has an associated timestamp set by the producer. The consumer uses this timestamp to determine when the image frame is intended to be displayed.
EGLStream Pipeline
EGL provides functions to create and destroy EGLStreams, to query and set attributes of EGLStreams, and to connect EGLStreams to producers and consumers.
Each EGLStream must be connected to only one producer and one consumer. Once an EGLStream is connected to a consumer, it is connected to that consumer until the EGLStream is destroyed. Likewise, once an EGLStream is connected to a producer, it is connected to that producer until the EGLStream is destroyed.
The EGLStream cannot be used until it has been connected to a consumer and then to a producer. It must be connected to a consumer before it is connected to a producer.
Building a Simple EGLStream Pipeline
Before you build a simple EGLStream pipeline, you must initialize the EGL interface on the platform and create the EGLDisplay.
1. Create the EGLDisplay object for the EGLStream to bind to it.
2. Call eglInitialize() to initialize EGL on the created display or on the default display.
If the GL consumer or EGLOutput consumer is used, you must initialize the rendering window before creating the EGLStream pipeline.
If the CUDA producer-consumer is used, window system initialization is not required.
3. Create an EGLStream.
Example for creating the EGLStream (from eglstreamcube.c):
client->stream = eglCreateStreamKHR(demoState.display, streamAttr);
if (client->stream == EGL_NO_STREAM_KHR) {
NvGlDemoLog("Couldn't create EGL stream.\n");
goto fail;
}
Depending on whether mailbox mode or FIFO mode is used, streamAttr is set accordingly. The attribute EGL_STREAM_FIFO_LENGTH_KHR is initialized for FIFO mode.
if (demoOptions.nFifo > 0) {
streamAttr[numAttrs++] = EGL_STREAM_FIFO_LENGTH_KHR;
streamAttr[numAttrs++] = demoOptions.nFifo;
}
4. Create a consumer and connect it to the EGLStream. It is specific to the consumer type.
Example for connecting a GL consumer to EGLStream (from eglstreamcube.c):
glBindTexture(GL_TEXTURE_EXTERNAL_OES, texture);
if (!eglStreamConsumerGLTextureExternalKHR(demoState.display, client->stream)) {
NvGlDemoLog("Couldn't bind texture.\n");
goto fail;
}
Once an EGLStream is connected to a consumer, it remains connected to the same consumer until the EGLStream is destroyed.
5. Create a producer and connect it to the EGLStream. It is specific to the producer type.
Example for connecting a GL producer to EGLStream (from nvgldemo_main.c):
eglCreateStreamProducerSurfaceKHR(demoState.display,
demoState.config,
demoState.stream,
srfAttrs);
Once an EGLStream is connected to a producer, it must remain connected to the same producer until the EGLStream is destroyed.
6. The producer posts the image frame to the consumer, depending on the producer type.
In the GL producer case, eglSwapBuffers posts the buffer to the consumer.
7. The consumer acquires the image frame posted by the producer, uses it, and then releases the frame back to the stream. Methods for acquiring frames from a stream and releasing them back to a stream are dependent on the type of consumer.
Example for acquiring and releasing the frame in the GL consumer case (from eglstreamcube.c):
eglStreamConsumerAcquireKHR(demoState.display,
clientList[i].stream)
eglStreamConsumerReleaseKHR(demoState.display,
client->stream)
In the GL consumer case, If eglStreamConsumerAcquireKHR() is called twice on the same EGLStream without an intervening call to eglStreamConsumerReleaseKHR(), then eglStreamConsumerReleaseKHR() is implicitly called at the start of eglStreamConsumerAcquireKHR().
Destroying the EGLStream Pipeline
Destroy the EGLStream pipeline in the following order:
1. Destroy the producer.
2. Destroy the consumer.
3. Destroy the EGLStream.
4. Destroy the window system resources.
EGLStream State
After an EGLStream is created, at any given time, the EGLStream is in one of the following defined states:
EGL_STREAM_STATE_CREATED_KHR: The EGLStream has been created but not yet connected to a producer or a consumer.
EGL_STREAM_STATE_CONNECTING_KHR: The EGLStream has been connected to a consumer but not yet connected to a producer.
EGL_STREAM_STATE_EMPTY_KHR: The EGLStream has been connected to a consumer and a producer, but the producer has not yet posted any image frames.
EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR: The producer has posted at least one image frame that the consumer has not yet acquired.
EGL_STREAM_STATE_OLD_FRAME_AVAILABLE_KHR: The producer has posted at least one image frame, and the consumer has acquired the most recently posted image frame.
EGL_STREAM_STATE_DISCONNECTED_KHR: The producer, the consumer, or both, are no longer connected to the EGLStream (e.g., they have been destroyed). Once the EGLStream is in this state it remains in this state until the EGLStream is destroyed. In this state only eglQueryStreamKHR() and eglDestroyStreamKHR() are valid operations.
 
The EGLStream state is queried as follows:
EGLBoolean eglQueryStreamKHR(
EGLDisplay dpy,
EGLStreamKHR stream,
EGLenum attribute,
EGLuint64KHR *value);
The following state transitions may occur:
1. EGL_STREAM_STATE_CREATED_KHR: A new EGLStream is created in this state.
2. EGL_STREAM_STATE_CREATED_KHR to EGL_STREAM_STATE_CONNECTING_KHR: Occurs when a consumer is connected to the EGLStream.
3. EGL_STREAM_STATE_CONNECTING_KHR to EGL_STREAM_STATE_EMPTY_KHR: Occurs when a producer is connected to the EGLStream.
4. EGL_STREAM_STATE_EMPTY_KHR to EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR: Occurs the first time the producer inserts an EGL image frame.
5. EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR to EGL_STREAM_STATE_OLD_FRAME_AVAILABLE_KHR: Occurs when the consumer begins acquiring a newly posted EGL image frame.
6. EGL_STREAM_STATE_OLD_FRAME_AVAILABLE_KHR to EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR: Occurs when the producer posts a new EGL image frame.
7. Any state to EGL_STREAM_STATE_DISCONNECTED_KHR: Occurs when the producer or consumer is destroyed.
Building a Cross-Process EGLStream Pipeline
A cross-process EGLStream is one where the producer and consumer are in different processes.
To build a cross-process EGLStream pipeline, the consumer process creates the EGLStream. The consumer then gets the file descriptor for the EGLStream and sends the file descriptor through the socket to the producer process.
In the producer process, the producer receives the file descriptor through the socket, then creates the EGLStream from the file descriptor. After the EGLStream is created in both consumer and producer processes, the consumer and the producer are created and connected to the EGLStream.
You can execute the following cross-process examples in two shells or run one in the background in one shell.
Cross-process EGLStream example
Consult the samples/opengles2/eglstreamcube and samples/opengles2/bubble examples. eglstreamcube is the consumer, and bubble app is the producer.
The steps involved on the consumer side:
1. Create the stream and get the file descriptor of the stream (refer to eglstreamcube.c).
client->stream = eglCreateStreamKHR(demoState.display, streamAttr);
client->fd = eglGetStreamFileDescriptorKHR(demoState.display, client->stream);
2. Share the file descriptor with the producer through the socket.
3. Bind the consumer end of the EGLStream with the GL texture (refer to eglstreamcube.c).
glBindTexture(GL_TEXTURE_EXTERNAL_OES, texture);
eglStreamConsumerGLTextureExternalKHR(demoState.display, client->stream)
4. Latch the recent image frame to the texture with eglStreamConsumerAcquireKHR:
eglStreamConsumerAcquireKHR(demoState.display,stream)
5. Render the texture as one face of the cube.
The steps involved on the producer side:
1. Receive the stream file descriptor through the socket (given as an eglstreamsocket argument) and create the stream (refer to nvgldemo_main.c):
eglCreateStreamFromFileDescriptorKHR(demoState.display, fd)
2. Create the EGLStream surface (refer to nvgldemo_main.c).
eglCreateStreamProducerSurfaceKHR(
demoState.display,
demoState.config,
demoState.stream,
srfAttrs)
3. Create EGLContext and eglMakeCurrent on the EGLStream surface created in step 2.
4. Start to render the frames and eglSwapBuffers.
Run the cross process example
./eglstreamcube -dispno 1 -layer 1 -windowoffset 1000 0 -socket /tmp/test &
./bubble -eglstreamsocket /tmp/test &
The bubble app content appears on one face of the cube.
Building a Cross-Partition EGLStream Pipeline
A cross-partition EGLStream is one where the producer and consumer are in two different partitions in a multi-OS environment.
To build a cross-partition EGLStream pipeline, the consumer process creates the EGLStream on the consumer partition. The consumer then creates a socket and waits for the producer to connect to that socket.
The producer uses the IP of the partition on which the consumer is running and the specific port on which the consumer created the EGLStream to connect to it. After a successful connection, the producer creates an EGLStream using the same handle. After the EGLStream is created in both consumer and producer processes, the consumer and the producer are created and connected to the same EGLStream.
You can execute the following cross-partition examples in two shells, one for each partition.
To run cross-partition samples, hv0 of the consumer and producer partition must be configured to use two different IP addresses. For example, on VM1 (the producer partition), set it to 12.0.0.2 using:
ifconfig hv0 12.0.0.2
Similarly, on the consumer partition, set it to 12.0.0.1.
Cross-partition EGLStream example
Consult the samples/opengles2/eglstreamcrosspart example.
The steps involved on the consumer side:
1. Create the socket on the port number given as an argument (default: 8888) and set the created socket ID to g_ServerID (refer to nvgldemo_main.c).
2. Create the cross-partition EGLStream for the consumer end by setting the attributes (refer to nvgldemo_main.c).
EGL_STREAM_TYPE_NV:EGL_STREAM_CROSS_PARTITION_NV, EGL_STREAM_ENDPOINT_NV:EGL_STREAM_PRODUCER_NV,
EGL_SOCKET_HANDLE_NV: g_SEGLerverID
demoState.stream = eglCreateStreamKHR(demoState.display, attr);
3. Bind the consumer end of the EGLStream with the GL texture (refer to eglstreamcrosspart/consumer.cpp).
glBindTexture(GL_TEXTURE_EXTERNAL_OES, args.videoTexID);
eglStreamConsumerGLTextureExternalKHR(args.display, args.eglStream)
4. Latch the recent image frame to the texture with eglStreamConsumerAcquireKHR, render the frame, and call eglSwapBuffers (refer to eglstreamcrosspart/consumer.cpp):
eglStreamConsumerAcquireKHR(args.display,args.eglStream)
The steps involved on the producer side:
1. Connect to the socket with the given IP address and the port number arguments. Set the created socketID to g_ClientID (refer to nvgldemo_main.c)
2. Create the cross-partition EGLStream for the producer end by setting the attributes (refer to nvgldemo_main.c):
EGL_STREAM_TYPE_NV:EGL_STREAM_CROSS_PARTITION_NV, EGL_STREAM_ENDPOINT_NV:EGL_STREAM_PRODUCER_NV,
EGL_SOCKET_HANDLE_NV: g_ClientID
3. Create the EGLStream surface (refer to nvgldemo_main.c):
eglCreateStreamProducerSurfaceKHR(
demoState.display,
demoState.config,
demoState.stream,
srfAttrs)
4. Create EGLContext and eglMakeCurrent the EGLStream surface created in step 2 (refer to nvgldemo_main.c).
5. Start to render the frames and eglSwapBuffers (refer to eglstreamcrosspart/producer.cpp).
To run cross-partition sample
On Linux, run EGLStream producer:
/samples/opengles2/eglcrosspart/egldevice/eglcrosspart -proctype producer -ip 12.0.0.1 -port 1111 &
Building a Cross-System EGLStream Pipeline
A cross-system EGLStream is one where the producer and consumer are in two different systems, and the systems are connected using ethernet/internet protocol in a multi-system environment.
To build a cross-system EGLStream pipeline, the consumer process creates the EGLStream on the consumer endpoint. The consumer then creates a socket and waits for the producer to connect to that socket.
The producer uses the IP address of the system on which the consumer is running and the specific port on which the consumer created the EGLStream to connect to it. After a successful connection, the producer creates an EGLStream using the same handle. After the EGLStream is created in both consumer and producer processes, the consumer and the producer are created and connected to the same EGLStream.
Only FIFO mode is supported in case of cross-system EGLStream.
Refer to the EGL_NV_stream_remote specification for more details.
The steps involved on the consumer side
1. Create the socket on the required port number. Set the created socketID to gsock.
2. Create the cross-partition EGLStream for the consumer end by setting the attributes.
EGL_STREAM_TYPE_NV: EGL_STREAM_CROSS_SYSTEM_NV, EGL_STREAM_ENDPOINT_NV: EGL_STREAM_CONSUMER_NV,
EGL_SOCKET_TYPE_NV: EGL_SOCKET_TYPE_INET_NV,
EGL_SOCKET_HANDLE_NV: gsock,
EGL_STREAM_FIFO_LENGTH_KHR: 4
eglstream = eglCreateStreamKHR(display, attrList);
 
Note:
Consult CreateConsumerEGLStream function in samples/nvmedia/eglstream/eglstrm_setup.c:
The cross-system eglstream is similar to cross-partition streams, except that:
a) EGL_STREAM_TYPE_NV must be set to EGL_STREAM_CROSS_SYSTEM_NV, instead of EGL_STREAM_CROSS_PARTITION_NV.
b) EGLStream FIFO mode must be enabled (-fifo command-line option in nvm_eglstream).
The steps involved on the producer side
1. Connect to the socket with the consumer IP address and the required port number. Set the created socketID to gsock (refer to nvgldemo_main.c).
2. Create the cross-partition EGLStream for the producer end by setting the attributes.
EGL_STREAM_TYPE_NV: EGL_STREAM_CROSS_SYSTEM_NV, EGL_STREAM_ENDPOINT_NV: EGL_STREAM_PRODUCER_NV,
EGL_SOCKET_TYPE_NV: EGL_SOCKET_TYPE_INET_NV,
EGL_SOCKET_HANDLE_NV: gsock,
EGL_STREAM_FIFO_LENGTH_KHR: 4
eglstream = eglCreateStreamKHR(display, attrList);
 
Note:
Consult CreateProducerEGLStream function in samples/nvmedia/eglstream/eglstrm_setup.c:
The cross-system eglstream is similar to cross-partition streams, except that:
a) EGL_STREAM_TYPE_NV must be set to EGL_STREAM_CROSS_SYSTEM_NV, instead of EGL_STREAM_CROSS_PARTITION_NV.
b) EGLStream FIFO mode must be enabled (-fifo command-line option in nvm_eglstream).
c) -consumervm <nvscic2c vm id> command line option must be used, where NvSciC2C VM ID represents the peer VM ID of the partition where the NvSciC2C server resides. Refer to the Enabling DMA transfers for cross-system EGLStream section for more details.
d) -ip <consumer ip address> must be specified, instead of the IP address of the partition.
Enabling DMA transfers for cross-system EGLStream
The prerequisite for enabling DMA transfers in cross-system EGLStream is to set up the NvSciC2C channels between the producer and consumer endpoints.
Follow the NVIDIA C2C Communication Over NVSCIIPC app note to set up NvSciC2C channels and to get the value of the NvSciC2C VM ID:
Once the NvSciC2C channels are setup successfully between the endpoints, the following two extra parameters must be passed during the EGLStream creation in both producer and consumer endpoints, in addition to the cross-system attributes described in The steps involved on the consumer side and The steps involved on the producer side sections above:
EGL_STREAM_DMA_NV: EGL_TRUE,
EGL_STREAM_DMA_SERVER_NV: <nvscic2c vm id>
Refer to the EGL_NV_stream_dma specification for more details.
The EGL_STREAM_DMA_SERVER_NV attribute accepts an integer value corresponding to the NvSciC2C VM ID of the partition where the NvSciC2C daemon is running.
For DMA transfers to be enabled, memory allocated by the producer endpoint must be visible to the NvSciC2C daemon and hence must be allocated from the memory carveout shared between the guest OS, where the producer is running, and the VM partition, where NvSciC2C daemon is running. This can be done by setting the value of the surface allocation attribute NVM_SURF_ATTR_PEER_VM_ID to NvSciC2C VM ID.
The EGLStream applications must run with root privilege to use the NvSciC2C channels.
Note:
Consult the usage of function NvMediaVideoSurfaceCreateNew in samples/nvmedia/eglstream/nvmvid_producer.c, where NVM_SURF_ATTR_PEER_VM_ID is used to allocate memory from required IVM carveout.