.. _SD.GraphicsProgramming.Eglstream:

.. include:: /content/swdocs.rsts

.. spelling::

EGLStream
!!!!!!!!!

EGLStream is a mechanism that efficiently transfers a sequence of image
frames from one API to another, e.g., from OpenGL\ |reg| to |NVIDIA(r)| CUDA\ |reg|.

In EGLStream architecture a producer and a consumer are attached to each
end of a stream object. A **producer** adds image frames to the stream.
A **consumer** retrieves image frames from the stream.

EGLStream Producers
@@@@@@@@@@@@@@@@@@@

An EGLStream producer posts EGL\ |tm| image frames to an EGLStream. The
producer is responsible for inserting each image frame into the
EGLStream at the correct time so that the consumer can display the image
frame for the appropriate period of time.

There are two types of producers:

-  **CUDA producers** post a CUDA array or CUDA pointer to the EGLStream
   as EGL image frames.

-  **OpenGL producers** post graphic surfaces to the EGLStream as EGL
   image frames.

EGLStream Consumers
@@@@@@@@@@@@@@@@@@@

An EGLStream consumer retrieves EGL image frames from the EGLStream. The
consumer is responsible for noticing that an image frame is available
and displaying it (or otherwise consuming it). The consumer is
responsible for the latency of the acquired frame (the time elapsed
between image frame retrieval and display).

There are three types of consumers:

-  **CUDA consumers** retrieve EGL image frames and deliver the frame
   information as a CUDA array or a CUDA pointer. The CUDA frames can
   then be processed in the CUDA domain.

-  **OpenGL consumers** retrieve EGL image frames that can then be bound
   as OpenGL textures for graphics rendering.

-  **EGLOutput consumers** (EGLDevice window system only) retrieve EGL
   image frames and render them directly to EGLOutput. An EGLOutput
   consumer is applicable when EGLOutput is used on the EGLDevice window
   system.

EGLStream Operation Modes
@@@@@@@@@@@@@@@@@@@@@@@@@

EGLStream operates in one of two modes: mailbox mode and FIFO mode.

Mailbox Mode
############

In **mailbox mode**, EGLStream conceptually operates as a mailbox. When
the producer has a new image frame it empties the mailbox (discards the
old contents) and inserts the new image frame into the mailbox. The
consumer retrieves the image frame from the mailbox and examines it.
When the consumer is finished examining the image frame, it either puts
the frame back in the mailbox (if the mailbox is empty) or discards it
(if the mailbox is not empty).

Timing is mainly controlled by the producer. The consumer operates with
a fixed latency that it indicates to the producer through the EGLStream
attribute ``EGL_CONSUMER_LATENCY_USEC_KHR``. The consumer must notice when a
new image frame is available in the EGLStream, retrieve it, and display
it in the time indicated by ``EGL_CONSUMER_LATENCY_USEC_KHR``. The producer
controls when the image frame is displayed by inserting it into the
stream at time *t*\ |nbsp|\ −\ |nbsp|\ ``EGL_CONSUMER_LATENCY_USEC_KHR``, where *t* is the
time when the image frame is to be displayed.

FIFO Mode
#########

In **FIFO mode** images are not discarded. When a producer adds image
frames to the stream they go in a FIFO queue for subsequent retrieval by
the consumer. EGLStream sets the queue size when it creates the queue.
If the producer attempts to insert a frame and the queue is full, the
producer blocks until there is room in the queue. When the consumer
retrieves an image frame from the EGLStream, it gets the image frame
that immediately follows the image frame last retrieved (unless no such
frame has been inserted yet, in which case it retrieves the same image
frame as last time).

Timing of an EGLStream in FIFO mode is the responsibility of the
consumer. Each image frame in the queue has a timestamp set by the
producer. The consumer uses this timestamp to determine when the image
frame is intended to be displayed.

EGLStream Pipeline
@@@@@@@@@@@@@@@@@@

EGL provides functions to create and destroy EGLStreams, to query and
set attributes of EGLStreams, and to connect EGLStreams to producers and
consumers.

An EGLStream cannot be used until it is connected to a consumer and then
to a producer. It must be connected to the consumer first.

An EGLStream may be connected to only one producer and one consumer.
Once it is connected to a producer or a consumer, it remains connected
to that entity until it is destroyed.

To build a simple EGLStream pipeline
####################################

1. Create the EGLDisplay object which the EGLStream is to bind to.

2. Call ``eglInitialize()`` to initialize EGL on the display.

   To use the OpenGL consumer or EGLOutput consumer, you must initialize the rendering window before you create the EGLStream pipeline.

   To use the CUDA producer-consumer, window system initialization is not required.

3. Create an EGLStream.

   This example of creating the EGLStream is from ``eglstreamcube.c``::

      client->stream = eglCreateStreamKHR(demoState.display, streamAttr);
      if (client->stream == EGL_NO_STREAM_KHR) {
          NvGlDemoLog("Couldn't create EGL stream.\n");
          goto fail;
      }

   Set ``streamAttr`` according to whether you are using mailbox mode or FIFO mode. For FIFO mode, the attribute ``EGL_STREAM_FIFO_LENGTH_KHR`` is initialized::

      if (demoOptions.nFifo > 0) {
          streamAttr[numAttrs++] = EGL_STREAM_FIFO_LENGTH_KHR;
          streamAttr[numAttrs++] = demoOptions.nFifo;
      }

4. Create a consumer of the appropriate type and connect it to the EGLStream.

   This example of connecting an OpenGL consumer to the EGLStream is from ``eglstreamcube.c``::

      glBindTexture(GL_TEXTURE_EXTERNAL_OES, texture);
      if (!eglStreamConsumerGLTextureExternalKHR(demoState.display, client->stream)) {
          NvGlDemoLog("Couldn't bind texture.\n");
          goto fail;
      }

   Remember that once an EGLStream is connected to a consumer, it remains connected to that consumer until the EGLStream is destroyed.

5. Create a producer of the appropriate type and connect it to the EGLStream.

   This example of connecting an OpenGL producer to the EGLStream is from ``nvgldemo_main.c``::

      eglCreateStreamProducerSurfaceKHR(demoState.display,
                                        demoState.config,
                                        demoState.stream,
                                        srfAttrs);

   Remember that once an EGLStream is connected to a producer, it remains connected to that producer until the EGLStream is destroyed.

6. The producer posts the image frames (the stream) to the consumer in order, depending on the type of stream (FIFO or mailbox).

   In the OpenGL producer case, ``eglSwapBuffers()`` posts the buffer to the consumer.

7. The consumer acquires the image frame posted by the producer, uses
   it, and then releases the frame back to the stream. Methods for
   acquiring frames from a stream and releasing them back to a stream
   are dependent on the type of consumer.

   This example of acquiring and releasing the frame in the OpenGL consumer case is from ``eglstreamcube.c``::

      eglStreamConsumerAcquireKHR(demoState.display, clientList[i].stream);
      eglStreamConsumerReleaseKHR(demoState.display, client->stream);

   In the OpenGL consumer case, if the application calls ``eglStreamConsumerAcquireKHR()`` twice on the same EGLStream without an intervening call to ``eglStreamConsumerReleaseKHR()``, then ``eglStreamConsumerReleaseKHR()`` is implicitly called at the start of ``eglStreamConsumerAcquireKHR()``.

To destroy the EGLStream pipeline
#################################

1. Destroy the producer.
2. Destroy the consumer.
3. Destroy the EGLStream.
4. Destroy the window system resources.

The application must destroy resources in this order to produce a correct result.

EGLStream State
###############

At any time after an EGLStream is created, it is in one of these states:

-  ``EGL_STREAM_STATE_CREATED_KHR``: Created but not yet connected to a producer or a consumer.
-  ``EGL_STREAM_STATE_CONNECTING_KHR``: Connected to a consumer but not yet connected to a producer.
-  ``EGL_STREAM_STATE_EMPTY_KHR``: Connected to a consumer and a producer, but the producer has not yet posted any image frames.
-  ``EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR``: The producer has posted at least one image frame that the consumer has not yet acquired.
-  ``EGL_STREAM_STATE_OLD_FRAME_AVAILABLE_KHR``: The producer has posted at least one image frame, and the consumer has acquired the most recently posted image frame.
-  ``EGL_STREAM_STATE_DISCONNECTED_KHR``: The producer, the consumer, or both are no longer connected to the EGLStream, i.e., they have been destroyed. Once the EGLStream is in this state it remains in this state until it is destroyed. In this state the only valid operations on the EGStream are ``eglQueryStreamKHR()`` and ``eglDestroyStreamKHR()``.

You query the EGLStream’s state like this::

   EGLBoolean eglQueryStreamKHR(
              EGLDisplay dpy,
              EGLStreamKHR stream,
              EGLenum attribute,
              EGLuint64KHR *value);

These state transitions can occur:

-  ``EGL_STREAM_STATE_CREATED_KHR``: The initial state, of an EGLStream that has just been created.
-  ``EGL_STREAM_STATE_CREATED_KHR`` to ``EGL_STREAM_STATE_CONNECTING_KHR``: Occurs when a consumer is connected to the EGLStream.
-  ``EGL_STREAM_STATE_CONNECTING_KHR`` to ``EGL_STREAM_STATE_EMPTY_KHR``: Occurs when a producer is connected to the EGLStream.
-  ``EGL_STREAM_STATE_EMPTY_KHR`` to ``EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR``: Occurs the first time the producer inserts an EGL image frame.
-  ``EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR`` to ``EGL_STREAM_STATE_OLD_FRAME_AVAILABLE_KHR``: Occurs when the consumer begins acquiring a newly posted EGL image frame.
-  ``EGL_STREAM_STATE_OLD_FRAME_AVAILABLE_KHR`` to ``EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR``: Occurs when the producer posts a new EGL image frame.
-  Any state to ``EGL_STREAM_STATE_DISCONNECTED_KHR``: Occurs when the producer or consumer is destroyed.

Building a Cross-Process EGLStream Pipeline
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

A cross-process EGLStream is one whose producer and consumer are in different processes.

To build a cross-process EGLStream pipeline
###########################################

1. The consumer process creates an EGLStream.

2. The consumer process gets the file descriptor for the EGLStream and
   sends it through the socket to the producer process.

3. In the producer process, the producer receives the file descriptor
   through the socket and creates a corresponding EGLStream from the
   file descriptor. After the EGLStreams are created in both consumer
   and producer processes, the consumer and the producer are created
   and are connected to the EGLStreams, and through them, to each other.

Cross-Process EGLStream Example
###############################

Jetson Linux provides a pair of sample applications that constitute an example of
communication through a cross-process EGLStream pipeline:

-  ``samples/opengles2/eglstreamcube`` is the consumer.
-  ``samples/opengles2/bubble`` is the producer.

You can demonstrate communication through a cross-process EGLStream
pipeline by starting the consumer, then the producer.

The consumer application performs these operations:

1. Creates the stream and get the file descriptor of the stream (see ``eglstreamcube.c``)::

      client->stream = eglCreateStreamKHR(demoState.display, streamAttr);
      client->fd = eglGetStreamFileDescriptorKHR(demoState.display, client->stream);

2. Shares the file descriptor with the producer through the socket.

3. Binds the consumer end of the EGLStream with the OPENGL texture (see ``eglstreamcube.c``)::

      glBindTexture(GL_TEXTURE_EXTERNAL_OES, texture);
      eglStreamConsumerGLTextureExternalKHR(demoState.display, client->stream);

4. Latches the most recent image frame to the texture with ``eglStreamConsumerAcquireKHR()``::

      eglStreamConsumerAcquireKHR(demoState.display,stream);

5. Renders the texture as one face of the cube.

The producer application performs these operations:

1. Receives the stream file descriptor through the socket
   (specified by a ``-eglstreamsocket`` option) and creates the stream (see ``nvgldemo_main.c``)::

      eglCreateStreamFromFileDescriptorKHR(demoState.display, fd);

2. Creates the EGLStream surface (see ``nvgldemo_main.c``)::

      eglCreateStreamProducerSurfaceKHR(
                          demoState.display,
                          demoState.config,
                          demoState.stream,
                          srfAttrs);

3. Creates an instance ``EGLContext`` and binds it to the thread by calling ``eglMakeCurrent()`` on the EGLStream surface created in step 2.

4. Renders the frames and calls ``eglSwapBuffers()``.

To run the cross-process EGLStream pipeline example
###################################################

1. Enter this command to start the consumer application::

      $ ./eglstreamcube -dispno 1 -layer 1 -windowoffset 1000 0 -socket /tmp/test &

2. Enter this command to start the producer application::

      $ ./bubble -eglstreamsocket /tmp/test &

The ``bubble`` application content appears on one face of the cube.