GXF benchmark#
GXF benchmarking can be accomplished with the help of benchmark extension. GXF benchmark extensions have components that allow GXF users to create benchmark applications that can give an insight into the performance outcomes of their GXF component/application designs and implementations.
Components of GXF benchmark extension#
Various benchmark componments that are used are described below with the help of a diagram
Benchmark Controller#
Benchmark controller connects every component in the benchmark application and orchestrates the benchmark flow. At the end of each benchmark iteration, the controller invokes a number of default and custom performance calculators for computing the performance outcome. A final performance report (in JSON format) is generated by the controller after all benchmark iterations are finished.
Benchmark Entity buffer#
Benchmark entity buffer receives all the messages provided from a producer and stores them in message buffers. After all the messages are buffered and the producer is terminated (which ensures that the producer will not induce any additional overhead when executing the main benchmark procedure), the benchmark controller will signal the beginning of the main benchmark procedure.
Benchmark Publisher#
Benchmark publisher is responsible for delivering the messages that are buffered in the Entity buffer component into the target graph. Timestamps are recorded when the messages are published for performance measurement.
Benchmark Sink#
Benchmark sink recieves the messages produced from the target graph. It records the timestamps when each message arrives for performance measurement. Benchmark sink optionally may have the handle of benchmark publisher to get the published timestamp or rely on the acqtime present in the timestamp of the received messages.
Target GXF graph#
Target GXF graph is a GXF subgraph to be benchmarked. It can contain as simple as one GXF component, or as complex as a GXF application graph
Stages of GXF benchmark#
Buffering#
Buffering is the first stage to load data to be used in benchmarking and buffer them in a entity buffer component. The separation of this stage from other stages is to ensure that the data loading overhead does not interfere with the performance measurement during benchmarking.
Benchmarking#
Benchmarking is the second stage which executes the main benchmarking procedure. Based on the benchmark’s configurations (a part of which are given as parameters in the benchmark controller component), a benchmark application can run multiple benchmark test cases with different settings (e.g., different publisher’s FPS). For each benchmark case, there can be a configurable number of test iterations for measuring the performance outcome in a statistical manner. When all test iterations for a benchmark case is done, the benchmark controller stores the concluded performance results and moves to testing the next benchmark test case.
Concluding#
Concluding is the last stage which occurs after all benchmark cases are tested. The benchmark controller exports all the performance results to a JSON file that can be processed by any other analysis tools further.
Sample graph to benchmark#
Below is an example of benchmarking a GXF graph. The GXF graph titled as graph_under_test consists of a Forward component which receives and transmits the messages
Example:
%YAML 1.2 --- ######################## # Benchmark Controller # ######################## name: benchmark_controller components: - name: allocator type: nvidia::gxf::UnboundedAllocator - name: report type: nvidia::gxf::File parameters: allocator: allocator file_path: /tmp/test_benchmark_one_source_one_sink_report.json file_mode: wb - name: benchmark_controller type: nvidia::gxf::benchmark::BenchmarkController parameters: data_source_async_scheduling_terms: [data_source/async_scheduling_term] data_source_boolean_scheduling_terms: [data_source/boolean_scheduling_term] benchmark_controller_target_time_scheduling_term: target_time_scheduling_term benchmark_controller_boolean_scheduling_term: boolean_scheduling_term benchmark_sinks: [benchmark_sink/sink] benchmark_publishers: [benchmark_publisher/benchmark_publisher] resource_profilers: [resource_profiler] title: "Test Benchmark Framework" exported_report: report benchmark_duration_ms: 500 benchmark_iterations: 3 entity_buffer_size: 10 - name: resource_profiler type: nvidia::gxf::benchmark::ResourceProfiler - name: boolean_scheduling_term type: nvidia::gxf::BooleanSchedulingTerm parameters: enable_tick: true - name: target_time_scheduling_term type: nvidia::gxf::TargetTimeSchedulingTerm parameters: clock: system/clock - type: nvidia::gxf::benchmark::test::BenchmarkReportChecker parameters: report: report --- ##################### # Dummy Data Source # ##################### name: data_source components: - name: transmitter type: nvidia::gxf::test::MockTransmitter - name: ping type: nvidia::gxf::PingTx parameters: signal: transmitter clock: system/clock - name: boolean_scheduling_term type: nvidia::gxf::BooleanSchedulingTerm parameters: enable_tick: true - name: async_scheduling_term type: nvidia::gxf::AsynchronousSchedulingTerm - type: nvidia::gxf::PeriodicSchedulingTerm parameters: recess_period: 10Hz --- #################### # Benchmark Buffer # #################### name: benchmark_buffer components: - name: receiver type: nvidia::gxf::test::MockReceiver - name: entity_buffer type: nvidia::gxf::benchmark::EntityBuffer parameters: receiver: receiver - type: nvidia::gxf::MessageAvailableSchedulingTerm parameters: receiver: receiver min_size: 1 --- ####################### # Benchmark Publisher # ####################### name: benchmark_publisher components: - name: async_scheduling_term type: nvidia::gxf::AsynchronousSchedulingTerm - type: nvidia::gxf::DownstreamReceptiveSchedulingTerm parameters: transmitter: transmitter min_size: 1 - name: benchmark_publisher type: nvidia::gxf::benchmark::BenchmarkPublisher parameters: entity_buffer: benchmark_buffer/entity_buffer transmitter: transmitter benchmark_publisher_async_scheduling_term: async_scheduling_term - name: transmitter type: nvidia::gxf::test::MockTransmitter --- #################### # Graph under Test # #################### name: graph_under_test components: - name: receiver type: nvidia::gxf::test::MockReceiver - name: transmitter type: nvidia::gxf::test::MockTransmitter - type: nvidia::gxf::Forward parameters: in: receiver out: transmitter - type: nvidia::gxf::MessageAvailableSchedulingTerm parameters: receiver: receiver min_size: 1 - type: nvidia::gxf::DownstreamReceptiveSchedulingTerm parameters: transmitter: transmitter min_size: 1 --- ################## # Benchmark Sink # ################## name: benchmark_sink components: - name: receiver type: nvidia::gxf::test::MockReceiver - name: sink type: nvidia::gxf::benchmark::BenchmarkSink parameters: receiver: receiver benchmark_publisher: benchmark_publisher/benchmark_publisher performance_calculators: [basic_metrics] - type: nvidia::gxf::MessageAvailableSchedulingTerm parameters: receiver: receiver min_size: 1 - name: basic_metrics type: nvidia::gxf::benchmark::BasicMetricsCalculator --- ############### # Connections # ############### components: - type: nvidia::gxf::Connection parameters: source: data_source/transmitter target: benchmark_buffer/receiver - type: nvidia::gxf::Connection parameters: source: benchmark_publisher/transmitter target: graph_under_test/receiver - type: nvidia::gxf::Connection parameters: source: graph_under_test/transmitter target: benchmark_sink/receiver --- ########## # System # ########## name: system components: - name: clock type: nvidia::gxf::RealtimeClock - type: nvidia::gxf::JobStatistics parameters: clock: clock - type: nvidia::gxf::EventBasedScheduler parameters: clock: clock worker_thread_number: 4
Results#
Benchmark controller creates a json file as explained above. Below is a sample of json file which consists of various statistis. The statistics is for a graph with two sources and two sinks.
{
"[Sink 1] # of Frames Received": 20270.0,
"[Sink 1] # of Frames Sent": 20270.0,
"[Sink 1] # of Missed Frames": 0.0,
"[Sink 1] Delta between First & Last Received Frames (ms)": 624.549761,
"[Sink 1] Delta between First & Last Sent Frames (ms)": 499.904755,
"[Sink 1] First Sent to First Received Latency (ms)": 0.200702,
"[Sink 1] Last Sent to Last Received Latency (ms)": 125.057111,
"[Sink 1] Max. End-to-End Latency (ms)": 127.770703,
"[Sink 1] Max. Frame-to-Frame Jitter (ms)": 0.214416,
"[Sink 1] Mean End-to-End Latency (ms)": 75.224532,
"[Sink 1] Mean Frame Rate (fps)": 32550.581433937827,
"[Sink 1] Mean Frame-to-Frame Jitter (ms)": 0.006864,
"[Sink 1] Mean Publisher Frame Rate (fps)": 40545.72355487997,
"[Sink 1] Min. End-to-End Latency (ms)": 0.129046,
"[Sink 1] Min. Frame-to-Frame Jitter (ms)": 0.0,
"[Sink 1] SD. End-to-End Latency (ms)": 40.329108290039265,
"[Sink 1] SD. Frame-to-Frame Jitter (ms)": 0.0062623999229486855,
"[Sink 2] # of Frames Received": 20270.0,
"[Sink 2] # of Frames Sent": 20270.0,
"[Sink 2] # of Missed Frames": 0.0,
"[Sink 2] Delta between First & Last Received Frames (ms)": 624.544266,
"[Sink 2] Delta between First & Last Sent Frames (ms)": 499.904755,
"[Sink 2] First Sent to First Received Latency (ms)": 0.211339,
"[Sink 2] Last Sent to Last Received Latency (ms)": 125.062253,
"[Sink 2] Max. End-to-End Latency (ms)": 127.772703,
"[Sink 2] Max. Frame-to-Frame Jitter (ms)": 0.196638,
"[Sink 2] Mean End-to-End Latency (ms)": 75.227042,
"[Sink 2] Mean Frame Rate (fps)": 32551.20225100854,
"[Sink 2] Mean Frame-to-Frame Jitter (ms)": 0.007822,
"[Sink 2] Mean Publisher Frame Rate (fps)": 40545.72355487997,
"[Sink 2] Min. End-to-End Latency (ms)": 0.133161,
"[Sink 2] Min. Frame-to-Frame Jitter (ms)": 0.0,
"[Sink 2] SD. End-to-End Latency (ms)": 40.32882660754853,
"[Sink 2] SD. Frame-to-Frame Jitter (ms)": 0.007074244521465658
}