Operators

Testing

View as Markdown

Overview

Holoscan provides a test harness interface for testing Holoscan operators with:

  • Input/Output Port Management: Easy setup of test data and validation
  • Condition Support: Add execution conditions like CountCondition, PeriodicCondition
  • Validation Framework: Built-in validators for exact equality, floating-point comparison, and custom validation
  • Fluent API: Chainable method calls for clean, readable test setup

Concepts

Test Harness

The OperatorTestHarness is a specialized Holoscan application designed to test individual operators in isolation. It automates the setup of a test pipeline by creating source operators to provide input data and sink operators to collect and validate output data.

Main Components:

  1. OperatorTestHarness<OperatorType, Args...>: The main test harness class that orchestrates the test

    • Creates and manages source operators for each input port
    • Creates and manages sink operators for each output port
    • Connects all operators in the test pipeline
    • Provides a fluent API for configuration

  2. TestHarnessSourceOp<T>: A specialized operator that emits predetermined test data

    • Emits one value per compute cycle from a provided vector
    • Automatically manages iteration through test data
    • Used internally by the test harness for each input port

  3. TestHarnessSinkOp<T>: A specialized operator that collects and validates outputs

    • Receives data from the operator under test
    • Applies validation functions to each received value
    • Tracks received data count for verification
    • Used internally by the test harness for each output port

  4. OperatorTestBase:

    • A base test fixture class designed for use with Google Test (gtest).
    • Just a skeleton fixture, does not currently extend SetUp or TearDown methods.

How It Works:

When you create a test harness with create_operator_test<YourOp>():

  1. The harness creates a Holoscan application with your operator in the middle
  2. For each add_input_port() call, it creates a TestHarnessSourceOp that feeds test data
  3. For each add_output_port() call, it creates a TestHarnessSinkOp that collects and validates output
  4. When you call run_test(), the entire pipeline executes and validations are performed automatically

The test harness ensures that:

  • All input ports receive the same number of data elements
  • Data flows correctly from sources through your operator to sinks
  • Conditions (like CountCondition) control execution properly
  • Validation failures are reported via Google Test assertions

Validator Functions

Validator functions are callable objects that verify operator outputs during test execution. They follow the signature void(const T&) where T is the output data type. Validators are called automatically by TestHarnessSinkOp for each value received from the operator under test.

Built-in Validators:

Holoscan provides three types of built-in validators:

  1. create_exact_equality_validator<T>(expected_values)

    • Compares each output against expected values using operator==
    • Best for: integers, strings, and other types with well-defined equality
    • Automatically tracks which output index is being validated
    1std::vector<int> expected = {2, 4, 6};
    2auto validator = create_exact_equality_validator(expected);

  2. create_float_equality_validator<T>(expected_values, tolerance = T{})

    • Compares floating-point values with approximate equality
    • Uses Google Test’s EXPECT_FLOAT_EQ (default) or EXPECT_NEAR (with tolerance)
    • Best for: float, double, and other floating-point types
    1std::vector<float> expected = {1.5f, 2.7f, 3.1f};
    2auto validator = create_float_equality_validator(expected, 0.01f);

  3. create_transform_equality_validator<InputT, OutputT>(expected_values, transform_func)

    • Applies a transformation before comparing
    • Best for: complex types where you want to validate a specific property
    1auto validator = create_transform_equality_validator&lt;MyStruct, int&gt;(
    2  {10, 20, 30},
    3  [](const MyStruct& s) { return s.field; }
    4);

Using Multiple Validators:

You can apply multiple validators to a single output port using the validators<T>() helper:

1test->add_output_port<float>("output", validators<float>(
2  create_float_equality_validator(expected_values),
3  custom_range_validator,
4  custom_format_validator
5));

Custom Validators:

Create custom validators by providing any callable that matches void(const T&):

1auto custom_validator = [](const int& value) {
2  EXPECT_GT(value, 0) << "Value must be positive";
3  EXPECT_LT(value, 100) << "Value must be less than 100";
4};
5 
6test->add_output_port<int>("output", validators<int>(custom_validator));

Custom validators can perform any verification logic and use any Google Test assertion macros (EXPECT_*, ASSERT_*). Validation failures will cause the test to fail with descriptive error messages.

Basic Test Structure

Suppose we have an operator (DoublerOp) that has a single input and single output port, both int. The operator simply doubles the input value and emits that.

There are two ways to set up tests:

Approach 1: Fluent API

Fluent is a design pattern that enables code to be written in a way that flows naturally, often by chaining method calls together. This style improves readability and expressiveness, making it easier to set up complex objects or configurations in a concise and intuitive manner. The term “fluent API” refers to this general programming approach and is not specific to Holoscan.

1#include <holoscan/test/test_harness.h>
2#include <holoscan/test/validation_functions.h>
3 
4using namespace holoscan::test {
5 
6TEST_F(OperatorTestBase, BasicFluentTest) {
7  // Test data
8  std::vector<int> input_data = {1, 2, 3};
9  std::vector<int> expected_output = {2, 4, 6};
10 
11  // Create and configure test with fluent API - everything in one chain
12  auto test_harness = create_operator_test<DoublerOp>()
13    ->add_input_port<int>("input", input_data)
14    ->add_output_port<int>("output", holoscan::test::validators<int>(
15      create_exact_equality_validator(expected_output)))
16    ->add_condition<holoscan::CountCondition>("count", input_data.size());
17 
18  test_harness->run_test();
19 
20  // Additional verification if needed
21  auto sink = test_harness->get_sink<int>("output");
22  EXPECT_EQ(sink->get_received_count(), expected_output.size());
23}
24 
25}  // namespace holoscan::test

Approach 2: Step-by-Step Setup

1#include <holoscan/test/test_harness.h>
2#include <holoscan/test/validation_functions.h>
3 
4using namespace holoscan::test {
5 
6TEST_F(OperatorTestBase, BasicStepByStepTest) {
7  // Test data
8  std::vector<int> input_data = {1, 2, 3};
9  std::vector<int> expected_output = {2, 4, 6};
10 
11  // Create test harness first
12  auto test_harness = create_operator_test<DoublerOp>();
13 
14  // Add components step by step (allows for conditional logic)
15  test_harness->add_input_port<int>("input", input_data);
16  test_harness->add_output_port<int>("output", holoscan::test::validators<int>(
17    create_exact_equality_validator(expected_output)));
18  test_harness->add_condition<holoscan::CountCondition>("count", input_data.size());
19 
20  test_harness->run_test();
21 
22  // Additional verification
23  auto sink = test_harness->get_sink<int>("output");
24  EXPECT_EQ(sink->get_received_count(), expected_output.size());
25}
26 
27}  // namespace holoscan::test

API Reference

Creating Test Harness

1// Without parameters
2auto test = create_operator_test<DoublerOp>();
3 
4// With parameters
5auto test = create_operator_test<DoublerOp>(
6  holoscan::Arg("param1") = value1,
7  holoscan::Arg("param2") = value2
8);

Adding Input Ports

The values that are passed to the operator’s input ports are defined and added via the add_input_port function. This function takes a std::vector<DataType> object, the elements of which will be passed to the operator one-by-one.

1test->add_input_port<DataType>("port_name", test_data_vector);

Adding Output Ports

The output ports of the operator are added to the test harness using the add_output_port function. This function can be used in two ways: without a validator, to simply collect the output data, or with a validator function to automatically check that the output matches expected results.

1// Without validation
2test->add_output_port<DataType>("port_name");
3 
4// With validation
5test->add_output_port<DataType>("port_name",
6  holoscan::test::validators<DataType>(validator_function));

Adding Conditions

1test->add_condition<holoscan::CountCondition>("name", count);
2test->add_condition<holoscan::PeriodicCondition>("name",
3  holoscan::Arg("recess_period") = std::string("100ms"));

Running Tests

1test->run_test();

Testing Patterns

Source Operators

Operators with output ports, but no input ports.

1auto test = create_operator_test<SourceOp>(params)
2  ->add_condition<CountCondition>("count", iterations)
3  ->add_output_port<T>("output", validators);
4 
5test->run_test();
6 
7// Verify internal state (optional)
8auto src_op = test->get_operator_under_test();
9// Check src_op internal state...

Sink Operators

Operators with input ports, but no input ports.

1auto test = create_operator_test<SinkOp>()
2  ->add_input_port<T>("input", test_data);
3 
4test->run_test();
5 
6// Verify internal state (optional)
7auto sink_op = test->get_operator_under_test();
8// Check sink_op internal state...

Transform Operators

Operators with both input and output ports

1auto test = create_operator_test<TransformOp>(params)
2  ->add_input_port<InputT>("input", input_data)
3  ->add_output_port<OutputT>("output", output_validators)
4  ->add_condition<CountCondition>("count", data_size);