nat.eval.runners.red_teaming_runner.report_utils

Report utilities for red teaming evaluation results.

Attributes

logger

Functions

_validate_columns(→ None)	Validate that required columns exist in the DataFrame.
plot_score_boxplot(, box_color, box_fill_opacity, ...)	Create a box plot with data points overlaid inside the boxes.
plot_success_rate_bar(→ plotly.graph_objects.Figure)	Create a bar chart showing attack success rate (percentage of scores above threshold).
generate_standard_plots(→ list[tuple[str, str, ...)	Generate standard plots for red teaming results, grouped by category.
_get_risk_color(→ str)	Get a color that transitions from low to high risk based on value.
_render_summary_html(→ str)	Render the summary section as HTML.
save_combined_html(→ pathlib.Path)	Save all plots in a single interactive HTML document.
generate_and_save_report(→ pathlib.Path | None)	Generate and save all plots from flat results.

Module Contents

logger

_validate_columns(

df: pandas.DataFrame,

required_columns: list[str],

context: str = '',

) → None

Validate that required columns exist in the DataFrame.

Args:

df: DataFrame to validate. required_columns: List of column names that must exist. context: Optional context string for error message (e.g., function name).

Raises:

ValueError: If any required column is missing.

plot_score_boxplot(

df: pandas.DataFrame,

x: str,

y: str = 'score',

title: str | None = None,

x_label: str | None = None,

y_label: str = 'Risk Score',

y_range: tuple[float, float] = (-0.05, 1.05),

box_color: str = 'rgb(55, 126, 184)',

box_fill_opacity: float = 0.3,

point_color: str = 'rgba(255, 50, 0, 0.5)',

point_size: int = 6,

jitter: float = 0.3,

) → plotly.graph_objects.Figure

Create a box plot with data points overlaid inside the boxes.

Args:

df: DataFrame containing the data. x: Column name for x-axis (grouping variable). y: Column name for y-axis (score values). title: Plot title. Defaults to “Score Distribution by {x}”. x_label: X-axis label. Defaults to the column name. y_label: Y-axis label. y_range: Tuple of (min, max) for y-axis range. box_color: RGB color for box outline. box_fill_opacity: Opacity for box fill (0-1). point_color: RGBA color for data points. point_size: Size of data points. jitter: Horizontal jitter for points (0-1).

Returns:

The Plotly Figure object.

Raises:

ValueError: If required columns are missing from the DataFrame.

plot_success_rate_bar(

df: pandas.DataFrame,

x: str,

y: str = 'score',

threshold: float = 0.5,

title: str | None = None,

x_label: str | None = None,

y_label: str = 'Attack Success Rate (%)',

bar_color: str = 'rgb(55, 126, 184)',

) → plotly.graph_objects.Figure

Create a bar chart showing attack success rate (percentage of scores above threshold).

Args:

df: DataFrame containing the data. x: Column name for x-axis (grouping variable). y: Column name for the score values. threshold: Score threshold to consider as “success” (default 0.5). title: Plot title. Defaults to “Attack Success Rate by {x}”. x_label: X-axis label. Defaults to the column name. y_label: Y-axis label. bar_color: Color for the bars.

Returns:

The Plotly Figure object.

Raises:

ValueError: If required columns are missing from the DataFrame.

generate_standard_plots(

df: pandas.DataFrame,

) → list[tuple[str, str, plotly.graph_objects.Figure | None]]

Generate standard plots for red teaming results, grouped by category.

Args:

df: DataFrame with columns: scenario_id, condition_name, tags, scenario_group, score.

Returns:

List of tuples (filename, title, figure) for each plot. Section headers have figure=None and are rendered as section titles.

Raises:

ValueError: If required columns are missing from the DataFrame.

_get_risk_color(value: float, max_value: float = 1.0) → str

Get a color that transitions from low to high risk based on value.

The color transitions and opacity increases as risk increases: - Opacity: 0.3 (at 0) -> 1.0 (at max) - Color: muted -> intense red

Args:

value: The risk value (0 to max_value). max_value: The maximum value (1.0 for scores, 100.0 for percentages).

Returns:

RGBA color string.

_render_summary_html(summary: dict[str, Any] | None) → str

Render the summary section as HTML.

Args:

summary: The summary dictionary from _compute_result_summary.

Returns:

HTML string for the summary section.

save_combined_html(

plots: list[tuple[str, str, plotly.graph_objects.Figure | None]],

output_path: pathlib.Path,

page_title: str = 'Red Teaming Evaluation Results',

summary: dict[str, Any] | None = None,

) → pathlib.Path

Save all plots in a single interactive HTML document.

Args:

plots: List of (filename, title, figure) tuples. output_path: Path for the combined HTML file. page_title: Title for the HTML page. summary: Optional summary dictionary to display at the top of the report.

Returns:

Path to the saved HTML file.

generate_and_save_report(

flat_results: list[dict[str, Any]] | pandas.DataFrame,

output_dir: pathlib.Path,

summary: dict[str, Any] | None = None,

) → pathlib.Path | None

Generate and save all plots from flat results.

This is the main entry point for plotting. It: 1. Converts flat results to a DataFrame 2. Generates standard plots (by scenario, group, condition, tags) 3. Saves a combined HTML report with all plots and summary

Args:

flat_results: List of flat result dictionaries from _build_flat_results. output_dir: Base output directory. Plots are saved in a ‘graphs’ subfolder. summary: Optional summary dictionary to display at the top of the report.

Returns:

Path to the combined HTML report.