Evaluate Configuration#

Learn how to adjust your NeMo Safe Synthesizer evaluation configuration to adjust what information shows up in the Quality and Privacy Report.

Overview#

Generally the default parameters for the Evaluation Report are sufficient for providing you the relevant metrics based on your dataset, but there are a handful of parameter that you can adjust as needed for unique situations.

Parameters#

Parameter	Type	Description	Default
`enabled`	`bool`	Enable or disable evaluation entirely.	`True`
`sqs_report_columns`	`int`	Number of columns to include in statistical quality reports. If the dataset has fewer columns, it will use what’s available.	`250`
`sqs_report_rows`	`int`	Number of rows to include in statistical quality reports.	`5000`
`mia_enabled`	`bool`	Enable membership inference attack evaluation.	`True`
`aia_enabled`	`bool`	Enable attribute inference attack evaluation.	`True`
`mandatory_columns`	`int` or `None`	Columns that must be present in generated data.	`None`
`quasi_identifier_count`	`int`	Number of quasi-identifiers (i.e. other columns) sampled when running Attribute Inference Protection. Quasi-identifiers are the values that are considered “known” by the attacker when attempted to predict a given attribute.	`3`
`pii_replay_enabled`	`bool`	Enable PII Replay measurement.	`True`
`pii_replay_entities`	`list[str]` or `None`	List of entities for PII Replay. If not provided, default entities will be used.	`None`
`pii_replay_columns`	`list[str]` or `None`	List of columns for PII Replay. If not provided, only columns where an entity was detected will be used.	`None`

Example#

Set these under the evaluation section:

evaluation:
  enabled: true
  mia_enabled: true
  aia_enabled: true
  sqs_report_columns: 250
  sqs_report_rows: 5000
  mandatory_columns: null
  quasi_identifier_count: 3
  pii_replay_enabled: true
  pii_replay_entities: null
  pii_replay_columns: null