nemo_microservices.types.beta.safe_synthesizer.generate_parameters_param

Module Contents

Classes

GenerateParametersParam

API

class nemo_microservices.types.beta.safe_synthesizer.generate_parameters_param.GenerateParametersParam

Bases: typing_extensions.TypedDict

invalid_fraction_threshold: float

None

The fraction of invalid records that will stop generation after the patience limit is reached.

num_records: int

None

Number of records to generate.

patience: int

None

Number of consecutive generations where the invalid_fraction_threshold is reached before stopping generation.

repetition_penalty: float

None

The value used to control the likelihood of the model repeating the same token.

structured_generation_backend: typing_extensions.Literal[auto, xgrammar, guidance, outlines, lm-format-enforcer]

None

The backend used by VLLM when use_structured_generation=True.

Supported backends (from vllm) are ‘outlines’, ‘guidance’, ‘xgrammar’, ‘lm-format-enforcer’. ‘auto’ will allow vllm to choose the backend.

structured_generation_schema_method: typing_extensions.Literal[regex, json_schema]

None

The method used to generate the schema from your dataset and pass it to the generation backend. auto will usually default to ‘json_schema’. Use ‘regex to use our custom regex construction method, which tends to be more comprehensive than ‘json_schema’ at the cost of speed.

temperature: float

None

Sampling temperature.

top_p: float

None

Nucleus sampling probability.

use_structured_generation: bool

None

Use structured generation.