aiq.profiler.utils#

Attributes#

_FRAMEWORK_REGEX_MAP

logger

Functions#

detect_llm_frameworks_in_build_fn(...)

Analyze a function's source (the build_fn) to see which LLM frameworks it uses. Also recurses

create_standardized_dataframe(→ pandas.DataFrame)

Merge usage stats for all requests into one DataFrame, each row representing a usage_stats entry.

Module Contents#

_FRAMEWORK_REGEX_MAP#
logger#
detect_llm_frameworks_in_build_fn(
registration: aiq.cli.type_registry.RegisteredFunctionInfo,
) → list[aiq.builder.framework_enum.LLMFrameworkEnum]#

Analyze a function’s source (the build_fn) to see which LLM frameworks it uses. Also recurses into any additional Python functions that the build_fn calls while passing builder, so that references to LLMFrameworkEnum in those helper calls are also detected.

  1. If registration.framework_wrappers is non-empty, we return that first. (We do convert them to LLMFrameworkEnum if possible.)

  2. Otherwise, we attempt to:

    • Get the build_fn’s source via inspect.getsource(...)

    • Parse it for references to LLMFrameworkEnum

    • Find any function calls that include the word “builder” in the arguments

      • Recursively parse those functions’ source code for frameworks

  3. If we cannot parse the source at all (e.g. OSError), we return a list of all frameworks.

create_standardized_dataframe(
requests_data: list[list[aiq.data_models.intermediate_step.IntermediateStep]],
) → pandas.DataFrame#

Merge usage stats for all requests into one DataFrame, each row representing a usage_stats entry. - Include a column ‘example_number’ to mark which request it originated from.