morpheus_dfp.modules.dfp_inference_pipe

Functions

dfp_inference_pipe(builder) This module function consolidates multiple dfp pipeline modules relevant to the inference process into a single module.

dfp_inference_pipe(builder)[source]

This module function consolidates multiple dfp pipeline modules relevant to the inference process into a single module.

Parameters

Notes

Configurable parameters:

batching_options (dict): Options for batching the data; Example: See Below
cache_dir (str): Directory to cache the rolling window data; Example: “/path/to/cache/dir”; Default: ./.cache
detection_criteria (dict): Criteria for filtering detections; Example: See Below
fallback_username (str): User ID to use if user ID not found; Example: “generic_user”; Default: “generic_user”
inference_options (dict): Options for the inference module; Example: See Below
model_name_formatter (str): Format string for the model name; Example: “model_{timestamp}”; Default: [Required]
timestamp_column_name (str): Name of the timestamp column in the input data; Example: “timestamp”; Default: “timestamp”
stream_aggregation_options (dict): Options for aggregating the data by stream; Example: See Below
user_splitting_options (dict): Options for splitting the data by user; Example: See Below
write_to_file_options (dict): Options for writing the detections to a file; Example: See Below
monitor_options (dict): Options for monitoring throughput; Example: See Below

batching_options:

end_time (datetime/str): End time of the time window; Example: “2023-03-14T23:59:59”; Default: None
iso_date_regex_pattern (str): Regex pattern for ISO date matching; Example: “d{4}-d{2}-d{2}Td{2}:d{2}:d{2}”; Default: <iso_date_regex_pattern>
parser_kwargs (dict): Additional arguments for the parser; Example: {}; Default: {}
period (str): Time period for grouping files; Example: “1d”; Default: “1d”
sampling_rate_s (int): Sampling rate in seconds; Example: 0; Default: None
start_time (datetime/str): Start time of the time window; Example: “2023-03-01T00:00:00”; Default: None

detection_criteria:

copy (bool): Whether to copy the rows or slice them; Example: true; Default: true
field_name (str): Name of the field to filter on; Example: probs; Default: probs
filter_source (str): Source of the filter field; Example: AUTO; Default: AUTO
schema (dict): Schema configuration; See Below; Default: -
threshold (float): Threshold value to filter on; Example: 0.5; Default: 0.5

schema:

encoding (str): Encoding; Example: “latin1”; Default: “latin1”
input_message_type (str): Pickled message type; Example: pickle_message_type; Default: [Required]
schema_str (str): Schema string; Example: “string”; Default: [Required]

inference_options:

model_name_formatter (str): Formatter for model names; Example: “user_{username}_model”; Default: [Required]
fallback_username (str): Fallback user to use if no model is found for a user; Example: “generic_user”; Default: generic_user
timestamp_column_name (str): Name of the timestamp column; Example: “timestamp”; Default: timestamp

stream_aggregation_options:

cache_mode (str): Mode for managing user cache. Setting to batch flushes cache once trigger conditions are met. Otherwise, continue to aggregate user’s history.; Example: ‘batch’; Default: ‘batch’
trigger_on_min_history (int): Minimum history to trigger a new training event; Example: 1; Default: 1
trigger_on_min_increment (int): Minimum increment from the last trained to new training event; Example: 0; Default: 0
timestamp_column_name (str): Name of the column containing timestamps; Example: ‘timestamp’; Default: ‘timestamp’
aggregation_span (str): Lookback timespan for training data in a new training event; Example: ‘60d’; Default: ‘60d’
cache_to_disk (bool): Whether to cache streaming data to disk; Example: false; Default: false
cache_dir (str): Directory to use for caching streaming data; Example: ‘./.cache’; Default: ‘./.cache’

user_splitting_options:

fallback_username (str): The user ID to use if the user ID is not found; Example: “generic_user”; Default: ‘generic_user’
include_generic (bool): Whether to include a generic user ID in the output; Example: false; Default: False
include_individual (bool): Whether to include individual user IDs in the output; Example: true; Default: False
only_users (list): List of user IDs to include; others will be excluded; Example: [“user1”, “user2”, “user3”]; Default: []
skip_users (list): List of user IDs to exclude from the output; Example: [“user4”, “user5”]; Default: []
timestamp_column_name (str): Name of the column containing timestamps; Example: “timestamp”; Default: ‘timestamp’
userid_column_name (str): Name of the column containing user IDs; Example: “username”; Default: ‘username’

monitor_options:

description (str): Name to show for this Monitor Stage in the console window; Example: ‘Progress’; Default: ‘Progress’
silence_monitors (bool): Slience the monitors on the console; Example: True; Default: False
smoothing (float): Smoothing parameter to determine how much the throughput should be averaged. 0 = Instantaneous, 1 = Average.; Example: 0.01; Default: 0.05
unit (str): Units to show in the rate value.; Example: ‘messages’; Default: ‘messages’
delayed_start (bool): When delayed_start is enabled, the progress bar will not be shown until the first message is received. Otherwise, the progress bar is shown on pipeline startup and will begin timing immediately. In large pipelines, this option may be desired to give a more accurate timing; Example: True; Default: False
determine_count_fn_schema (str): Custom function for determining the count in a message. Gets called for each message. Allows for correct counting of batched and sliced messages.; Example: func_str; Default: None
log_level (str): Enable this stage when the configured log level is at log_level or lower; Example: ‘DEBUG’; Default: INFO

write_to_file_options:

filename (str): Path to the output file; Example: output.csv; Default: None
file_type (FileTypes): Type of file to write; Example: FileTypes.CSV; Default: FileTypes.Auto
flush (bool): If true, flush the file after each write; Example: false; Default: false
include_index_col (bool): If true, include the index column; Example: false; Default: true
overwrite (bool): If true, overwrite the file if it exists; Example: true; Default: false