SQL Loader

DataLoader module is configured to use this loader function. SQL loader to fetch data from a SQL database and store it in a DataFrame, and returns the updated ControlMessage object with payload as MessageMeta.

Copy
Copied!
            

{ "loaders": [ { "id": "SQLLoader" } ] }

Note : Loaders can receive configuration from the load task via ControlMessage during runtime.

The parameters that can be configured for this specific loader at load task level:

Parameter

Type

Description

Example Value

Default Value

strategy string Strategy for combining queries “aggregate” aggregate
loader_id string Unique identifier for the loader “file_to_df” [Required]
sql_config dictionary Dictionary containing SQL queries to run “file_to_df” See below

sql_config

Parameter

Type

Description

Example Value

Default Value

queries list List of dictionaries composing a query definition “[query_dict_1, …, query_dict_n]” See below

queries

Parameter

Type

Description

Example Value

Default Value

connection_string string Strategy for combining queries “postgresql://postgres:postgres@localhost:5432/postgres” [required]
query string SQL Query to execute “SELECT * FROM test_table WHERE id IN (?, ?, ?)” [Required]
params dictionary Named or positional paramters values “[foo, bar, baz]” -

Below JSON configuration specifies how to pass additional configuration to the loader through a control message task at runtime.

Copy
Copied!
            

{ "type": "load", "properties": { "loader_id": "SQLLoader", "strategy": "aggregate", "sql_config": { "queries": [ { "connection_string": "postgresql://postgres:postgres@localhost:5431/postgres", "query": "SELECT * FROM test_table WHERE id IN (?, ?, ?)", "params": [ "foo", "bar", "baz" ] } ] } } }

Previous Filesystem Spec Loader
Next REST to DataFrame Loader
© Copyright 2023, NVIDIA. Last updated on Feb 2, 2024.