DataLoader module is configured to use this loader function. SQL loader to fetch data from a SQL database and store it in a DataFrame, and returns the updated ControlMessage object with payload as MessageMeta.
{
"loaders": [
{
"id": "SQLLoader"
}
]
}
Note : Loaders can receive configuration from the load task via ControlMessage during runtime.
The parameters that can be configured for this specific loader at load task level:
Parameter |
Type |
Description |
Example Value |
Default Value |
|---|---|---|---|---|
strategy |
string | Strategy for combining queries | “aggregate” | aggregate |
loader_id |
string | Unique identifier for the loader | “file_to_df” | [Required] |
sql_config |
dictionary | Dictionary containing SQL queries to run | “file_to_df” | See below |
sql_config
Parameter |
Type |
Description |
Example Value |
Default Value |
|---|---|---|---|---|
queries |
list | List of dictionaries composing a query definition | “[query_dict_1, …, query_dict_n]” | See below |
queries
Parameter |
Type |
Description |
Example Value |
Default Value |
|---|---|---|---|---|
connection_string |
string | Strategy for combining queries | “postgresql://postgres:postgres@localhost:5432/postgres” | [required] |
query |
string | SQL Query to execute | “SELECT * FROM test_table WHERE id IN (?, ?, ?)” | [Required] |
params |
dictionary | Named or positional paramters values | “[foo, bar, baz]” | - |
Below JSON configuration specifies how to pass additional configuration to the loader through a control message task at runtime.
{
"type": "load",
"properties": {
"loader_id": "SQLLoader",
"strategy": "aggregate",
"sql_config": {
"queries": [
{
"connection_string": "postgresql://postgres:postgres@localhost:5431/postgres",
"query": "SELECT * FROM test_table WHERE id IN (?, ?, ?)",
"params": [
"foo",
"bar",
"baz"
]
}
]
}
}
}