DataLoader module is configured to use this loader function. SQL loader to fetch data from a SQL database and store it in a DataFrame, and returns the updated ControlMessage object with payload as MessageMeta.
{
"loaders": [
{
"id": "SQLLoader"
}
]
}
Note : Loaders can receive configuration from the load
task via ControlMessage during runtime.
The parameters that can be configured for this specific loader at load task level:
Parameter |
Type |
Description |
Example Value |
Default Value |
---|---|---|---|---|
|
string |
Strategy for combining queries |
“aggregate” |
|
|
string |
Unique identifier for the loader |
“file_to_df” |
|
|
dictionary |
Dictionary containing SQL queries to run |
“file_to_df” |
|
sql_config
Parameter |
Type |
Description |
Example Value |
Default Value |
---|---|---|---|---|
|
list |
List of dictionaries composing a query definition |
“[query_dict_1, …, query_dict_n]” |
|
queries
Parameter |
Type |
Description |
Example Value |
Default Value |
---|---|---|---|---|
|
string |
Strategy for combining queries |
“postgresql://postgres:postgres@localhost:5432/postgres” |
|
|
string |
SQL Query to execute |
“SELECT * FROM test_table WHERE id IN (?, ?, ?)” |
|
|
dictionary |
Named or positional paramters values |
“[foo, bar, baz]” |
|
Below JSON configuration specifies how to pass additional configuration to the loader through a control message task at runtime.
{
"type": "load",
"properties": {
"loader_id": "SQLLoader",
"strategy": "aggregate",
"sql_config": {
"queries": [
{
"connection_string": "postgresql://postgres:postgres@localhost:5431/postgres",
"query": "SELECT * FROM test_table WHERE id IN (?, ?, ?)",
"params": [
"foo",
"bar",
"baz"
]
}
]
}
}
}