SQL Loader

DataLoader module is configured to use this loader function. SQL loader to fetch data from a SQL database and store it in a DataFrame, and returns the updated ControlMessage object with payload as MessageMeta.

Copy
Copied!
            

{ "loaders": [ { "id": "SQLLoader" } ] }

Note : Loaders can receive configuration from the load task via ControlMessage during runtime.

The parameters that can be configured for this specific loader at load task level:

Parameter

Type

Description

Example Value

Default Value

strategy

string

Strategy for combining queries

“aggregate”

aggregate

loader_id

string

Unique identifier for the loader

“file_to_df”

[Required]

sql_config

dictionary

Dictionary containing SQL queries to run

“file_to_df”

See below

sql_config

Parameter

Type

Description

Example Value

Default Value

queries

list

List of dictionaries composing a query definition

“[query_dict_1, …, query_dict_n]”

See below

queries

Parameter

Type

Description

Example Value

Default Value

connection_string

string

Strategy for combining queries

“postgresql://postgres:postgres@localhost:5432/postgres”

[required]

query

string

SQL Query to execute

“SELECT * FROM test_table WHERE id IN (?, ?, ?)”

[Required]

params

dictionary

Named or positional paramters values

“[foo, bar, baz]”

-

Below JSON configuration specifies how to pass additional configuration to the loader through a control message task at runtime.

Copy
Copied!
            

{ "type": "load", "properties": { "loader_id": "SQLLoader", "strategy": "aggregate", "sql_config": { "queries": [ { "connection_string": "postgresql://postgres:postgres@localhost:5431/postgres", "query": "SELECT * FROM test_table WHERE id IN (?, ?, ?)", "params": [ "foo", "bar", "baz" ] } ] } } }

© Copyright 2023, NVIDIA. Last updated on Aug 23, 2023.