SQL Loader

DataLoader module is configured to use this loader function. SQL loader to fetch data from a SQL database and store it in a DataFrame, and returns the updated ControlMessage object with payload as MessageMeta.

Example Loader Configuration

{
  "loaders": [
    {
      "id": "SQLLoader"
    }
  ]
}

Note : Loaders can receive configuration from the load task via ControlMessage during runtime.

Task Configurable Parameters

The parameters that can be configured for this specific loader at load task level:

Parameter	Type	Description	Example Value	Default Value
strategy	string	Strategy for combining queries	"aggregate"	aggregate
loader_id	string	Unique identifier for the loader	"file_to_df"	[Required]
sql_config	dictionary	Dictionary containing SQL queries to run	"file_to_df"	Refer Below

sql_config

Parameter	Type	Description	Example Value	Default Value
queries	list	List of dictionaries composing a query definition	"[query_dict_1, ..., query_dict_n]"	Refer Below

queries

Parameter	Type	Description	Example Value	Default Value
connection_string	string	Strategy for combining queries	"postgresql://postgres:postgres@localhost:5432/postgres"	[required]
query	string	SQL Query to execute	"SELECT * FROM test_table WHERE id IN (?, ?, ?)"	[Required]
params	dictionary	Named or positional parameters values	"[foo, bar, baz]"	-

Example Load Task Configuration

Below JSON configuration specifies how to pass additional configuration to the loader through a control message task at runtime.

{
  "type": "load",
  "properties": {
    "loader_id": "SQLLoader",
    "strategy": "aggregate",
    "sql_config": {
      "queries": [
        {
          "connection_string": "postgresql://postgres:postgres@localhost:5431/postgres",
          "query": "SELECT * FROM test_table WHERE id IN (?, ?, ?)",
          "params": [
            "foo",
            "bar",
            "baz"
          ]
        }
      ]
    }
  }
}