SQL Loader - NVIDIA Docs

DataLoader module is configured to use this loader function. SQL loader to fetch data from a SQL database and store it in a DataFrame, and returns the updated ControlMessage object with payload as MessageMeta.

Example Loader Configuration

Copy
Copied!

            
            {
  "loaders": [
    {
      "id": "SQLLoader"
    }
  ]
}

Note : Loaders can receive configuration from the load task via ControlMessage during runtime.

Task Configurable Parameters

The parameters that can be configured for this specific loader at load task level:

Parameter	Type	Description	Example Value	Default Value
`strategy`	string	Strategy for combining queries	“aggregate”	`aggregate`
`loader_id`	string	Unique identifier for the loader	“file_to_df”	`[Required]`
`sql_config`	dictionary	Dictionary containing SQL queries to run	“file_to_df”	`See below`

sql_config

Parameter	Type	Description	Example Value	Default Value
`queries`	list	List of dictionaries composing a query definition	“[query_dict_1, …, query_dict_n]”	`See below`

queries

Parameter	Type	Description	Example Value	Default Value
`connection_string`	string	Strategy for combining queries	“postgresql://postgres:postgres@localhost:5432/postgres”	`[required]`
`query`	string	SQL Query to execute	“SELECT * FROM test_table WHERE id IN (?, ?, ?)”	`[Required]`
`params`	dictionary	Named or positional paramters values	“[foo, bar, baz]”	`-`

Example Load Task Configuration

Below JSON configuration specifies how to pass additional configuration to the loader through a control message task at runtime.

Copy
Copied!

            
            {
  "type": "load",
  "properties": {
    "loader_id": "SQLLoader",
    "strategy": "aggregate",
    "sql_config": {
      "queries": [
        {
          "connection_string": "postgresql://postgres:postgres@localhost:5431/postgres",
          "query": "SELECT * FROM test_table WHERE id IN (?, ?, ?)",
          "params": [
            "foo",
            "bar",
            "baz"
          ]
        }
      ]
    }
  }
}