REST to DataFrame Loader

DataLoader module is used to load data files content into a dataframe using custom loader function. This loader function can be configured to send REST requests with customized parameters to retrieve data from endpoints. See below for the specific configuration format.

Using below configuration while loading DataLoader module, specifies that the DataLoader module should utilize the rest loader when loading files into a dataframe.

Copy
Copied!
            

{ "loaders": [{ "id": "rest" }] }

Note : Loaders can receive configuration from the load task via [control message] during runtime.

The parameters that can be configured for this specific loader at load task level:

Parameter

Type

Description

Example Value

Default Value

loader_id string Unique identifier for the loader “rest” [Required]
strategy string Strategy for constructing dataframe “aggregate” [Required]
queries array parameters of REST queries See below [Required]

Key

Type

Description

Example Value

Default Value

method string Method of request “GET” "GET"
endpoint string Endpoint of request “0.0.0.0/path/to/target?param1=true” [Required]
port string Target port of request “80” "80"
http_version string HTTP version of request “1.1” "1.1"
content_type string Content type of request body in a POST request “text/plain” -
body string Request body in a POST request “param1=true&param2=false” -
X-Headers dictionary Customized X-Headers of request “{“X-Header1”:”header1”, “X-Header2”:”header2”}” -
params array Parameters of requested URL, override values included in endpoint “[{“param1”: “true”, “param2”:”false”}, {“param1”: “false”, “param2”:”true”}]” -

Below JSON configuration specifies how to pass additional configuration to the loader through a control message task at runtime.

Copy
Copied!
            

{ "type":"load", "properties": { "loader_id":"rest", "strategy":"aggregate", "queries":[ { "method":"<GET/POST>", "endpoint":"0.0.0.0/?param1=false&param2=true", "port": "80", "http_version": "1.1", "content_type":"text/plain", "body":"http POST body", "x-headers": { "X-Header1":"header1", "X-Header2":"header2" }, "params": [ { "param1":"true" }, { "param1":"false", "param2":"true" } ] } ] } }

Previous SQL Loader
Next Glossary of Common Morpheus Terms
© Copyright 2023, NVIDIA. Last updated on Feb 2, 2024.