# data Property

Use the Vega `data` property to specify the visualization data sources by providing an array of one or more data definitions. A data definition must be an object identified by a unique name, which can be referenced in other areas of the specification. Data can be statically defined inline (`"values":`), can reference columns from a database table using a SQL statement (`"SQL":`), or can be loaded from an existing data set (`"source":`).

JSON format:

```
"data": [
  {
    "name": <dataID>,
    "format": {
      "type": "lines" | "polys",
      "coords": {
        "x": <array>
        "y": <array>
      }
      "layout": "interleaved" | "sequential"
    "values": <valueSet> | "SQL": <dataSource> | "source": <dataSource>,
    "transform": [
      {
        "type": "aggregate"
         "fields": ["string":"string"]
         "ops": ["keyword":"keyword"]
         "as": ["string":"string"]
      }  
  },
  {
     ...
  }
]
```

The data specification has the following properties:

| Property                                                                                             | Data Type     | Required | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| ---------------------------------------------------------------------------------------------------- | ------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [name](/apis-and-interfaces/vega/vega-reference-overview/data-property#name)                         | string        | X        | User-assigned database table name.                                                                                                                                                                                                                                                                                                                                                                                                                              |
| [format](/apis-and-interfaces/vega/vega-reference-overview/data-property#format)                     | string/object |          | How the data are parsed. `polys` and `lines` are the only supported `format` mark types and are for rendering purposes only. Use the single string "short form" for polygon and simple linestring renders. Use the JSON object "long form" to provide more information for rendering more complex line types.                                                                                                                                                   |
| [Data Source](/apis-and-interfaces/vega/vega-reference-overview/data-property#data-source)           | string        |          | <p>Data source:</p><p><code>values</code>: Embedded, static data values defined inline as JSON.</p><p><code>source</code>: Name of an existing Vega data set to use as this data set’s source. Use in combination with a <a href="/apis-and-interfaces/vega/vega-reference-overview/data-property#transform">transform</a> pipeline to derive new data. You can source only one existing data set.</p><p><code>sql</code>: A SQL query that loads the data.</p> |
| [transform](/apis-and-interfaces/vega/vega-reference-overview/data-property#transform)               | string        |          | An array of transforms to perform on the input data. The output of the transform pipeline then becomes the value of this data set. Currently, can only be used with `source` data set types.                                                                                                                                                                                                                                                                    |
| [enableHitTesting](/apis-and-interfaces/vega/vega-reference-overview/data-property#enablehittesting) | boolean       |          | If true, automatically adds rowid column(s) to the SQL statement, which is required for hit-testing using the `get_result_row_for_pixel` endpoint.                                                                                                                                                                                                                                                                                                              |

## Examples

Load discrete x- and y column values using the `values` database table type:

```
vegaSpec = {
    width: 384,
    height: 564,
    data: [
        {
          name: "coordinates",
          values: [ {"x":0, "y":3}, {"x":1, "y":5} ],
    scales: [ ... elided ... ],
    marks: [ ... elided ... ]
};
```

Use the `sql` database table type to load latitude and longitude coordinates from the `tweets_data` database table:

```
vegaSpec = {
    width: 384,
    height: 564,
    data: [
        {
          name: "tweets",
          sql: "SELECT lon as x, lat as y FROM tweets_data WHERE (lon >= -32 AND lon < 66) AND (lat >= -45 AND lat < 68)"
        }
    ],
    scales: [ ... elided ... ],
    marks: [ ... elided ... ]
};
```

Use the `source` type to use the data set defined in the `sql` data section and perform aggregation transforms:

```
vegaSpec = {
      width: 384,
      height: 564,
      data: [
              {
                      name: "tweets",
                      sql: "SELECT lon as x, lat as y FROM tweets_data WHERE (lon >= -32 AND lon < 66) AND (lat >= -45 AND lat < 68)"
              },
              {
                      name: "tweets_stats",
                      source: "tweets",
                      transform: [
                              {
                                      type: "aggregate",
                                      fields: ["x", "x"],
                                      ops: ["min", "max"],
                                      as: ["minx", "maxx"]
                              }
                      ]
              },
      ],
      scales: [ ... elided ... ],
      marks: [ ... elided ... ]
}
```

## Data Properties

### name

The `name` property uniquely identifies a data set, and is used for reference by other Vega properties, such as the [Marks](/apis-and-interfaces/vega/vega-reference-overview/marks-property) property.

### format

The `format` property indicates that data preprocessing is needed before rendering the query result. If this property is not specified, data is assumed to be in row-oriented JSON format.

This property is required for [Polys](/apis-and-interfaces/vega/vega-reference-overview/marks-property#polys-type) and [Lines](/apis-and-interfaces/vega/vega-reference-overview/marks-property#lines-type) mark types. The property has one of two forms:

* The "short form", where `format` is a single string, which must be either `polys` or `lines`. This form is used for all polygon rendering, and for fast ‘in-situ’ rendering of LINESTRING data.
* The "long form", where `format` is an object containing other properties, as follows:

| Format Property | Description                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type`          | <p>Marks property type:</p><ul><li><a href="/apis-and-interfaces/vega/vega-reference-overview/marks-property#polys-type"><code>polys</code></a>\<code>\</code></li><li><a href="/apis-and-interfaces/vega/vega-reference-overview/marks-property#lines-type"><code>lines</code></a>\<code>\</code></li></ul>                                                                                                                                          |
| `coords`        | <p>Applies to <code>type: lines</code>.</p><p>Specifies <code>x</code> and <code>y</code> arrays, which must both be the same size.</p><p>This permits column extraction pertaining to line rendering and place them in a rendering buffer. The <code>coords</code> property also dictates the ordering of points in the line.</p><p>Separate x- and y-array columns are also supported.</p>                                                          |
| `layout`        | <p>(optional) Applies to <code>type: lines</code>.</p><p>Specifies how vertices are packed in the vertices column. All arrays must have the same layout:</p><ul><li><code>interleaved</code>: (default) All elements corresponding to a single vertex are ordered in adjacent pairs. For example, x0, y0, x1, y1, x2, y2.</li><li><code>sequential</code>: All elements of the same axis are adjacent. For example, x0, x1, x2, y0, y1, y2.</li></ul> |

For `lines`, each row in the query corresponds to a single line.

This lines `format` example of `interleaved` data renders ten lines, all of the same length.

```
"data": [
  {
    "name": "table",
    "sql": "select lineArrayTest.rowid as rowid, vertices, color from lineArrayTest order by color desc limit 10;",
    "format": {
      "type": "lines",
      "coords": {
        "x": ["vertices"],
        "y": [
          {"from": "vertices" }
        ]
      },
      "layout": "interleaved"
    }
  }
]
```

In this lines `format` example of `sequential` data, `x` only stores points corresponding to the x coordinate and `y` only stores points corresponding to the y coordinate. Make sure that columns only contain a single coordinate if using multiple columns in sequential layout.

```
"data": [
  {
    "name": "table",
    "sql": "select lineArrayTestSeq.rowid as rowid, x, y, color from lineArrayTestSeq order by color desc limit 10;",
    "format": {
      "type": "lines",
      "coords": {
        "x": ["x"],
        "y": ["y"]
      },
    "layout": "sequential"
    }
  }
],
```

The following example shows a fast "in-situ" LINESTRING `format`:

```
"data": [
  {
    "name": "table",
    "format": "lines",
    "sql": "SELECT rowid, linestring_column, ... FROM ..."
  }
]
```

The following example shows a polys `format`:

```
"data": [
  {
    "name": "polys",
    "format": "polys",
    "sql": "SELECT ... elided ..."
  }
]
```

### Data Source

The database table source property key-value pair specifies the location of the data and defines how the data is loaded:

| Key      | Value         | Description                                                                                                                                                                                                                                                                                                                                                                                                                       |
| -------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `source` | String        | Data is loaded from an existing data set.                                                                                                                                                                                                                                                                                                                                                                                         |
| `sql`    | SQL statement | <p>Data is loaded using a SQL statement.</p><p>You can use <a href="/sql/data-manipulation-dml/sql-extensions">extention functions</a> to convert distance in meters from a coordinate or point to a pixel size, and determine if a coordinate or point is located within a view defined by latitude and longitude. For more information, see <a href="/sql/data-manipulation-dml/sql-extensions">OmniSci SQL Extensions</a>.</p> |
| `values` | JSON data     | Data is loaded from static, key-value pair data definitions.                                                                                                                                                                                                                                                                                                                                                                      |

### transform

Transforms process a data stream to calculate new aggregated statistic fields and derive new data streams from them. Currently, transforms are specified only as part of a `source` data definition. Transforms are defined as an array of specific transform types that are executed in sequential order. Each element of the array must be an object and must contain a `type` property. Currently, two transform types are supported: `aggregate` and `formula`.

| Type        | Description and Properties                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `aggregate` | <p>Performs aggregation operations on input data columns to calculate new aggregated statistic fields and derive new data streams from them. The following properties are required:</p><p><code>fields</code>: An array of strings referencing columns from the sourced data table.</p><p><code>ops</code>: An array of keyword strings and objects indicating the predefined operation to perform. For objects, the <code>type</code> property is required to name the type of the aggregation function. Supported operators:</p><ul><li><code>count</code>: The total count of data objects in the group.</li><li><code>countdistinct</code>: The number of distinct values in an input data column; operates only on numeric or dictionary-encoded string columns.</li><li><code>distinct</code>: An array of distinct values from an input data column; operates only on numeric or dictionary-encoded string columns.</li><li><code>max</code>: The maximum field value.</li><li><code>mean / average / avg</code>: The mean (average) field value.</li><li><code>median</code>: The median of an input data column; operates only on numeric columns.</li><li><code>min</code>: The minimum field value.</li><li><code>missing</code>: The count of field values that are null or undefined.</li><li><p><code>quantile</code>: An array of quantile separators; see [https://en.wikipedia.org/wiki/Quantile](https://en.wikipedia.org/wiki/Quantile). Operates only on numeric columns:</p><ul><li><code>numQuantiles</code>: The number of contiguous intervals to create; returns the separators for the intervals. The number of separators equals <code>numQuantiles - 1</code>. <strong>Range:</strong> 1-100. <strong>Default:</strong> 4</li><li><code>includeExtrema</code>: Whether to include min and max values (extrema) in the resulting separator array. When <code>true</code>, the resulting array size is <code>numQuantiles</code> + 1. <strong>Values:</strong> <code>true</code> or <code>false</code>. <strong>Default:</strong> false</li></ul></li><li><code>sum</code>: The sum of field values.</li><li><code>stddev</code>: The sample standard deviation of field values.</li><li><code>stddevp</code>: The population standard deviation of field values.</li><li><code>valid</code>: The count of field values that are not null nor undefined.</li><li><code>variance</code>: The sample variance of field values.</li><li><code>variancep</code>: The population variance of field values.</li></ul><p><code>as</code>: An array of strings used as output names of the operations for later reference.</p> |
| `formula`   | <p>Evaluates a user-defined expression. The following properties are required:</p><p><code>expr</code>: An expression string to be evaluated. Expressions currently support <a href="https://beltoforion.de/en/muparsersse/index.php#idFeatures">these operators and functions.</a></p><p><code>as</code>: A string used as an output name for later reference.</p><p><strong>Note:</strong> Currently, expressions can only be performed against outputs (as values) from prior aggregate transforms.</p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |

See [Tutorial: Using Transforms](/apis-and-interfaces/vega/vega-tutorials/tutorial-using-transforms-aggregation) for more detailed examples.

### enableHitTesting

If `true`, automatically adds rowid column(s) to the SQL statement where appropriate, enabling the data block for hit-testing using the `get_result_row_for_pixel` endpoint.

If `false`, the data block is not automatically hit-test enabled, and any later `get_result_row_for_pixel` calls return empty hit-test results.

If the enableHitTesting property is not present, the following legacy behavior is used as the default:

* If the SQL statement represents a projection query, hit-testing is enabled if a rowid column is explicitly projected.
* If the SQL statement represents an aggregate query, hit-testing is always enabled.

<Info>
  This legacy behavior will likely be deprecated and removed in an upcoming version of OmniSci. At that point, the enableHitTesting property will be required for activating hit-test support for the data.
</Info>