{ "cells": [ { "cell_type": "markdown", "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "db9cf912-1f7b-431d-9628-1afdea311779", "showTitle": false, "title": "" }, "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "library/dask", "library/dask-cudf", "library/xgboost", "library/dask-deltatable", "library/dask-databricks", "library/dask-ml", "workflow/xgboost", "dataset/higgs", "data-format/csv", "data-storage/databricks-delta-lake", "platforms/databricks" ] }, "source": [ "# Training XGBoost with Dask RAPIDS in Databricks\n", "\n", "_January, 2024_\n", "\n", "This notebook shows how to deploy Dask RAPIDS workflow in Databricks. We will focus on the HIGGS dataset, a moderately sized classification problem from the [UCI Machine Learning repository.](https://archive.ics.uci.edu/dataset/280/higgs)\n", "\n", "In the following sections, we will begin by loading the dataset from [Delta Lake](https://delta.io/) and performing preprocessing with [Dask](https://github.com/dask/dask). Then train an [XGBoost](https://xgboost.readthedocs.io/en/stable/) model with various configurations and explore techniques for optimizing inference.\n" ] }, { "cell_type": "markdown", "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "1518eb70-21f5-4768-b06a-b376ec379335", "showTitle": false, "title": "" }, "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "data-storage/databricks-delta-lake" ] }, "source": [ "## Launch multi-node Dask Cluster\n", "\n", "This workflow example can be ran on GPU, and you don't even need to have the GPU locally since Databricks can provide one for you. Whereas Dask enables users to easily distribute or scale up computation tasks within a single GPU or across multiple GPUs.\n", "\n", "Dask recently introduced [**dask-databricks**](https://github.com/dask-contrib/dask-databricks) (available via [conda](https://github.com/conda-forge/dask-databricks-feedstock) and [pip](https://pypi.org/project/dask-databricks/)). With this CLI tool, the `dask databricks run --cuda` command will launch a Dask scheduler in the driver node and [`cuda` workers](https://docs.rapids.ai/api/dask-cuda/~~~rapids_api_docs_version~~~) in the remaining nodes.\n", "\n", "From a high level, we could break down this section into the following steps:\n", "* Create a new [init script](https://docs.databricks.com/en/init-scripts/index.html) that installs [RAPIDS](https://rapids.ai/) and runs `dask-databricks`\n", "* Create a new multi-node cluster that uses the init script\n", "* Once the cluster is running upload this notebook to Databricks and continue running these cells on there\n", "\n", "```{docref} https://github.com/rapidsai/deployment/blob/v25.08.00/source/platforms/databricks.md#multi-node-dask-cluster\n", "For more detailed information on launching Dask-RAPIDS in Databricks see the documentation.\n", "```\n" ] }, { "cell_type": "markdown", "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "dd79b033-6013-471d-a70c-a55d55fe52c1", "showTitle": false, "title": "" }, "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "## Import packages \n", "\n", "Once your cluster has launched, start by importing all necessary libraries and dependencies." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "109cfcaa-f2e4-446a-a3fa-c4025e0b70a4", "showTitle": false, "title": "" }, "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "data": { "text/plain": [] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import os\n", "\n", "import dask_cudf\n", "import dask_databricks\n", "import dask_deltatable as ddt\n", "import numpy as np\n", "import xgboost as xgb\n", "from dask_ml.model_selection import train_test_split\n", "from distributed import wait\n", "from xgboost import dask as dxgb" ] }, { "cell_type": "markdown", "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "90de1e5e-d645-46db-85ac-874f0a191412", "showTitle": false, "title": "" } }, "source": [ "## Connect to Dask Client\n", "\n", "Connect to the client (and optionally Dashboard) to submit tasks." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "0b6e31e3-d4da-4c86-9b10-d9eac4e4a40f", "showTitle": false, "title": "" } }, "outputs": [ { "data": { "text/html": [ "
\n", "
\n", "
\n", "

Client

\n", "

Client-23114b4f-b7aa-11ee-87d9-9a67d50005f3

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
Connection method: Cluster objectCluster type: dask_databricks.DatabricksCluster
\n", " Dashboard: https://dbc-dp-8721196619973675.cloud.databricks.com/driver-proxy/o/8721196619973675/1031-230718-l2ubf858/8087/status\n", "
\n", "\n", " \n", "\n", " \n", "
\n", "

Cluster Info

\n", "
\n", "
\n", "
\n", "
\n", "

DatabricksCluster

\n", "

1031-230718-l2ubf858

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", " Dashboard: https://dbc-dp-8721196619973675.cloud.databricks.com/driver-proxy/o/8721196619973675/1031-230718-l2ubf858/8087/status\n", " \n", " Workers: 2\n", "
\n", " Total threads: 2\n", " \n", " Total memory: 30.65 GiB\n", "
\n", "\n", "
\n", " \n", "

Scheduler Info

\n", " \n", "\n", "
\n", "
\n", "
\n", "
\n", "

Scheduler

\n", "

Scheduler-f908617a-76cd-4f5b-8fc9-fb04a02e0c99

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", " Comm: tcp://10.59.146.44:8786\n", " \n", " Workers: 2\n", "
\n", " Dashboard: http://10.59.146.44:8087/status\n", " \n", " Total threads: 2\n", "
\n", " Started: 22 minutes ago\n", " \n", " Total memory: 30.65 GiB\n", "
\n", "
\n", "
\n", "\n", "
\n", " \n", "

Workers

\n", " \n", "\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://10.59.135.19:33999

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://10.59.135.19:33999\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: http://10.59.135.19:35075/status\n", " \n", " Memory: 15.33 GiB\n", "
\n", " Nanny: tcp://10.59.135.19:41477\n", "
\n", " Local directory: /tmp/dask-scratch-space/worker-639byx42\n", "
\n", " GPU: Tesla T4\n", " \n", " GPU memory: 15.00 GiB\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://10.59.155.0:45293

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://10.59.155.0:45293\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: http://10.59.155.0:44287/status\n", " \n", " Memory: 15.33 GiB\n", "
\n", " Nanny: tcp://10.59.155.0:35699\n", "
\n", " Local directory: /tmp/dask-scratch-space/worker-i0pmkkyv\n", "
\n", " GPU: Tesla T4\n", " \n", " GPU memory: 15.00 GiB\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", "\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
" ] }, "metadata": { "application/vnd.databricks.v1+output": { "addedWidgets": {}, "arguments": {}, "data": "
\n
\n
\n

Client

\n

Client-23114b4f-b7aa-11ee-87d9-9a67d50005f3

\n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n\n
Connection method: Cluster objectCluster type: dask_databricks.DatabricksCluster
\n Dashboard: https://dbc-dp-8721196619973675.cloud.databricks.com/driver-proxy/o/8721196619973675/1031-230718-l2ubf858/8087/status\n
\n\n \n\n \n
\n

Cluster Info

\n
\n
\n
\n
\n

DatabricksCluster

\n

1031-230718-l2ubf858

\n \n \n \n \n \n \n \n \n \n \n
\n Dashboard: https://dbc-dp-8721196619973675.cloud.databricks.com/driver-proxy/o/8721196619973675/1031-230718-l2ubf858/8087/status\n \n Workers: 2\n
\n Total threads: 2\n \n Total memory: 30.65 GiB\n
\n\n
\n \n

Scheduler Info

\n \n\n
\n
\n
\n
\n

Scheduler

\n

Scheduler-f908617a-76cd-4f5b-8fc9-fb04a02e0c99

\n \n \n \n \n \n \n \n \n \n \n \n \n \n
\n Comm: tcp://10.59.146.44:8786\n \n Workers: 2\n
\n Dashboard: http://10.59.146.44:8087/status\n \n Total threads: 2\n
\n Started: 22 minutes ago\n \n Total memory: 30.65 GiB\n
\n
\n
\n\n
\n \n

Workers

\n \n\n \n
\n
\n
\n
\n \n

Worker: tcp://10.59.135.19:33999

\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n\n
\n Comm: tcp://10.59.135.19:33999\n \n Total threads: 1\n
\n Dashboard: http://10.59.135.19:35075/status\n \n Memory: 15.33 GiB\n
\n Nanny: tcp://10.59.135.19:41477\n
\n Local directory: /tmp/dask-scratch-space/worker-639byx42\n
\n GPU: Tesla T4\n \n GPU memory: 15.00 GiB\n
\n
\n
\n
\n \n
\n
\n
\n
\n \n

Worker: tcp://10.59.155.0:45293

\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n\n \n\n
\n Comm: tcp://10.59.155.0:45293\n \n Total threads: 1\n
\n Dashboard: http://10.59.155.0:44287/status\n \n Memory: 15.33 GiB\n
\n Nanny: tcp://10.59.155.0:35699\n
\n Local directory: /tmp/dask-scratch-space/worker-i0pmkkyv\n
\n GPU: Tesla T4\n \n GPU memory: 15.00 GiB\n
\n
\n
\n
\n \n\n
\n
\n\n
\n
\n
\n
\n \n\n
\n
", "datasetInfos": [], "metadata": {}, "removedWidgets": [], "textData": null, "type": "htmlSandbox" } }, "output_type": "display_data" } ], "source": [ "client = dask_databricks.get_client()\n", "client" ] }, { "cell_type": "markdown", "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "c658c7b9-d0ce-43d1-ac5c-78e271bef57f", "showTitle": false, "title": "" } }, "source": [ "\n", "## Download dataset\n", "\n", "First we download the dataset to Databrick File Storage (DBFS). Alternatively, you could also use cloud storage ([S3](https://aws.amazon.com/s3/), [Google Cloud](https://cloud.google.com/storage?hl=en), [Azure Data Lake](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction)\n", "Refer to [docs](https://docs.databricks.com/en/storage/index.html#:~:text=Databricks%20uses%20cloud%20object%20storage,storage%20locations%20in%20your%20account.) for more information\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "373c083a-44a0-4cc3-9ac7-ffc6f927643d", "showTitle": false, "title": "" } }, "outputs": [], "source": [ "import subprocess\n", "\n", "# Define the directory and file paths\n", "directory_path = \"/dbfs/databricks/rapids\"\n", "file_path = f\"{directory_path}/HIGGS.csv.gz\"\n", "\n", "# Check if directory already exists\n", "if not os.path.exists(directory_path):\n", " os.makedirs(directory_path)\n", "\n", "# Check if the file already exists\n", "if not os.path.exists(file_path):\n", " # If not, download dataset to the directory\n", " data_url = (\n", " \"https://archive.ics.uci.edu/ml/machine-learning-databases/00280/HIGGS.csv.gz\"\n", " )\n", " download_command = f\"curl {data_url} --output {file_path}\"\n", " subprocess.run(download_command, shell=True)\n", "\n", " # decompress the csv file\n", " decompress_command = f\"gunzip {file_path}\"\n", " subprocess.run(decompress_command, shell=True)" ] }, { "cell_type": "markdown", "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "ea03cd9a-9077-4cfc-a1a0-becdc84efe1a", "showTitle": false, "title": "" } }, "source": [ "Next we load the data into GPUs. Because the data is loaded multiple times during parameter tuning, we convert the original CSV file into Parquet format for better performance. This can be easily done using delta lake as shown in the next steps." ] }, { "cell_type": "markdown", "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "2a87ba1c-14be-4d99-9b79-15d62052a902", "showTitle": false, "title": "" } }, "source": [ "## Integrating Dask and Delta Lake\n", "\n", "[**Delta Lake**](https://docs.databricks.com/en/delta/index.html) is an optimized storage layer within the Databricks lakehouse that provides a foundational platform for storing data and tables. This open-source software extends Parquet data files by incorporating a file-based transaction log to support [ACID transactions](https://docs.databricks.com/en/lakehouse/acid.html) and scalable metadata handling. \n", "\n", "Delta Lake is the default storage format for all operations on Databricks, i.e (unless otherwise specified, all tables on Databricks are Delta tables). \n", "Check out [tutorial for examples with basic Delta Lake operations](https://docs.databricks.com/en/delta/tutorial.html).\n", "\n", "Let's explore step-by-step how we can leverage Delta Lake tables with Dask to accelerate data pre-processing with RAPIDS." ] }, { "cell_type": "markdown", "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "48b26e2b-5563-4836-a67c-dae5c9607db8", "showTitle": false, "title": "" } }, "source": [ "## Read from Delta table with Dask\n", "\n", "With Dask's [**dask-deltatable**](https://github.com/dask-contrib/dask-deltatable/tree/main), we can write the `.csv` file into a Delta table using [**Spark**](https://spark.apache.org/docs/latest/) then read and parallelize with [**Dask**](https://docs.dask.org/en/stable/). " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "a2bd85a8-c4e2-419e-885a-b0a64c796dee", "showTitle": false, "title": "" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The Delta table 'higgs_delta_table' already exists.\n" ] } ], "source": [ "delta_table_name = \"higgs_delta_table\"\n", "\n", "# Check if the Delta table already exists\n", "if spark.catalog.tableExists(delta_table_name):\n", " # If it exists, print a message\n", " print(f\"The Delta table '{delta_table_name}' already exists.\")\n", "else:\n", " # If not, Load csv file into a Spark dataframe then\n", " # Write the spark dataframe into delta table\n", " data = spark.read.csv(file_path, header=True, inferSchema=True)\n", " data.write.saveAsTable(delta_table_name)\n", " print(f\"The Delta table '{delta_table_name}' has been created.\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "6ebe932f-097a-4b6c-80d2-c4676c807d19", "showTitle": false, "title": "" } }, "outputs": [ { "data": { "text/html": [ "
formatidnamedescriptionlocationcreatedAtlastModifiedpartitionColumnsnumFilessizeInBytespropertiesminReaderVersionminWriterVersiontableFeaturesstatistics
delta90cdac79-5500-4a20-914b-47f86b616275spark_catalog.default.higgs_delta_tablenulldbfs:/user/hive/warehouse/higgs_delta_table2024-01-09T15:01:35.629+00002024-01-09T15:04:37.000+0000List()60906326187Map()12List(appendOnly, invariants)Map()
" ] }, "metadata": { "application/vnd.databricks.v1+output": { "addedWidgets": {}, "aggData": [], "aggError": "", "aggOverflow": false, "aggSchema": [], "aggSeriesLimitReached": false, "aggType": "", "arguments": {}, "columnCustomDisplayInfos": {}, "data": [ [ "delta", "90cdac79-5500-4a20-914b-47f86b616275", "spark_catalog.default.higgs_delta_table", null, "dbfs:/user/hive/warehouse/higgs_delta_table", "2024-01-09T15:01:35.629+0000", "2024-01-09T15:04:37.000+0000", [], 60, 906326187, {}, 1, 2, [ "appendOnly", "invariants" ], {} ] ], "datasetInfos": [], "dbfsResultPath": null, "isJsonSchema": true, "metadata": {}, "overflow": false, "plotOptions": { "customPlotOptions": {}, "displayType": "table", "pivotAggregation": null, "pivotColumns": null, "xColumns": null, "yColumns": null }, "removedWidgets": [], "schema": [ { "metadata": "{}", "name": "format", "type": "\"string\"" }, { "metadata": "{}", "name": "id", "type": "\"string\"" }, { "metadata": "{}", "name": "name", "type": "\"string\"" }, { "metadata": "{}", "name": "description", "type": "\"string\"" }, { "metadata": "{}", "name": "location", "type": "\"string\"" }, { "metadata": "{}", "name": "createdAt", "type": "\"timestamp\"" }, { "metadata": "{}", "name": "lastModified", "type": "\"timestamp\"" }, { "metadata": "{}", "name": "partitionColumns", "type": "{\"type\":\"array\",\"elementType\":\"string\",\"containsNull\":true}" }, { "metadata": "{}", "name": "numFiles", "type": "\"long\"" }, { "metadata": "{}", "name": "sizeInBytes", "type": "\"long\"" }, { "metadata": "{}", "name": "properties", "type": "{\"type\":\"map\",\"keyType\":\"string\",\"valueType\":\"string\",\"valueContainsNull\":true}" }, { "metadata": "{}", "name": "minReaderVersion", "type": "\"integer\"" }, { "metadata": "{}", "name": "minWriterVersion", "type": "\"integer\"" }, { "metadata": "{}", "name": "tableFeatures", "type": "{\"type\":\"array\",\"elementType\":\"string\",\"containsNull\":true}" }, { "metadata": "{}", "name": "statistics", "type": "{\"type\":\"map\",\"keyType\":\"string\",\"valueType\":\"long\",\"valueContainsNull\":true}" } ], "type": "table" } }, "output_type": "display_data" } ], "source": [ "display(spark.sql(\"DESCRIBE DETAIL higgs_delta_table\"))" ] }, { "cell_type": "markdown", "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "9e4f2b0f-527e-4672-87c4-9850f3084fc0", "showTitle": false, "title": "" } }, "source": [ "Calling `dask_deltalake.read_deltalake()` will return a `dask dataframe`. However, our objective is to utilize GPU acceleration for the entire ML pipeline, including data processing, model training and inference. For this reason, we will read the dask dataframe into a `cuDF dask-dataframe` using `dask_cudf.from_dask_dataframe()`\n", "\n", "**Note** that these operations will automatically leverage the Dask client we created, ensuring optimal performance boost through parallelism with dask." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "585648f2-1799-45b7-b1ca-a6a74525b136", "showTitle": false, "title": "" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
1.000000000000000000e+008.692932128906250000e-01-6.350818276405334473e-012.256902605295181274e-013.274700641632080078e-01-6.899932026863098145e-017.542022466659545898e-01-2.485731393098831177e-01-1.092063903808593750e+000.000000000000000000e+009...-1.045456994324922562e-02-4.576716944575309753e-023.101961374282836914e+001.353760004043579102e+009.795631170272827148e-019.780761599540710449e-019.200048446655273438e-017.216574549674987793e-019.887509346008300781e-018.766783475875854492e-01
01.00.9075420.3291470.3594121.497970-0.3130101.095531-0.557525-1.5882302.173076...-1.138930-0.0008190.0000000.3022200.8330480.9857000.9780980.7797320.9923560.798343
11.00.7988351.470639-1.6359750.4537730.4256291.1048751.2823221.3816640.000000...1.1288480.9004610.0000000.9097531.1083300.9856920.9513310.8032520.8659240.780118
20.01.344385-0.8766260.9359131.9920500.8824541.786066-1.646778-0.9423830.000000...-0.678379-1.3603560.0000000.9466521.0287040.9986560.7282810.8692001.0267360.957904
31.01.1050090.3213561.5224010.882808-1.2053490.681466-1.070464-0.9218710.000000...-0.3735660.1130410.0000000.7558561.3610570.9866100.8380851.1332950.8722450.808487
40.01.595839-0.6078110.0070751.818450-0.1119060.847550-0.5664371.5812392.173076...-0.654227-1.2743453.1019610.8237610.9381910.9717580.7891760.4305530.9613570.957818
\n", "

5 rows × 29 columns

\n", "
" ] }, "metadata": { "application/vnd.databricks.v1+output": { "addedWidgets": {}, "arguments": {}, "data": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
1.000000000000000000e+008.692932128906250000e-01-6.350818276405334473e-012.256902605295181274e-013.274700641632080078e-01-6.899932026863098145e-017.542022466659545898e-01-2.485731393098831177e-01-1.092063903808593750e+000.000000000000000000e+009...-1.045456994324922562e-02-4.576716944575309753e-023.101961374282836914e+001.353760004043579102e+009.795631170272827148e-019.780761599540710449e-019.200048446655273438e-017.216574549674987793e-019.887509346008300781e-018.766783475875854492e-01
01.00.9075420.3291470.3594121.497970-0.3130101.095531-0.557525-1.5882302.173076...-1.138930-0.0008190.0000000.3022200.8330480.9857000.9780980.7797320.9923560.798343
11.00.7988351.470639-1.6359750.4537730.4256291.1048751.2823221.3816640.000000...1.1288480.9004610.0000000.9097531.1083300.9856920.9513310.8032520.8659240.780118
20.01.344385-0.8766260.9359131.9920500.8824541.786066-1.646778-0.9423830.000000...-0.678379-1.3603560.0000000.9466521.0287040.9986560.7282810.8692001.0267360.957904
31.01.1050090.3213561.5224010.882808-1.2053490.681466-1.070464-0.9218710.000000...-0.3735660.1130410.0000000.7558561.3610570.9866100.8380851.1332950.8722450.808487
40.01.595839-0.6078110.0070751.818450-0.1119060.847550-0.5664371.5812392.173076...-0.654227-1.2743453.1019610.8237610.9381910.9717580.7891760.4305530.9613570.957818
\n

5 rows × 29 columns

\n
", "datasetInfos": [], "metadata": {}, "removedWidgets": [], "textData": null, "type": "htmlSandbox" } }, "output_type": "display_data" } ], "source": [ "# Read the Delta Lake into a Dask DataFrame using `dask-deltatable`\n", "df = ddt.read_deltalake(\"/dbfs/user/hive/warehouse/higgs_delta_table\")\n", "\n", "# Convert Dask DataFrame to Dask cuDF for GPU acceleration\n", "ddf = dask_cudf.from_dask_dataframe(df)\n", "\n", "ddf.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "1cdae47a-46a3-4015-bdc6-836c2823e295", "showTitle": false, "title": "" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
labelfeature-01feature-02feature-03feature-04feature-05feature-06feature-07feature-08feature-09...feature-19feature-20feature-21feature-22feature-23feature-24feature-25feature-26feature-27feature-28
01.00.9075420.3291470.3594121.497970-0.3130101.095531-0.557525-1.5882302.173076...-1.138930-0.0008190.0000000.3022200.8330480.9857000.9780980.7797320.9923560.798343
11.00.7988351.470639-1.6359750.4537730.4256291.1048751.2823221.3816640.000000...1.1288480.9004610.0000000.9097531.1083300.9856920.9513310.8032520.8659240.780118
20.01.344385-0.8766260.9359131.9920500.8824541.786066-1.646778-0.9423830.000000...-0.678379-1.3603560.0000000.9466521.0287040.9986560.7282810.8692001.0267360.957904
31.01.1050090.3213561.5224010.882808-1.2053490.681466-1.070464-0.9218710.000000...-0.3735660.1130410.0000000.7558561.3610570.9866100.8380851.1332950.8722450.808487
40.01.595839-0.6078110.0070751.818450-0.1119060.847550-0.5664371.5812392.173076...-0.654227-1.2743453.1019610.8237610.9381910.9717580.7891760.4305530.9613570.957818
\n", "

5 rows × 29 columns

\n", "
" ] }, "metadata": { "application/vnd.databricks.v1+output": { "addedWidgets": {}, "arguments": {}, "data": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
labelfeature-01feature-02feature-03feature-04feature-05feature-06feature-07feature-08feature-09...feature-19feature-20feature-21feature-22feature-23feature-24feature-25feature-26feature-27feature-28
01.00.9075420.3291470.3594121.497970-0.3130101.095531-0.557525-1.5882302.173076...-1.138930-0.0008190.0000000.3022200.8330480.9857000.9780980.7797320.9923560.798343
11.00.7988351.470639-1.6359750.4537730.4256291.1048751.2823221.3816640.000000...1.1288480.9004610.0000000.9097531.1083300.9856920.9513310.8032520.8659240.780118
20.01.344385-0.8766260.9359131.9920500.8824541.786066-1.646778-0.9423830.000000...-0.678379-1.3603560.0000000.9466521.0287040.9986560.7282810.8692001.0267360.957904
31.01.1050090.3213561.5224010.882808-1.2053490.681466-1.070464-0.9218710.000000...-0.3735660.1130410.0000000.7558561.3610570.9866100.8380851.1332950.8722450.808487
40.01.595839-0.6078110.0070751.818450-0.1119060.847550-0.5664371.5812392.173076...-0.654227-1.2743453.1019610.8237610.9381910.9717580.7891760.4305530.9613570.957818
\n

5 rows × 29 columns

\n
", "datasetInfos": [], "metadata": {}, "removedWidgets": [], "textData": null, "type": "htmlSandbox" } }, "output_type": "display_data" } ], "source": [ "colnames = [\"label\"] + [f\"feature-{i:02d}\" for i in range(1, 29)]\n", "ddf.columns = colnames\n", "ddf.head()" ] }, { "cell_type": "markdown", "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "79d52d53-c22b-485c-82f5-7c6e8e9f486a", "showTitle": false, "title": "" } }, "source": [ "## Split data\n", "\n", "In the preceding step, we used [**`dask-cudf`**](https://docs.rapids.ai/api/dask-cudf/~~~rapids_api_docs_version~~~/) for loading data from the Delta table, now use `train_test_split()` function from [**`dask-ml`**](https://ml.dask.org/modules/api.html) to split up the dataset. \n", "\n", "Most of the time, the GPU backend of Dask works seamlessly with utilities in `dask-ml` and we can accelerate the entire ML pipeline as such: \n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "7f2f4238-a2a0-445a-a5f5-ee1e554a7548", "showTitle": false, "title": "" } }, "outputs": [], "source": [ "def load_higgs(\n", " ddf,\n", ") -> tuple[\n", " dask_cudf.core.DataFrame,\n", " dask_cudf.core.Series,\n", " dask_cudf.core.DataFrame,\n", " dask_cudf.core.Series,\n", "]:\n", " y = ddf[\"label\"]\n", " X = ddf[ddf.columns.difference([\"label\"])]\n", "\n", " X_train, X_valid, y_train, y_valid = train_test_split(\n", " X, y, test_size=0.33, random_state=42\n", " )\n", " X_train, X_valid, y_train, y_valid = client.persist(\n", " [X_train, X_valid, y_train, y_valid]\n", " )\n", " wait([X_train, X_valid, y_train, y_valid])\n", "\n", " return X_train, X_valid, y_train, y_valid" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "8539a1aa-fc35-4add-a067-81ca7ceaea5a", "showTitle": false, "title": "" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/databricks/python/lib/python3.10/site-packages/dask_ml/model_selection/_split.py:462: FutureWarning: The default value for 'shuffle' must be specified when splitting DataFrames. In the future DataFrames will automatically be shuffled within blocks prior to splitting. Specify 'shuffle=True' to adopt the future behavior now, or 'shuffle=False' to retain the previous behavior.\n", " warnings.warn(\n" ] } ], "source": [ "X_train, X_valid, y_train, y_valid = load_higgs(ddf)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "450ae6b7-b33e-490b-a509-9b7a90319422", "showTitle": false, "title": "" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
feature-01feature-02feature-03feature-04feature-05feature-06feature-07feature-08feature-09feature-10...feature-19feature-20feature-21feature-22feature-23feature-24feature-25feature-26feature-27feature-28
00.9075420.3291470.3594121.497970-0.3130101.095531-0.557525-1.5882302.1730760.812581...-1.138930-0.0008190.0000000.3022200.8330480.9857000.9780980.7797320.9923560.798343
10.7988351.470639-1.6359750.4537730.4256291.1048751.2823221.3816640.0000000.851737...1.1288480.9004610.0000000.9097531.1083300.9856920.9513310.8032520.8659240.780118
31.1050090.3213561.5224010.882808-1.2053490.681466-1.070464-0.9218710.0000000.800872...-0.3735660.1130410.0000000.7558561.3610570.9866100.8380851.1332950.8722450.808487
100.739357-0.1782900.8299340.504539-0.1302170.961051-0.355518-1.7173992.1730760.620956...0.7740650.3988203.1019610.9445361.0262610.9821970.5421151.2509790.8300450.761308
111.3840980.116822-1.1798790.762913-0.0797821.0198630.8773181.2768872.1730760.331252...0.8465210.5048093.1019610.9593250.8073761.1918141.2212100.8611410.9293410.838302
\n", "

5 rows × 28 columns

\n", "
" ] }, "metadata": { "application/vnd.databricks.v1+output": { "addedWidgets": {}, "arguments": {}, "data": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
feature-01feature-02feature-03feature-04feature-05feature-06feature-07feature-08feature-09feature-10...feature-19feature-20feature-21feature-22feature-23feature-24feature-25feature-26feature-27feature-28
00.9075420.3291470.3594121.497970-0.3130101.095531-0.557525-1.5882302.1730760.812581...-1.138930-0.0008190.0000000.3022200.8330480.9857000.9780980.7797320.9923560.798343
10.7988351.470639-1.6359750.4537730.4256291.1048751.2823221.3816640.0000000.851737...1.1288480.9004610.0000000.9097531.1083300.9856920.9513310.8032520.8659240.780118
31.1050090.3213561.5224010.882808-1.2053490.681466-1.070464-0.9218710.0000000.800872...-0.3735660.1130410.0000000.7558561.3610570.9866100.8380851.1332950.8722450.808487
100.739357-0.1782900.8299340.504539-0.1302170.961051-0.355518-1.7173992.1730760.620956...0.7740650.3988203.1019610.9445361.0262610.9821970.5421151.2509790.8300450.761308
111.3840980.116822-1.1798790.762913-0.0797821.0198630.8773181.2768872.1730760.331252...0.8465210.5048093.1019610.9593250.8073761.1918141.2212100.8611410.9293410.838302
\n

5 rows × 28 columns

\n
", "datasetInfos": [], "metadata": {}, "removedWidgets": [], "textData": null, "type": "htmlSandbox" } }, "output_type": "display_data" } ], "source": [ "X_train.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "62ffc8ef-194c-4664-a45d-1a5bb0733925", "showTitle": false, "title": "" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Out[14]: 0 1.0\n", "1 1.0\n", "3 1.0\n", "10 0.0\n", "11 1.0\n", "Name: label, dtype: float64" ] } ], "source": [ "y_train.head()" ] }, { "cell_type": "markdown", "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "9190527c-ad0c-4f96-ac60-d38f6e0fd615", "showTitle": false, "title": "" } }, "source": [ "## Model training\n", "\n", "There are two things to notice here. Firstly, we specify the number of rounds to trigger early stopping for training. [XGBoost](https://xgboost.readthedocs.io/en/release_1.7.0/) will stop the training process once the validation metric fails to improve in consecutive X rounds, where **X** is the number of rounds specified for early \n", "stopping. \n", "\n", "Secondly, we use a data type called `DaskDeviceQuantileDMatrix` for training but `DaskDMatrix` for validation. `DaskDeviceQuantileDMatrix` is a drop-in replacement of `DaskDMatrix` for GPU-based training inputs that avoids extra data copies." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "514a1d11-8257-4920-a25d-bb2401b9dd43", "showTitle": false, "title": "" } }, "outputs": [], "source": [ "def fit_model_es(client, X, y, X_valid, y_valid) -> dxgb.Booster:\n", " early_stopping_rounds = 5\n", " Xy = dxgb.DaskDeviceQuantileDMatrix(client, X, y)\n", " Xy_valid = dxgb.DaskDMatrix(client, X_valid, y_valid)\n", " # train the model\n", " booster = dxgb.train(\n", " client,\n", " {\n", " \"objective\": \"binary:logistic\",\n", " \"eval_metric\": \"error\",\n", " \"tree_method\": \"gpu_hist\",\n", " },\n", " Xy,\n", " evals=[(Xy_valid, \"Valid\")],\n", " num_boost_round=1000,\n", " early_stopping_rounds=early_stopping_rounds,\n", " )[\"booster\"]\n", " return booster" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "3f3f8935-4f76-47c1-9280-386df3e6a49d", "showTitle": false, "title": "" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/databricks/python/lib/python3.10/site-packages/xgboost/dask.py:703: FutureWarning: Please use `DaskQuantileDMatrix` instead.\n", " warnings.warn(\"Please use `DaskQuantileDMatrix` instead.\", FutureWarning)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Out[16]: " ] } ], "source": [ "booster = fit_model_es(client, X=X_train, y=y_train, X_valid=X_valid, y_valid=y_valid)\n", "booster" ] }, { "cell_type": "markdown", "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "83932465-f261-4ad8-82da-f696e727edc4", "showTitle": false, "title": "" } }, "source": [ "## Train with Customized objective and evaluation metric\n", "\n", "In the example below the XGBoost model is trained using a custom logistic regression-based objective function (`logit`) and a custom evaluation metric (`error`) along with early stopping.\n", "\n", "Note that the function returns both gradient and hessian, which XGBoost uses to optimize the model. Also, the parameter named `metric_name` needs to be specified in our callback. It is used to inform XGBoost that the custom error function should be used for evaluating early stopping criteria." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "2b40ef80-c783-4216-9434-0a55c2eb3a63", "showTitle": false, "title": "" } }, "outputs": [], "source": [ "def fit_model_customized_objective(client, X, y, X_valid, y_valid) -> dxgb.Booster:\n", " def logit(predt: np.ndarray, Xy: xgb.DMatrix) -> tuple[np.ndarray, np.ndarray]:\n", " predt = 1.0 / (1.0 + np.exp(-predt))\n", " labels = Xy.get_label()\n", " grad = predt - labels\n", " hess = predt * (1.0 - predt)\n", " return grad, hess\n", "\n", " def error(predt: np.ndarray, Xy: xgb.DMatrix) -> tuple[str, float]:\n", " label = Xy.get_label()\n", " r = np.zeros(predt.shape)\n", " predt = 1.0 / (1.0 + np.exp(-predt))\n", " gt = predt > 0.5\n", " r[gt] = 1 - label[gt]\n", " le = predt <= 0.5\n", " r[le] = label[le]\n", " return \"CustomErr\", float(np.average(r))\n", "\n", " # Use early stopping with custom objective and metric.\n", " early_stopping_rounds = 5\n", " # Specify the metric we want to use for early stopping.\n", " es = xgb.callback.EarlyStopping(\n", " rounds=early_stopping_rounds, save_best=True, metric_name=\"CustomErr\"\n", " )\n", "\n", " Xy = dxgb.DaskDeviceQuantileDMatrix(client, X, y)\n", " Xy_valid = dxgb.DaskDMatrix(client, X_valid, y_valid)\n", " booster = dxgb.train(\n", " client,\n", " {\"eval_metric\": \"error\", \"tree_method\": \"gpu_hist\"},\n", " Xy,\n", " evals=[(Xy_valid, \"Valid\")],\n", " num_boost_round=1000,\n", " obj=logit, # pass the custom objective\n", " feval=error, # pass the custom metric\n", " callbacks=[es],\n", " )[\"booster\"]\n", " return booster" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "2fa0482c-319a-412a-a156-e03fc0186982", "showTitle": false, "title": "" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/databricks/python/lib/python3.10/site-packages/xgboost/dask.py:703: FutureWarning: Please use `DaskQuantileDMatrix` instead.\n", " warnings.warn(\"Please use `DaskQuantileDMatrix` instead.\", FutureWarning)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Out[18]: " ] } ], "source": [ "booster_custom = fit_model_customized_objective(\n", " client, X=X_train, y=y_train, X_valid=X_valid, y_valid=y_valid\n", ")\n", "booster_custom" ] }, { "cell_type": "markdown", "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "746097aa-6b0a-43b8-be1a-fe476206f057", "showTitle": false, "title": "" } }, "source": [ "## Running inference\n", "\n", "After some tuning, we arrive at the final model for performing inference on new data. \n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "4ad9abd0-10b2-46e5-8d57-3b163b421888", "showTitle": false, "title": "" } }, "outputs": [], "source": [ "def predict(client, model, X):\n", " predt = dxgb.predict(client, model, X)\n", " return predt" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "1bd1cdcc-5296-4671-b339-87862db38fde", "showTitle": false, "title": "" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Out[20]: 0 0.843650\n", "1 0.975618\n", "3 0.378462\n", "10 0.293985\n", "11 0.966303\n", "Name: 0, dtype: float32" ] } ], "source": [ "preds = predict(client, booster, X_train)\n", "preds.head()" ] }, { "cell_type": "markdown", "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "c000a28d-473b-487b-92c1-1f5965f1898b", "showTitle": false, "title": "" } }, "source": [ "## Clean up\n", "\n", "When finished, be sure to destroy your cluster to avoid incurring extra costs for idle resources.\n", "\n", "**Note** If you forget to destroy the cluster manually, it's important to note that Databricks clusters will automatically time out after a period (specified during cluster creation)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "application/vnd.databricks.v1+cell": { "cellMetadata": { "byteLimit": 2048000, "rowLimit": 10000 }, "inputWidgets": {}, "nuid": "191ca13c-1678-4b9e-9235-6ef2b4f1dbb9", "showTitle": false, "title": "" } }, "outputs": [], "source": [ "client.close()" ] } ], "metadata": { "application/vnd.databricks.v1+notebook": { "dashboards": [], "language": "python", "notebookMetadata": { "mostRecentlyExecutedCommandWithImplicitDF": { "commandId": -1, "dataframes": [ "_sqldf" ] }, "pythonIndentUnit": 2 }, "notebookName": "xgboost-dask", "widgets": {} }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.8" } }, "nbformat": 4, "nbformat_minor": 4 }