# Dask Multi-GPU Guide

This guide demonstrates how to use cuML estimators in multi-GPU and multi-node contexts using Dask. While cuML’s single-GPU implementations are highly optimized, distributed computing with Dask enables you to:

- **Scale beyond single GPU memory**: Process datasets larger than what fits on a single GPU
- **Accelerate training**: Distribute computation across multiple GPUs for faster model training
- **Handle production workloads**: Deploy models that serve high-throughput prediction requests

cuML’s Dask integration uses a **One Process Per GPU (OPG)** architecture, where each Dask worker manages a single GPU. This design maximizes GPU utilization and simplifies memory management across the cluster.

## Setup and Configuration

### Installing Dependencies

To use cuML with Dask, you need to install additional dependencies. If you haven’t already, install them using conda or pip:

```bash
# Install with Conda:
conda install rapids-dask-dependency dask-cudf raft-dask

# Or install with pip (replace cu13 with your CUDA version):
pip install cuml-cu13[dask]
```

### Setting Up a CUDA Cluster

For single-node, multi-GPU execution, use `LocalCUDACluster` from `dask-cuda`. This automatically creates one worker per available GPU. For detailed information on configuring local CUDA clusters, including advanced networking options (UCX, InfiniBand) and multi-node cluster setup, see the [dask-cuda documentation](https://docs.nvidia.com/dask-cuda/latest/examples/ucx/).

```none
[1]:
```

## Example 1: K-Means Clustering

K-Means is one of the most commonly used clustering algorithms. cuML’s distributed implementation parallelizes the fit operation for each iteration, sharing only the centroids between iterations.

```none
[2]:
```

Now we can train a distributed K-Means model. The API is nearly identical to the single-GPU version:

```none
[3]:
```

### Convert into a single-GPU model

We can use the distributed model for inference directly (as shown above) or convert it back into a single-GPU version based on needs.

```none
[4]:
```

```none
[4]:
```

### Model Serialization

Distributed models cannot be pickled directly. For pickling we need to first extract the single-GPU version as shown above and can then serialize as usual:

> ⚠️ **Security Warning:** Only unpickle models from trusted sources. The `pickle` module is not secure – malicious pickle data can execute arbitrary code during deserialization. See the [Python pickle documentation](https://docs.python.org/3/library/pickle.html) for details.
```none
[5]:
```

The loaded model is a single-GPU estimator that can be used for inference. For distributed inference across a Dask cluster, consider using [Dask-ML’s ParallelPostFit](https://ml.dask.org/meta-estimators.html) meta-estimator.

## Example 2: Random Forest Classification

Random Forest is an ensemble learning method that builds multiple decision trees. cuML’s distributed implementation uses embarrassingly parallel training: for a forest with N trees trained by W workers, each worker builds N/W trees.

```none
[6]:
```

### Data Distribution Best Practices

For Random Forest, data distribution is critical for model accuracy:

- **Option 1: Well-shuffled data**: Distribute shuffled data evenly across workers (used above)
- **Option 2: Replicated data**: If memory allows, replicate the entire dataset to all workers for training that most closely matches single-GPU behavior

Both approaches ensure each worker sees a representative sample of the data distribution.

```none
[7]:
```

### Performance Notes

Random Forest training performance is heavily influenced by:

- `max_depth`: Lower values significantly speed up training but may reduce accuracy. Start with 12-16 for balanced performance.
- `n_estimators`: More trees improve accuracy but increase training time linearly. The work is distributed across workers.
- `n_bins`: Controls histogram granularity for split finding. Lower values (8-32) are faster but may miss optimal splits.

## Example 3: Linear Regression

Linear models in cuML support both Dask DataFrame and Dask Array inputs. This example demonstrates a distributed linear regression workflow.

```none
[8]:
```

```none
[9]:
```

### Using Dask DataFrames

You can also use Dask cuDF DataFrames as input, which is useful when loading data from files:

```none
[10]:
```

## Available Dask-Enabled Estimators

cuML provides Dask implementations for many popular algorithms including clustering (KMeans, DBSCAN), ensemble methods (Random Forest), linear models (LinearRegression, Logistic Regression, Ridge, Lasso), decomposition (PCA, TruncatedSVD), nearest neighbors, and more. All are available in the `cuml.dask` module.

For a comprehensive list of supported algorithms and their documentation, see the [Multi-Node, Multi-GPU Algorithms](../api/index.html.md#multi-node-multi-gpu-algorithms) section of the API reference.

## Performance Considerations and Best Practices

### When to Use Multi-GPU

Use distributed computing with Dask when:

- **Dataset exceeds single GPU memory**: Your data doesn’t fit on a single GPU
- **Training time is critical**: Multiple GPUs can accelerate model training
- **Scaling to production**: You need to handle high-throughput inference workloads

Single-GPU implementations are often sufficient and simpler for:

- **Small to medium datasets**: Data that comfortably fits in single GPU memory (typically < 80% of GPU RAM)
- **Rapid prototyping**: Simpler setup and debugging
- **Latency-sensitive inference**: Single-GPU inference has lower overhead

### Data Partitioning Strategy

The `n_parts` parameter controls how data is distributed across workers:

```python
# Rule of thumb: 2-4 partitions per worker
n_parts = n_workers * 2  # Good starting point

# More partitions: Better load balancing, more overhead
# Fewer partitions: Lower overhead, potential load imbalance
```

### Network Optimization

For multi-node clusters, high-performance networking options are available:

- **NVLink**: For multi-GPU communication on the same node
- **InfiniBand**: For fast inter-node communication
- **UCX protocol**: Unified communication framework for optimal performance

For detailed configuration examples and setup instructions, see the [dask-cuda documentation](https://docs.nvidia.com/dask-cuda/latest/examples/ucx/).

### Memory Management

- **RMM pool size**: Pre-allocate GPU memory to reduce allocation overhead
- **Worker memory limit**: Set limits to prevent out-of-memory errors
- **Partition size**: Keep partitions small enough to fit comfortably in GPU memory

### Performance Profiling

Monitor your distributed computation:

1. **Dask Dashboard**: Access at `client.dashboard_link` to visualize task execution
2. **NVIDIA tools**: Use `nvidia-smi` to monitor GPU utilization
3. **NVIDIA RMM**: Enable RMM logging for memory profiling

### Input Data Types

cuML’s Dask estimators accept multiple input formats:

- **Dask Array**: Use `dask.array` with CuPy backend for array operations
- **Dask DataFrame**: Use `dask_cudf.DataFrame` for structured data

Choose based on your workflow:

```python
# Dask Array - good for numerical operations
import dask.array as da
X = da.random.random((10000, 50), chunks=(1000, 50))

# Dask DataFrame - good for mixed types and data loading
import dask_cudf
df = dask_cudf.read_csv("data.csv")
```

## Clean Up

Optionally close your Dask client and cluster when finished to free up resources. Both the client and cluster will be shut down automatically when the notebook kernel process is shut down.

```none
[11]:
```

## Additional Resources

- **API Reference**: [Multi-Node, Multi-GPU Algorithms](../api/index.html.md#multi-node-multi-gpu-algorithms)
- **Cluster Setup**: [dask-cuda Examples (UCX, multi-node configuration)](https://docs.nvidia.com/dask-cuda/latest/examples/ucx/)
- **Dask Documentation**: [Dask Distributed](https://distributed.dask.org/)
- **Dask-CUDA**: [Dask-CUDA Documentation](https://docs.nvidia.com/dask-cuda/latest/)