{ "cells": [ { "cell_type": "markdown", "metadata": { "tags": [ "cloud/aws/sagemaker", "workflow/hpo", "library/cudf", "library/cuml", "library/scikit-learn", "data-format/csv", "data-storage/s3" ] }, "source": [ "# Running RAPIDS Hyperparameter Experiments at Scale on Amazon SageMaker" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "_January, 2023_\n", "\n", "### Import packages and create Amazon SageMaker and Boto3 sessions" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "import time\n", "\n", "import boto3\n", "import sagemaker" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "tags": [] }, "outputs": [], "source": [ "execution_role = sagemaker.get_execution_role()\n", "session = sagemaker.Session()\n", "\n", "region = boto3.Session().region_name\n", "account = boto3.client(\"sts\").get_caller_identity().get(\"Account\")" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "('561241433344', 'us-east-2')" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "account, region" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Upload the higgs-boson dataset to s3 bucket" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "!mkdir -p ./dataset\n", "!if [ ! -f \"dataset/HIGGS.csv\" ]; then wget -P dataset https://archive.ics.uci.edu/ml/machine-learning-databases/00280/HIGGS.csv.gz; fi\n", "!if [ ! -f \"dataset/HIGGS.csv\" ]; then gunzip dataset/HIGGS.csv.gz; fi" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "tags": [] }, "outputs": [], "source": [ "s3_data_dir = session.upload_data(path=\"dataset\", key_prefix=\"dataset/higgs-dataset\")" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "'s3://sagemaker-us-east-2-561241433344/dataset/higgs-dataset'" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s3_data_dir" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "### Download latest RAPIDS container from DockerHub\n", "\n", "To build our RAPIDS Docker container compatible with Amazon SageMaker, you’ll start with base RAPIDS container, which the nice people at NVIDIA have already built and pushed to [DockerHub](https://hub.docker.com/r/rapidsai/base/tags).\n", "\n", "You will need to extend this container by creating a Dockerfile, copying the training script and installing [SageMaker Training toolkit](https://github.com/aws/sagemaker-training-toolkit) to makes RAPIDS compatible with SageMaker " ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "tags": [] }, "outputs": [], "source": [ "estimator_info = {\n", " \"rapids_container\": \"rapidsai/base:26.08a-cuda13-py3.13\",\n", " \"ecr_image\": \"sagemaker-rapids-higgs:latest\",\n", " \"ecr_repository\": \"sagemaker-rapids-higgs\",\n", "}" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "%%time\n", "!docker pull {estimator_info['rapids_container']}" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ARG RAPIDS_IMAGE\n", "\n", "FROM $RAPIDS_IMAGE as rapids\n", "\n", "# Installs a few more dependencies\n", "RUN conda install --yes -n base \\\n", " cupy \\\n", " flask \\\n", " protobuf \\\n", " 'sagemaker-python-sdk>=2.239.0'\n", "\n", "# Copies the training code inside the container\n", "COPY rapids-higgs.py /opt/ml/code/rapids-higgs.py\n", "\n", "# Defines rapids-higgs.py as script entry point\n", "# ref: https://docs.aws.amazon.com/sagemaker/latest/dg/adapt-training-container.html\n", "ENV SAGEMAKER_PROGRAM rapids-higgs.py\n", "\n", "# override entrypoint from the base image with one that accepts\n", "# 'train' and 'serve' (as SageMaker expects to provide)\n", "COPY entrypoint.sh /opt/entrypoint.sh\n", "ENTRYPOINT [\"/opt/entrypoint.sh\"]\n" ] } ], "source": [ "!cat Dockerfile" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Sending build context to Docker daemon 7.68kB\n", "Step 1/7 : ARG RAPIDS_IMAGE\n", "Step 2/7 : FROM $RAPIDS_IMAGE as rapids\n", " ---> a80bdce0d796\n", "Step 3/7 : RUN conda install --yes -n base cupy flask protobuf sagemaker\n", " ---> Running in f6522ce9b303\n", "Channels:\n", " - rapidsai-nightly\n", " - dask/label/dev\n", " - pytorch\n", " - conda-forge\n", " - nvidia\n", "Platform: linux-64\n", "Collecting package metadata (repodata.json): ...working... done\n", "Solving environment: ...working... done\n", "\n", "## Package Plan ##\n", "\n", " environment location: /opt/conda\n", "\n", " added / updated specs:\n", " - cupy\n", " - flask\n", " - protobuf\n", " - sagemaker\n", "\n", "\n", "The following packages will be downloaded:\n", "\n", " package | build\n", " ---------------------------|-----------------\n", " blinker-1.8.2 | pyhd8ed1ab_0 14 KB conda-forge\n", " boto3-1.34.118 | pyhd8ed1ab_0 78 KB conda-forge\n", " botocore-1.34.118 |pyge310_1234567_0 6.8 MB conda-forge\n", " dill-0.3.8 | pyhd8ed1ab_0 86 KB conda-forge\n", " flask-3.0.3 | pyhd8ed1ab_0 79 KB conda-forge\n", " google-pasta-0.2.0 | pyh8c360ce_0 42 KB conda-forge\n", " itsdangerous-2.2.0 | pyhd8ed1ab_0 19 KB conda-forge\n", " jmespath-1.0.1 | pyhd8ed1ab_0 21 KB conda-forge\n", " multiprocess-0.70.16 | py310h2372a71_0 238 KB conda-forge\n", " openssl-3.3.1 | h4ab18f5_0 2.8 MB conda-forge\n", " pathos-0.3.2 | pyhd8ed1ab_1 52 KB conda-forge\n", " pox-0.3.4 | pyhd8ed1ab_0 26 KB conda-forge\n", " ppft-1.7.6.8 | pyhd8ed1ab_0 33 KB conda-forge\n", " protobuf-4.25.3 | py310ha8c1f0e_0 325 KB conda-forge\n", " protobuf3-to-dict-0.1.5 | py310hff52083_8 14 KB conda-forge\n", " s3transfer-0.10.1 | pyhd8ed1ab_0 61 KB conda-forge\n", " sagemaker-2.75.1 | pyhd8ed1ab_0 377 KB conda-forge\n", " smdebug-rulesconfig-1.0.1 | pyhd3deb0d_1 20 KB conda-forge\n", " werkzeug-3.0.3 | pyhd8ed1ab_0 237 KB conda-forge\n", " ------------------------------------------------------------\n", " Total: 11.2 MB\n", "\n", "The following NEW packages will be INSTALLED:\n", "\n", " blinker conda-forge/noarch::blinker-1.8.2-pyhd8ed1ab_0 \n", " boto3 conda-forge/noarch::boto3-1.34.118-pyhd8ed1ab_0 \n", " botocore conda-forge/noarch::botocore-1.34.118-pyge310_1234567_0 \n", " dill conda-forge/noarch::dill-0.3.8-pyhd8ed1ab_0 \n", " flask conda-forge/noarch::flask-3.0.3-pyhd8ed1ab_0 \n", " google-pasta conda-forge/noarch::google-pasta-0.2.0-pyh8c360ce_0 \n", " itsdangerous conda-forge/noarch::itsdangerous-2.2.0-pyhd8ed1ab_0 \n", " jmespath conda-forge/noarch::jmespath-1.0.1-pyhd8ed1ab_0 \n", " multiprocess conda-forge/linux-64::multiprocess-0.70.16-py310h2372a71_0 \n", " pathos conda-forge/noarch::pathos-0.3.2-pyhd8ed1ab_1 \n", " pox conda-forge/noarch::pox-0.3.4-pyhd8ed1ab_0 \n", " ppft conda-forge/noarch::ppft-1.7.6.8-pyhd8ed1ab_0 \n", " protobuf conda-forge/linux-64::protobuf-4.25.3-py310ha8c1f0e_0 \n", " protobuf3-to-dict conda-forge/linux-64::protobuf3-to-dict-0.1.5-py310hff52083_8 \n", " s3transfer conda-forge/noarch::s3transfer-0.10.1-pyhd8ed1ab_0 \n", " sagemaker conda-forge/noarch::sagemaker-2.75.1-pyhd8ed1ab_0 \n", " smdebug-rulesconf~ conda-forge/noarch::smdebug-rulesconfig-1.0.1-pyhd3deb0d_1 \n", " werkzeug conda-forge/noarch::werkzeug-3.0.3-pyhd8ed1ab_0 \n", "\n", "The following packages will be UPDATED:\n", "\n", " openssl 3.3.0-h4ab18f5_3 --> 3.3.1-h4ab18f5_0 \n", "\n", "\n", "\n", "Downloading and Extracting Packages: ...working... done\n", "Preparing transaction: ...working... done\n", "Verifying transaction: ...working... done\n", "Executing transaction: ...working... done\n", "Removing intermediate container f6522ce9b303\n", " ---> 883c682b36bc\n", "Step 4/7 : COPY rapids-higgs.py /opt/ml/code/rapids-higgs.py\n", " ---> 2f6b3e0bec44\n", "Step 5/7 : ENV SAGEMAKER_PROGRAM rapids-higgs.py\n", " ---> Running in df524941c02e\n", "Removing intermediate container df524941c02e\n", " ---> 4cf437176c8c\n", "Step 6/7 : COPY entrypoint.sh /opt/entrypoint.sh\n", " ---> 32d95ff5bd74\n", "Step 7/7 : ENTRYPOINT [\"/opt/entrypoint.sh\"]\n", " ---> Running in c396fa9e98ad\n", "Removing intermediate container c396fa9e98ad\n", " ---> 39f900bfeba0\n", "Successfully built 39f900bfeba0\n", "Successfully tagged sagemaker-rapids-higgs:latest\n" ] } ], "source": [ "!docker build -t {estimator_info['ecr_image']} --build-arg RAPIDS_IMAGE={estimator_info['rapids_container']} ." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "!docker images" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Publish to Elastic Container Registry" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When running a large-scale training job either for distributed training or for independent experiments, you will need to make sure that datasets and training scripts are all replicated at each instance in your cluster. Thankfully, the more painful of the two — moving datasets — is taken care of by Amazon SageMaker. As for the training code, you already have a Docker container ready, you simply need to push it to a container registry, and Amazon SageMaker will then pull it into each of the training compute instances in the cluster. \n", "\n", "Note: SageMaker does not support using training images from private docker registry (ie. DockerHub), so we need to push\n", "the SageMaker-compatible RAPIDS container to the Amazon Elastic Container Registry (Amazon ECR) to store your Amazon SageMaker compatible RAPIDS container and make it available for Amazon SageMaker." ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "tags": [] }, "outputs": [], "source": [ "ECR_container_fullname = (\n", " f\"{account}.dkr.ecr.{region}.amazonaws.com/{estimator_info['ecr_image']}\"\n", ")" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "'561241433344.dkr.ecr.us-east-2.amazonaws.com/sagemaker-rapids-higgs:latest'" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ECR_container_fullname" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "tags": [] }, "outputs": [], "source": [ "!docker tag {estimator_info['ecr_image']} {ECR_container_fullname}" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "source : sagemaker-rapids-higgs:latest\n", "destination : 561241433344.dkr.ecr.us-east-2.amazonaws.com/sagemaker-rapids-higgs:latest\n" ] } ], "source": [ "print(\n", " f\"source : {estimator_info['ecr_image']}\\n\"\n", " f\"destination : {ECR_container_fullname}\"\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "!aws ecr create-repository --repository-name {estimator_info['ecr_repository']}\n", "!$(aws ecr get-login --no-include-email --region {region})" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The push refers to repository [561241433344.dkr.ecr.us-east-2.amazonaws.com/sagemaker-rapids-higgs]\n", "\n", "\u001b[1B3be3c6f4: Preparing \n", "\u001b[1Ba7112765: Preparing \n", "\u001b[1B5c05c772: Preparing \n", "\u001b[1Bbdce5066: Preparing \n", "\u001b[1B923ec1b3: Preparing \n", "\u001b[1B3fcfb3d4: Preparing \n", "\u001b[1Bbf18a086: Preparing \n", "\u001b[1Bf3ff1008: Preparing \n", "\u001b[1Bb6fb91b8: Preparing \n", "\u001b[1B7bf1eb99: Preparing \n", "\u001b[1B264186e1: Preparing \n", "\u001b[1B7d7711e0: Preparing \n", "\u001b[1Bee96f292: Preparing \n", "\u001b[1Be2a80b3f: Preparing \n", "\u001b[1B0a873d7a: Preparing \n", "\u001b[1Bbcc60d01: Preparing \n", "\u001b[1B1dcee623: Preparing \n", "\u001b[1B9a46b795: Preparing \n", "\u001b[1B5e83c163: Preparing \n", "\u001b[18Bc05c772: Pushed 643.1MB/637.1MB9A\u001b[2K\u001b[18A\u001b[2K\u001b[10A\u001b[2K\u001b[9A\u001b[2K\u001b[7A\u001b[2K\u001b[2A\u001b[2K\u001b[1A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2Klatest: digest: sha256:c8172a0ad30cd39b091f5fc3f3cde922ceabb103d0a0ec90beb1a5c4c9c6c97c size: 4504\n" ] } ], "source": [ "!docker push {ECR_container_fullname}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Testing your Amazon SageMaker compatible RAPIDS container locally\n", "\n", "Before you go off and spend time and money on running a large experiment on a large cluster, you should run a local Amazon SageMaker training job to ensure the container performs as expected. Make sure you have [SageMaker SDK](https://github.com/aws/sagemaker-python-sdk#installing-the-sagemaker-python-sdk) installed on your local machine.\n", "\n", "Define some default hyperparameters. Take your best guess, you can find the full list of RandomForest hyperparameters on the [cuML docs](https://docs.rapids.ai/api/cuml/~~~rapids_api_docs_version~~~/api.html#random-forest) page." ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "tags": [] }, "outputs": [], "source": [ "hyperparams = {\n", " \"n_estimators\": 15,\n", " \"max_depth\": 5,\n", " \"n_bins\": 8,\n", " \"split_criterion\": 0, # GINI:0, ENTROPY:1\n", " \"bootstrap\": 0, # true: sample with replacement, false: sample without replacement\n", " \"max_leaves\": -1, # unlimited leaves\n", " \"max_features\": 0.2,\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, specify the instance type as `local_gpu`. This assumes that you have a GPU locally. If you don’t have a local GPU, you can test this on a Amazon SageMaker managed GPU instance — simply replace `local_gpu` with with a `p3` or `p2` GPU instance by updating the `instance_type` variable." ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker.estimator import Estimator\n", "\n", "rapids_estimator = Estimator(\n", " image_uri=ECR_container_fullname,\n", " role=execution_role,\n", " instance_count=1,\n", " instance_type=\"ml.p3.2xlarge\", #'local_gpu'\n", " max_run=60 * 60 * 24,\n", " max_wait=(60 * 60 * 24) + 1,\n", " use_spot_instances=True,\n", " hyperparameters=hyperparams,\n", " metric_definitions=[{\"Name\": \"test_acc\", \"Regex\": \"test_acc: ([0-9\\\\.]+)\"}],\n", ")" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "INFO:sagemaker:Creating training-job with name: sagemaker-rapids-higgs-2024-06-05-02-14-30-371\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2024-06-05 02:14:30 Starting - Starting the training job...\n", "2024-06-05 02:14:54 Starting - Preparing the instances for training...\n", "2024-06-05 02:15:26 Downloading - Downloading input data..................\n", "2024-06-05 02:18:16 Downloading - Downloading the training image...\n", "2024-06-05 02:18:47 Training - Training image download completed. Training in progress...\u001b[34m@ entrypoint -> launching training script \u001b[0m\n", "\n", "2024-06-05 02:19:27 Uploading - Uploading generated training model\u001b[34mtest_acc: 0.7133834362030029\u001b[0m\n", "\n", "2024-06-05 02:19:35 Completed - Training job completed\n", "Training seconds: 249\n", "Billable seconds: 78\n", "Managed Spot Training savings: 68.7%\n", "CPU times: user 793 ms, sys: 29.8 ms, total: 823 ms\n", "Wall time: 5min 43s\n" ] } ], "source": [ "%%time\n", "rapids_estimator.fit(inputs=s3_data_dir)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Congrats, you successfully trained your Random Forest model on the HIGGS dataset using an Amazon SageMaker compatible RAPIDS container. Now you are ready to run experiments on a cluster to try out different hyperparameters and options in parallel." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Define hyperparameter ranges and run a large-scale search experiment\n", "There’s not a whole lot of code changes required to go from local training to training at scale. First, rather than define a fixed set of hyperparameters, you’ll define a range using the SageMaker SDK:" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker.tuner import (\n", " CategoricalParameter,\n", " ContinuousParameter,\n", " HyperparameterTuner,\n", " IntegerParameter,\n", ")\n", "\n", "hyperparameter_ranges = {\n", " \"n_estimators\": IntegerParameter(10, 200),\n", " \"max_depth\": IntegerParameter(1, 22),\n", " \"n_bins\": IntegerParameter(5, 24),\n", " \"split_criterion\": CategoricalParameter([0, 1]),\n", " \"bootstrap\": CategoricalParameter([True, False]),\n", " \"max_features\": ContinuousParameter(0.01, 0.5),\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, you’ll change the instance type to the actual GPU instance you want to train on in the cloud. Here you’ll choose an Amazon SageMaker compute instance with 4 NVIDIA Tesla V100 based GPU instance — `ml.p3.8xlarge`. If you have a training script that can leverage multiple GPUs, you can choose up to 8 GPUs per instance for faster training." ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "tags": [] }, "outputs": [], "source": [ "from sagemaker.estimator import Estimator\n", "\n", "rapids_estimator = Estimator(\n", " image_uri=ECR_container_fullname,\n", " role=execution_role,\n", " instance_count=2,\n", " instance_type=\"ml.p3.8xlarge\",\n", " max_run=60 * 60 * 24,\n", " max_wait=(60 * 60 * 24) + 1,\n", " use_spot_instances=True,\n", " hyperparameters=hyperparams,\n", " metric_definitions=[{\"Name\": \"test_acc\", \"Regex\": \"test_acc: ([0-9\\\\.]+)\"}],\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now you define a HyperparameterTuner object using the estimator you defined above." ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "tags": [] }, "outputs": [], "source": [ "tuner = HyperparameterTuner(\n", " rapids_estimator,\n", " objective_metric_name=\"test_acc\",\n", " hyperparameter_ranges=hyperparameter_ranges,\n", " strategy=\"Bayesian\",\n", " max_jobs=2,\n", " max_parallel_jobs=2,\n", " objective_type=\"Maximize\",\n", " metric_definitions=[{\"Name\": \"test_acc\", \"Regex\": \"test_acc: ([0-9\\\\.]+)\"}],\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "job_name = \"rapidsHPO\" + time.strftime(\"%Y-%m-%d-%H-%M-%S-%j\", time.gmtime())\n", "tuner.fit({\"dataset\": s3_data_dir}, job_name=job_name)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Clean up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Delete S3 buckets and files you don't need\n", "- Kill training jobs that you don't want running\n", "- Delete container images and the repository you just created" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "!aws ecr delete-repository --force --repository-name {estimator_info['ecr_repository']}" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.8" }, "vscode": { "interpreter": { "hash": "f7a54d993f849a0f97fda357a1a3bac7e25a43aff77e618e8d69a4ad36661dba" } } }, "nbformat": 4, "nbformat_minor": 4 }