Docker

Introduction

AIStore can be run as a cluster of Docker containers. There are two modes of operation: development and quick-start. They can be found in the deploy/dev/docker directory.

Development Mode

This mode is currently used for development purposes. All docker containers mount the same host’s AIStore source directory, and then execute from this single source. Upon restart (of the AIStore cluster), all changes made in the host will, therefore, take an immediate effect.

The development mode is currently being maintained and updated.

Quick Start Mode

Use this mode if you would like to set up a containerized, one-proxy, one-target deployment of AIStore within seconds. See quick start AIS with Docker. You just need Docker installed to quick-start AIS.

For an introduction to Docker, please watch Docker 101 youtube

Install Docker and Docker Compose

Note: Using Docker requires one of the following versions of Ubuntu:

• Bionic 18.04 (LTS)
• Xenial 16.04 (LTS)
• Trusty 14.04 (LTS)

1. Uninstall any old versions of docker:

1
$ sudo apt-get remove docker docker-engine docker.io

It’s OK if apt-get reports that none of these packages are installed.

2. Update the apt package index:

1
$ sudo apt-get update

3. Install packages to allow apt to use a repository over HTTPS:

1
$ sudo apt-get install \
2
    apt-transport-https \
3
    ca-certificates \
4
    curl \
5
    software-properties-common

4. Install Docker

1
$ sudo apt-get install docker-ce

Note: For version 16.04 and up, docker-ce is not in the default Ubuntu repository. See this guide for docker-ce installation.

5. Verify that Docker CE is installed correctly by running the “hello-world” image.

1
$ sudo docker run hello-world

6. Add your current user to the Docker group (but only if you are not the root). After executing the command, restart your machine for it to take effect.

1
$ sudo usermod -aG docker $(whoami)

7. Install Docker-Compose using python pip. Install pip if you don’t have it:

1
$ sudo apt-get install -y python-pip
2
$ sudo pip install docker-compose

OR

1
sudo apt install docker-compose

8. Test the installation:

1
$ docker-compose --version
2
docker-compose version 1.23.1, build 1719ceb

9. If you have any troubles with your installation, consider using the latest version of Docker and Docker-Compose.

Uninstall Docker and Docker Compose

1. To uninstall Docker, run the following:

1
$ sudo apt-get purge docker-ce
2
$ sudo apt-get purge docker-ce-cli

2. Ensure docker is completely uninstalled by running the following command:

1
$ dpkg -l | grep -i docker

There should be no docker-ce and docker-ce-cli packages listed.

3. To uninstall Docker-Compose, run the following:

1
$ pip uninstall docker-compose

4. Images, containers, volumes, or customized configuration files on your host are not automatically removed. To delete all images, containers, and volumes:

1
$ sudo rm -rf /var/lib/docker

Starting AIStore

If not already done, install Go and set $GOPATH environment variable.

The steps:

1. Build aisnode.

1
$ go install github.com/NVIDIA/aistore/cmd/aisnode@latest

2. To get all sources and use scripts, you can clone AIStore repo.

1
$ git clone https://github.com/NVIDIA/aistore.git

3. Optionally, setup AWS configuration by running aws configure command. To run AIStore Docker containers, you will need to pass your AWS config and credential directory via flag -a=<aws directory> CLI. By default, AWS stores config and credential files in ~/.aws/ Example:

1
$ ./deploy_docker.sh -a=~/.aws/

Building and running “containerized” AIStore is further documented at deploy/dev/docker. Please note: the image building process will take some time but only for the first time; subsequent runs will use already cached images.

Helpful docker commands

List Running Containers

List all of the running containers using docker ps. Many commands require the CONTAINER ID or NAME fields. Example output:

List All Containers

Lists all containers (not only the running ones).

1
$ docker ps -a

View Container Logs

To view docker logs, use docker logs <container_name>. Example:

1
$ docker logs ais0_proxy_1
2
I 21:23:56.400794 metasync.go:142] Starting metasyncer
3
I 21:24:06.415473 stats.go:422] {"err.n":0,"get.n":0,"del.n":0,"get.ns":0,"kalive.ns":0,"err.get.n":0,"err.list.n":0,"pst.n":0,"ren.n":0,
4
"lst.ns":0,"uptime.ns":0,"kalive.ns.max":0,"err.delete.n":0,"err.post.n":0,"err.range.n":0,"err.head.n":0,"put.n":0,"lst.n":0,
5
"kalive.ns.min":0,"err.put.n":0}
6
I 21:24:08.386182 proxy.go:2236] joined target 1463af8ddcd3 (num targets 1)
7
I 21:24:11.759453 proxy.go:2236] joined target 3c86e5e71978 (num targets 2)
8
I 21:24:12.411714 earlystart.go:262] Reached the expected 2/2 target registrations
9
I 21:24:12.412098 earlystart.go:228] 0e76b56086eb: merged local Smap (2/1)
10
I 21:24:12.412244 metasync.go:290] dosync: smaptag, action=early-start-have-registrations, version=2
11
I 21:24:12.412772 metasync.go:290] dosync: bucketmdtag, action=early-start-have-registrations, version=1
12
I 21:24:13.425617 earlystart.go:307] 0e76b56086eb: merging discovered Smap v2 (2, 1)
13
...

Note:

• You can obtain the container name by running command docker ps
• The deploy/dev/docker/ directory has a more comprehensive script named logs.sh to view logs

SSH Into a Container

1
$ docker exec -it CONTAINER_NAME /bin/bash

Note:

• In production mode, the logs are expected to be in /var/log/ais/.By default (development mode) the logs are under tmp/ais/log
• The deploy/dev/docker/ directory has a script named container_shell.sh that does the same thing

List Docker Images

1
$ docker image ls
2
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
3
ais1_proxy          latest              ced2cbd2ff2f        27 minutes ago      676MB
4
ais1_target         latest              ced2cbd2ff2f        27 minutes ago      676MB
5
ais0_proxy          latest              4c3fbcb54e4d        27 minutes ago      676MB
6
ais0_target         latest              4c3fbcb54e4d        27 minutes ago      676MB

List Docker Networks

1
$ docker network ls
2
NETWORK ID          NAME                    DRIVER              SCOPE
3
cb05b22edcb3        bridge                  bridge              local
4
86517bef938d        ais0_internal_control   bridge              local
5
51f49b56fe57        ais0_internal_data      bridge              local
6
2cee0651aa72        ais0_public             bridge              local
7
e11635deb15d        ais1_internal_control   bridge              local
8
9eefba13018f        ais1_internal_data      bridge              local
9
674fee80acac        ais1_public             bridge              local

Start a Container

1
$ docker start CONTAINER_NAME

Stop a Container

1
$ docker stop CONTAINER_NAME

Restart a Container

1
$ docker restart CONTAINER_NAME

Kill a Container

1
$ docker kill CONTAINER_NAME

View Resource Usage Statistics for all Containers

1
$ docker stats

Remove Unused Images

1
$ docker image prune -f

Remove all Stopped Containers

1
$ docker container prune -f

Remove all Unused Networks

1
$ docker network prune -f

Stop all Running Containers

1
$ docker stop $(docker ps -a -q)

Delete all Existing Containers

1
$ docker rm $(docker ps -a -q)

Delete all Existing Images

1
$ docker rmi $(docker images -q -a)

Docker playground

Following is a super-simple presentation to showcase some of the AIS capabilities.

In Docker playground, you can find the scripts to download different popular AI datasets (e.g., MNIST and ImageNet). The datasets are downloaded with the AIS-integrated Downloader that stores all downloaded objects directly into the AIStore.

During the download, you can monitor:

• number of requests made
• number of requests failed
• number of bytes transferred

In the example below, AIS downloads a handful of ImageNet images and collects/visualizes the corresponding statistics:

1
$ cd path_to/deploy/dev/docker
2
$ ./deploy_docker.sh -d=2 -p=2 -t=4 -c=1 -grafana -nocloud # start 2 proxies and 4 targets
3
$ ./playground/download_imagenet.sh # download some of ImageNet images into AIS and show stats
4
# once Downloader will save the files...
5
$ ./playground/stress_get.sh imagenet # do gets on saved files (requires jq command)
6
$ ./stop_docker.sh -l # stop docker