Docker Notes

[TOC]

Setup dreamer builder for IGLU Minecraft task — Docker Session

The author suggested to use Docker environment.

HOWEVER, due to some unknown reason, my current ubuntu desktop cannot run nvidia-docker, and thus cannot connect to GPU device in the container, thus I decided that we do not deploy the model on docker!

The following are useful tips regarding to docker commands.

docker permission issue

in order to connect to docker sock

one can sudo chmod 666 /var/run/docker.sock

BTW: The storage location of Docker images and containers Ubuntu: /var/lib/docker/

NVIDIA docker

https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker

update docker’s tensorflow (not recommended)

https://www.tensorflow.org/install/source

when you find that your RTX 3000 series GPU cannot work on tensorflow 2.4.2, you can upgrade tensorflow to 2.7.0

pip install tensorflow==2.7.0

also apt install cuda-toolkit-11-2

The cuDNN should also be upgraded.

apt install --no-install-recommends libcudnn8=8.1* libcudnn8-dev=8.1*

Others: you might find out that the error message

Could not load dynamic library 'libcusolver.so.11'

so just create a soft link

1

sudo ln -s /usr/local/cuda-11.1/lib64/libcusolver.so.11 /usr/local/cuda-11.1/lib64/libcusolver.so.10

Note: not recommended because it cause the container unstable and there is just a lot of bugs there.

docker setup

docker build from Dockerfile

https://docs.docker.com/engine/reference/commandline/build/

-t means you set a name for it with version tag

1

docker build - < Dockerfile -t minerl:0.1

docker check image

just use docker image ls

docker remove image

docker image rm

use docker container ls -a to check the container cache

it will remove docker image (but other necessary dependencies will remain)

docker run image

The difference between “docker run” and “docker exec” is that “docker exec” executes a command on a running container. On the other hand, “docker run” creates a temporary container, executes the command in it and stops the container when it is done.

so for this one, we use docker run -t -i -d minerl:0.1

after you create a container, remember to check its id and re-enter this container rather than create a new one.

use docker exec -it 0b0afaedba4a bash

docker check running process

docker container ls

docker mount local repo to container

use -v your/local/path:/root/container/path

e.g.,

1

docker run --network host --rm -it -v $(pwd):/root/iglu_dev docker-image-name command parameters

docker network

https://www.youtube.com/watch?v=c6Ord0GAOp8

there are five different types of network that docker provides

• bridge
  • private default network created on the host
  • containers linked to this network have an internal IP address
  • the docker server create virtual ethernet bridge docker0
• host
  • public network
  • Utilise the host’s IP address and TCP port space in order to display the services running inside container
• none
  • just do not let the container connect to the network
• overlay
  • utilised for creating internal private network to the Docker nodes in the docker swarm cluster
• macvlan
  • this network assigns a MAC address to docker container. With this Mac address, the Docker server routes the network traffic to a router

docker network inspect bridge to check the network

docker attach vs docker exec -it

if you exit the shell from docker attach, the whole running container may just stop.

So use docker exec rather than docker attach

avoid interaction during docker build

example dockerfile

1
2
3

FROM UBUNTU
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=Australia/Sydney

ENV is for future running containers. ARG for building your Docker image.

conda env check

conda env list