Environment deployment Notes

[TOC]

Setup dreamer builder for IGLU Minecraft task — setup notes

Advantages of using `pip install git`

check https://dev.to/fronkan/pip-install-a-git-repository-111b

This will not create dependencies on the local machine

`python -c` flag

execute the python code in command

e.g,

1
2
3
4


python -c '
import pprint
pprint.pprint(1)
'

set TF_XLA variable

TF_XLA_FLAGS=--tf_xla_auto_jit=2

use `python=3.7`

otherwise the environment will crash.

some strange bugs of the environment

if I directly follow the instructions from https://github.com/iglu-contest/iglu, I will install the iglu:0.1.0 version, which will crash

if I use pip install git+https://github.com/iglu-contest/iglu.git, I will download 0.2.2 version

but in the end the environment did not show up.

Update

the latest numpy require python=3.8, while the project requires python=3.7

In the end we update the python from 3.7 to 3.8

Update 2

The minerl-patched==0.1.0 has some issue on my device, following is the error message

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


Traceback (most recent call last):
  File "test_env.py", line 2, in <module>
    import minerl_patched
  File "/home/sukai/anaconda3/envs/iglu_env337/lib/python3.7/site-packages/minerl_patched-0.1.0-py3.7-linux-x86_64.egg/minerl_patched/__init__.py", line 7, in <module>
    import minerl_patched.herobraine.envs
  File "/home/sukai/anaconda3/envs/iglu_env337/lib/python3.7/site-packages/minerl_patched-0.1.0-py3.7-linux-x86_64.egg/minerl_patched/herobraine/envs.py", line 19, in <module>
    MINERL_TREECHOP_V0 = Treechop()
  File "/home/sukai/anaconda3/envs/iglu_env337/lib/python3.7/site-packages/minerl_patched-0.1.0-py3.7-linux-x86_64.egg/minerl_patched/herobraine/env_specs/treechop_specs.py", line 44, in __init__
    **kwargs)
  File "/home/sukai/anaconda3/envs/iglu_env337/lib/python3.7/site-packages/minerl_patched-0.1.0-py3.7-linux-x86_64.egg/minerl_patched/herobraine/env_specs/simple_embodiment.py", line 34, in __init__
    super().__init__(name, *args, **kwargs)
  File "/home/sukai/anaconda3/envs/iglu_env337/lib/python3.7/site-packages/minerl_patched-0.1.0-py3.7-linux-x86_64.egg/minerl_patched/herobraine/env_spec.py", line 38, in __init__
    self.reset()
  File "/home/sukai/anaconda3/envs/iglu_env337/lib/python3.7/site-packages/minerl_patched-0.1.0-py3.7-linux-x86_64.egg/minerl_patched/herobraine/env_spec.py", line 44, in reset
    self.actionables = self.create_actionables()
  File "/home/sukai/anaconda3/envs/iglu_env337/lib/python3.7/site-packages/minerl_patched-0.1.0-py3.7-linux-x86_64.egg/minerl_patched/herobraine/env_specs/simple_embodiment.py", line 47, in create_actionables
    handlers.KeybasedCommandAction(k, v) for k, v in INVERSE_KEYMAP.items()
  File "/home/sukai/anaconda3/envs/iglu_env337/lib/python3.7/site-packages/minerl_patched-0.1.0-py3.7-linux-x86_64.egg/minerl_patched/herobraine/env_specs/simple_embodiment.py", line 48, in <listcomp>
    if k in SIMPLE_KEYBOARD_ACTION
  File "/home/sukai/anaconda3/envs/iglu_env337/lib/python3.7/site-packages/minerl_patched-0.1.0-py3.7-linux-x86_64.egg/minerl_patched/herobraine/hero/handlers/agent/actions/keyboard.py", line 60, in __init__
    super().__init__(command, spaces.Discrete(len(keys) + 1))
  File "/home/sukai/anaconda3/envs/iglu_env337/lib/python3.7/site-packages/minerl_patched-0.1.0-py3.7-linux-x86_64.egg/minerl_patched/herobraine/hero/spaces.py", line 206, in __init__
    self.shape = ()
AttributeError: can't set attribute

However, if I install minerl==0.4.2.1 and run the sample codes on https://github.com/minerllabs/minerl , it can work.

minerl-patched==0.1.0 is the iglu version

after I installed minerl==0.4.2.1, the error message changed

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


(iglu_env437) ➜  iglu git:(master) ✗ python test/test_env.py   
/home/sukai/anaconda3/envs/iglu_env437/lib/python3.7/site-packages/gym/logger.py:34: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize("%s: %s" % ("WARN", msg % args), "yellow"))
/home/sukai/anaconda3/envs/iglu_env437/lib/python3.7/site-packages/gym/logger.py:34: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize("%s: %s" % ("WARN", msg % args), "yellow"))
action space: Dict(attack:Discrete(2), back:Discrete(2), camera:Box(low=-180.0, high=180.0, shape=(2,)), forward:Discrete(2), hotbar:Discrete(7), jump:Discrete(2), left:Discrete(2), right:Discrete(2), use:Discrete(2))
observation space: Dict(agentPos:Box(low=-180.0, high=180.0, shape=(5,)), chat:<iglu.handlers.String object at 0x7fb573945890>, compass:Dict(angle:Box(low=-180.0, high=180.0, shape=())), grid:Box(low=0.0, high=6.0, shape=(9, 11, 11)), inventory:Box(low=0.0, high=20.0, shape=(6,)), pov:Box(low=0, high=255, shape=(64, 64, 3)), ray:<iglu.handlers.String object at 0x7fb573c09f50>)
current task: None
/home/sukai/anaconda3/envs/iglu_env437/lib/python3.7/site-packages/gym/logger.py:34: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize("%s: %s" % ("WARN", msg % args), "yellow"))
/home/sukai/anaconda3/envs/iglu_env437/lib/python3.7/site-packages/gym/logger.py:34: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize("%s: %s" % ("WARN", msg % args), "yellow"))
/home/sukai/anaconda3/envs/iglu_env437/lib/python3.7/runpy.py:125: RuntimeWarning: 'minerl_patched.utils.process_watcher' found in sys.modules after import of package 'minerl_patched.utils', but prior to execution of 'minerl_patched.utils.process_watcher'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
72it [00:00, 125.10it/s]No pitch observation! Yielding default value 0.0 for pitch
No xpos observation! Yielding default value 0.0 for xpos
No yaw observation! Yielding default value 0.0 for yaw
No ypos observation! Yielding default value 0.0 for ypos
No zpos observation! Yielding default value 0.0 for zpos
Traceback (most recent call last):
  File "test/test_env.py", line 24, in <module>
    obs, reward, done, info = env.step(action)
  File "/home/sukai/anaconda3/envs/iglu_env437/lib/python3.7/site-packages/gym/wrappers/time_limit.py", line 18, in step
    observation, reward, done, info = self.env.step(action)
  File "/home/sukai/anaconda3/envs/iglu_env437/lib/python3.7/site-packages/iglu-0.2.2-py3.7.egg/iglu/env.py", line 185, in step
    obs, reward, done, info = super().step(action)
  File "/home/sukai/anaconda3/envs/iglu_env437/lib/python3.7/site-packages/minerl_patched-0.1.0-py3.7-linux-x86_64.egg/minerl_patched/env/_singleagent.py", line 32, in step
    obs, rew, done, info = super().step(multi_agent_action)
  File "/home/sukai/anaconda3/envs/iglu_env437/lib/python3.7/site-packages/minerl_patched-0.1.0-py3.7-linux-x86_64.egg/minerl_patched/env/_multiagent.py", line 308, in step
    out_obs, monitor = self._process_observation(actor_name, obs, _malmo_json)
  File "/home/sukai/anaconda3/envs/iglu_env437/lib/python3.7/site-packages/minerl_patched-0.1.0-py3.7-linux-x86_64.egg/minerl_patched/env/_multiagent.py", line 217, in _process_observation
    obs_dict[h.to_string()] = h.from_hero(info)
  File "/home/sukai/anaconda3/envs/iglu_env437/lib/python3.7/site-packages/minerl_patched-0.1.0-py3.7-linux-x86_64.egg/minerl_patched/herobraine/hero/handlers/translation.py", line 126, in from_hero
    for h in self.handlers
  File "/home/sukai/anaconda3/envs/iglu_env437/lib/python3.7/site-packages/minerl_patched-0.1.0-py3.7-linux-x86_64.egg/minerl_patched/herobraine/hero/handlers/translation.py", line 126, in <dictcomp>
    for h in self.handlers
  File "/home/sukai/anaconda3/envs/iglu_env437/lib/python3.7/site-packages/minerl_patched-0.1.0-py3.7-linux-x86_64.egg/minerl_patched/herobraine/hero/handlers/translation.py", line 91, in from_hero
    return self.walk_dict(hero_dict, self.hero_keys)
  File "/home/sukai/anaconda3/envs/iglu_env437/lib/python3.7/site-packages/minerl_patched-0.1.0-py3.7-linux-x86_64.egg/minerl_patched/herobraine/hero/handlers/translation.py", line 80, in walk_dict
    raise KeyError()
KeyError

Update 3

Interestingly, I can run the program in Docker container.

By looking at the Dockerfile

the only difference seems that the docker container uses a older Ubuntu system with cudnn7 rather than the latest cudnn8

1

FROM nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04

In order to find the bug, we change the line to

1

FROM nvidia/cuda:11.4.2-cudnn8-runtime-ubuntu20.04

Nope…

Final update

I don’t need to figure out what’s wrong now. I just use the docker version

git – how to remove uncommitted files

1
2


git clean -df
git checkout -- .

Setup dreamer builder for IGLU Minecraft task — setup notes#

Advantages of using pip install git#

python -c flag#

set TF_XLA variable#

use python=3.7#

some strange bugs of the environment#

git – how to remove uncommitted files#