Alex_petrekno Sample Factory Asynchronous Rl at Very High Fps 2020

[TOC]

1. Title: Sample Factory: Asynchronous Rl at Very High FPS
2. Author: Alex Petrenko
3. Publish Year: Oct, 2020
4. Review Date: Sun, Sep 25, 2022

Summary of paper

Motivation

Identifying performance bottlenecks

1. RL involves three workloads:

   1. environment simulation
   2. inference
   3. backpropagation
   • overall performance depends on the lowest workload
   • In existing methods (A2C/PPO/IMPALA) the computational workloads are dependent -> under-utilisation of the system resources.

2. Existing high-throughput methods focus on distributed training, therefore introducing a lot of overhead such as networking serialisation, etc.

   • e.g., (Ray & RLLib <==> Redis/Plasma, Seed RL <==> GRPC, Catalyst <==> Mongo DB)

Contribution

Some key terms

Double-buffered sampling

• with double-buffered approach, environments simulators never wait
• 

Resolving bottleneck # 2 (communication)

• RL training process is based on static data structures: allocate all memory in the beginning of training, and then only send pointers around (shared memory suits for single server)
• 

Good things about the paper (one paragraph)

Major comments

Minor comments

Incomprehension

Potential future work