Xiaotian_liu a Planning Based Neural Symbolic Approach for Embodied Instruction Following 2022

[TOC]

1. Title: A Planning Based Neural Symbolic Approach for Embodied Instruction Following
2. Author: Xiaotian Liu et. al.
3. Publish Year: 2022
4. Review Date: Thu, Feb 2, 2023
5. url: https://embodied-ai.org/papers/2022/15.pdf

Summary of paper

Motivation

• end-to-end deep learning methods struggle at these tasks due to long-horizon and sparse rewards.

Contribution

• Our main innovation relies on combining DL models for perception and NLP with a new egocentric planner based on successive planning problems formulated using the PDDL syntax, both for exploration and task accomplishment.
• our planning framework can naturally recover from action failures at any stage of the planned trajectory.

Some key terms

Embodied Instruction Following

• EIF requires an agent to process multimodal information and plan over long task horizons.
• however, end-to-end optimization leads to entangled latent state representation where compositional and long-horizon tasks are difficult to solve
• other approaches use neural networks to ground visual information into persistent memory structures to store information. These approaches rely on templates of existing tasks, making them difficult to generalize to new problems or unexpected action outcomes.

Classical planning in EIF

• however, classical planning assumes full observability,

Language module

• 
• for task goals, we extracted features directly from labels produced by FILM authors that are trained on multiple transformers
• 
  • limitation
  • the template may not cover all situations
  • does not utilise the detailed instructions.

Potential future work

there can be improvements on the language parser