[TOC]

  1. Title: Diffusion-LM Improves Controllable Text Generation
  2. Author: Xiang Lisa Li
  3. Publish Year: May 2022
  4. Review Date: Mon, Nov 14, 2022

https://arxiv.org/pdf/2205.14217.pdf

Summary of paper

Motivation

  • can language tokens be represented as floating number?
  • they develop a new non-autoregressive language model based on continuous diffusion
  • Diffusion LM iteratively denoises as sequence of Gaussian vectors into word vectors, yielding a sequence of intermediate latent variable.
  • how to convert from continuous embeddings back to words
    • they used rounding and many other tricks to stabilise the training process

Contribution

  • they tried diffusion model for Language Model

Incomprehension

Not sure if the model is good at text generation.