[TOC]
- Title: Diffusion-LM Improves Controllable Text Generation
- Author: Xiang Lisa Li
- Publish Year: May 2022
- Review Date: Mon, Nov 14, 2022
https://arxiv.org/pdf/2205.14217.pdf
Summary of paper
Motivation
- can language tokens be represented as floating number?
- they develop a new non-autoregressive language model based on continuous diffusion
- Diffusion LM iteratively denoises as sequence of Gaussian vectors into word vectors, yielding a sequence of intermediate latent variable.
- how to convert from continuous embeddings back to words
- they used rounding and many other tricks to stabilise the training process
Contribution
- they tried diffusion model for Language Model
Incomprehension
Not sure if the model is good at text generation.