Attention? Attention!
Lilian Weng
This article explains the attention mechanism in deep learning, which allows models to focus on relevant input elements when producing output by assigning importance weights to different parts of the input sequence. The attention mechanism was introduced by Bahdanau et al. in 2015 to address the limitation of fixed-length context vectors in seq2seq models for neural machine translation. The article then describes various attention variants including self-attention, soft/hard attention, and global/local attention, with their respective alignment score functions and applications in machine translation, computer vision, and image captioning.
Why it matters
[Updated on 2018-10-28: Add Pointer Network and the link to my implementation of Transformer.] [Updated on 2018-11-06: Add a link to the implementation of Transformer model.] [Updated on 2018-11-18: Add Neural Turing Machines.] [Updated on 2019-07-18: Correct the mistake on using the term “self-attention” when introducing the show-attention-tell paper; moved it to Self-Attention section.] [Updated on 2020-04-07: A follow-up post on improved Transformer models is here.]
