Attention? Attention!

Lilian Weng · 8 years ago

This article explains the attention mechanism in deep learning, which allows models to focus on relevant input elements when producing output by assigning importance weights to different parts of the input sequence. The attention mechanism was introduced by Bahdanau et al. in 2015 to address the limitation of fixed-length context vectors in seq2seq models for neural machine translation. The article then describes various attention variants including self-attention, soft/hard attention, and global/local attention, with their respective alignment score functions and applications in machine translation, computer vision, and image captioning.