Learning Word Embedding
Lilian Weng
Word embedding transforms text words into numeric vectors of lower dimensions than one-hot encoding, enabling machine learning models to process natural language efficiently. Skip-gram and CBOW are context-based approaches that learn embeddings by predicting words from context, with techniques like hierarchical softmax reducing computational complexity from O(V) to O(log V) during training. This foundational method enables models to capture semantic relationships between words and has become standard for natural language processing tasks.
Why it matters
Human vocabulary comes in free text. In order to make a machine learning model understand and process the natural language, we need to transform the free-text words into numeric values. One of the simplest transformation approaches is to do a one-hot encoding in which each distinct word stands for one dimension of the resulting vector and a binary value indicates whether the word presents (1) or not (0).
