From GPT-2 to gpt-oss: Analyzing the Architectural Advances

Ahead of AI · 1 year ago · 32

OpenAI released gpt-oss-120b and gpt-oss-20b, their first open-weight models since GPT-2 in 2019, featuring standard transformer architecture with optimizations including RoPE positional embeddings, SwiGLU activation functions, and Mixture-of-Experts modules. The 20B model runs on 16GB consumer GPUs while the 120B model requires an H100 with 80GB RAM. The architectural changes from GPT-2 reflect iterative improvements across the industry in parameter efficiency and inference optimization rather than fundamental innovations.