Faster Text Generation with TensorFlow and XLA

Hugging Face · 4 years ago · 42

TensorFlow text generation compiled with XLA now runs up to 100x faster than the uncompiled version, with performance exceeding PyTorch equivalents on GPU or TPU hardware. The optimization requires a single line of code (`jit_compile=True`) but introduces a tradeoff where the first call with a new tensor shape or type triggers slow recompilation, while subsequent calls with identical shapes reuse the compiled binary. Users must pad inputs to consistent lengths and accept initial overhead when using different generation options to maintain the speed benefits.