Efficient Table Pre-training without Real Data: An Introduction to TAPEX
Hugging Face Blog
Researchers developed TAPEX, a table pre-training method that uses synthetic SQL queries and their execution results instead of real data to train language models for table question answering tasks. The approach achieved a 50-times speedup compared to previous table pre-training method TaBERT while using only 2% of the pre-training corpus, reaching state-of-the-art results on four benchmark datasets including WikiSQL (89.6% accuracy) and WikiTableQuestions (57.5% accuracy). This demonstrates that domain-specific pre-training via synthetic executable programs can be more efficient than general language modeling approaches that rely on large amounts of real textual data.
