DSGym: A holistic framework for evaluating and training data science agents
Together AI 6 months ago 19
Researchers introduced DSGym, a unified framework for evaluating and training data science agents that integrates diverse benchmarks behind a single API and includes 90 bioinformatics tasks and 92 Kaggle competitions. A 4-billion parameter model trained on 2,000 generated examples from the framework achieved competitive performance with much larger open-source models on general analysis tasks while frontier models like Claude Sonnet 4.5 reached only 37% accuracy on multi-step reasoning tasks. The framework reveals that current models rely on memorization for general tasks and lack domain grounding for scientific problems, enabling more systematic development of reasoning-based data science agents.