Scaling Point-in-Time Language Models
arXiv cs.CL 18 hours ago
Researchers developed point-in-time language models trained only on text available up to specific calendar dates to eliminate lookahead bias for financial and social science applications. Models with up to 4 billion parameters trained on 1 trillion chronologically filtered tokens achieved performance approaching comparable unrestricted models like Gemma-3-4B and LLaMA-7B across common reasoning and language understanding benchmarks. The release of the training pipeline and dataset enables reproducible temporal validity research that was previously compromised by models inadvertently using future information.