DiScoFormer: One transformer for density and score, across distributions

Hugging Face Blog · 2 weeks ago

Researchers at Google DeepMind introduced DiScoFormer, a transformer model that estimates both the probability density and score (gradient of log-density) of a distribution from a set of data points in a single forward pass without retraining. In 100 dimensions, DiScoFormer reduces score estimation error by 6.5x and density estimation error by more than 37x compared to kernel density estimation while continuing to improve with more samples. The model enables a single pretrained estimator to work across different problems in generative modeling, Bayesian inference, and scientific computing, eliminating the need to retrain for each new distribution.