DiScoFormer: One transformer for density and score, across distributions

Allen Institute (AI2) · 2 weeks ago

Researchers introduced DiScoFormer, a transformer-based model that estimates both the probability density and score (gradient of log-density) of a distribution from data in a single forward pass without retraining. In 100 dimensions, DiScoFormer achieved 6.5x lower score error and 37x lower density error compared to kernel density estimation while continuing to improve with more samples. The model generalizes across different distribution types and could reduce computational costs for generative modeling, Bayesian inference, and scientific computing by serving as a reusable estimator.