Nvidia’s new DNA model learns what token prediction misses
The New Stack Amanda Caswell
Nvidia released JEPA-DNA, a genomic AI model that learns DNA patterns beyond just guessing missing letters. It mixes two training methods instead of relying only on text-style prediction, a shift researchers like Yann LeCun have pushed for.
Based on reporting by The New Stack, Amanda Caswell — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Nvidia just put a new genomic model on Hugging Face, and it's a quiet departure from how most AI in this space has worked. Called JEPA-DNA, the model doesn't just do what nearly every language model does — mask some tokens and train the network to guess them back. Instead it adds a second objective: predicting the functional representation of a masked DNA segment in latent space, not the literal sequence of letters. Token prediction is still in there, but it's no longer running the whole show.
That distinction matters more than it sounds. Standard genomic base models have basically copied the NLP playbook, using masked language modeling to teach a model the local "syntax" of a DNA sequence. It's good at that. But it tends to miss the bigger functional picture — the wider "meaning" of what a stretch of DNA actually does. JEPA-DNA is built to supervise the model's global sequence embedding directly, which Nvidia argues produces representations that are easier to use for downstream biological work without giving up the generative side of things.
The released checkpoint, JEPA-DNA-DNABERT2, isn't a standalone model built from scratch. It's a continual pre-training framework layered on top of DNABERT-2, the 117 million-parameter model originally developed by Zhihan Zhou and collaborators. Nvidia's contribution is the training recipe that lets the architecture learn from both token-level prediction and latent-space representation at once, and the framework is designed to be model-agnostic, meaning it could in principle attach to other base architectures down the line.
Nvidia is positioning this strictly as a research tool. It's released globally, but for non-commercial use only, aimed at things like feature extraction, linear probing, continual pre-training experiments, and zero-shot scoring of DNA sequence changes. Nvidia is explicit that it is not a diagnostic tool and has no clinical validation, so nobody should be reading this as a medical product in waiting.
What's more interesting than the checkpoint itself is the philosophy behind it. Yann LeCun, now Executive Chairman of AMI Labs and formerly Meta's Chief AI Scientist, has spent years arguing that predictive architectures — models that learn structure in latent space rather than reconstructing raw tokens — deserve to be a general alternative to next-token prediction. JEPA-DNA is one of the more concrete signs that this argument is migrating out of language research and into biology, where DNA's patterns clearly hold relationships that plain sequence prediction alone doesn't capture.
My take — AI-written commentary, not fact-checked reporting
Genomics has spent years borrowing NLP's homework, and it shows: masked-token prediction is great at local grammar and mediocre at global meaning. Nvidia leaning into a JEPA-style objective for DNA is a sign that the field is finally admitting text-generation tricks don't automatically transfer to biology. Credit to LeCun for pushing this idea long before it was fashionable — and credit to Nvidia for shipping it as an honest research tool rather than dressing it up as something clinically ready, which it explicitly is not.
Read more about this at: The New Stack
