Can a Language Model Learn Facts Continually in Its Weights?

arXiv cs.CL · 18 hours ago

Researchers tested whether language models can continually learn new facts by updating their weights, conducting experiments with Qwen3 models through sequences of 20 to 100 sequential writes using different training data types. Facts trained on diverse restatements retained 46% accuracy after 20 sequential writes compared to 1% for bare-statement training, but regardless of training approach, later weight updates caused interference that made earlier facts unreachable despite them being stored in log-probability. The findings show that while models can store knowledge in weights through broad training data, using context via prompts rather than relying on weight updates is the more reliable way to preserve and access multiple facts in continual learning scenarios.