SemaDiff: Identifying Semantic-Changing Commits with Generated Code and Tests

arXiv cs.AI · 18 hours ago

SemaDiff is a new method that uses large language models to generate tests and dependent code for identifying whether commits preserve program behavior or introduce changes. The approach achieved 76% accuracy and 100% precision in semantic-changing commit detection when evaluated on a manually annotated dataset of 183 commits from open-source Java projects. This enables better support for debugging, fault localization, bug dataset construction, and other software maintenance tasks that require distinguishing purely refactoring changes from behavior-altering modifications.