Better AI Code Comment Detector

The old AI comment classifier had a problem before it even started: it was trained with some private data, built on shaky ground, and couldn’t be shared. The new version fixes that by moving to public data and a cleaner setup. There’s also a browser demo, and anything typed into it stays in the browser, so people can try it without handing their text to the server.

The headline result is a balanced accuracy of 77%. On a cross-validated test set with an even mix of human and robot comments, the classifier got 73% of human inputs right and 80% of robot inputs right. The author also ran a smaller real-world set of non-synthetic comments, and that did better: 88% accuracy, with 89% precision and 86% recall. That’s the kind of twist that makes these projects more interesting than the usual AI demo. Real comments, at least in this case, were easier than the synthetic ones.

The interface leans into transparency. Instead of hiding the logic behind a single yes-or-no answer, users can click parts of the text and see which features are firing and how they affect the result. The classifier also returns a calibrated probability, so a high-confidence answer is supposed to mean something concrete. When it says 80% confidence or more, the false-positive risk drops to about 5%. Around 50% confidence, it’s basically guessing half the time.

Most of the work, though, was not glamorous. The dataset was supposed to cost about $30 if everything had gone smoothly. It didn’t. The author had to recollect it more than once after finding problems like subject-matter leakage, missed docstrings, unstripped comments in odd syntaxes, and human comments that were so short they taught the model the wrong lesson. The source of the data was permissively licensed or copyleft repositories, sampled from their latest 2021 commits, with human comments taken from real files and robot comments generated by llms on the same code.

Feature work was its own little slog. Evaluating every combination of 15 candidate features would have meant training over 30,000 classifiers, five-way cross-validation included, so the author used a more practical ranking approach instead. The result is a classifier built around stylistic clues like word lengths, word frequencies, function-word patterns, and character n-grams. It’s a very technical answer to a very weird question: can code comments be sorted into human and robot piles without cheating? This time, apparently, yes — at least well enough to make the demo worth poking at.