Testing robustness against unforeseen adversaries
OpenAI Blog
Researchers created a method to measure how well neural networks resist adversarial attacks they haven't encountered before, introducing a metric called UAR (Unforeseen Attack Robustness). The metric evaluates a single model's performance against unanticipated attacks rather than only those seen during training. This work emphasizes the importance of testing AI systems across a broader spectrum of novel attack types to better understand their real-world vulnerability.
Why it matters
We’ve developed a method to assess whether a neural network classifier can reliably defend against adversarial attacks not seen during training. Our method yields a new metric, UAR (Unforeseen Attack Robustness), which evaluates the robustness of a single model against an unanticipated attack, and highlights the need to measure performance across a more diverse range of unforeseen attacks.