An Introduction to AI Secure LLM Safety Leaderboard

Hugging Face · 2 years ago · 13

Researchers released the LLM Safety Leaderboard, a benchmark tool that evaluates language models across eight trustworthiness dimensions including toxicity, bias, adversarial robustness, privacy, and fairness using automated red-teaming tests. The DecodingTrust framework tests models through 33 system prompts for toxicity, 24 demographic groups for bias assessment, five adversarial attack algorithms, and privacy attacks designed to extract sensitive information like email addresses. Model developers can now submit their systems for standardized evaluation, with results showing that no single LLM performs consistently across all safety dimensions and that GPT-4 exhibits greater vulnerabilities than GPT-3.5 in certain scenarios.