StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows

arXiv cs.AI · 6 hours ago

StructureClaw is a workbench and executable benchmark for evaluating LLM agents on structural engineering tasks, requiring agents to produce complete artifact chains including models, validation records, and reports rather than just final answers. The benchmark contains 150 scenarios with performance rising from 56.8% success rate with baseline agents to 88.6% with the full automatic workflow across ten model configurations. The artifact-centered evaluation approach reveals workflow-level failures that standard metrics miss, particularly in handling invalid numerical inputs and reconstructing structural models consistently.