Fast and Reliable Detection of Silent Data Corruption in AI Accelerator Systems

Large-scale AI training and inference increasingly rely on specialized AI accelerators. A critical reliability threat in such systems is silent data corruption (SDC): an operator or training step completes without an exception, yet returns numerically incorrect tensor values. Unlike crashes or timeout failures, SDC may remain hidden until it appears as unstable loss curves, abnormal gradients, degraded model quality, or unexplained divergence. Detecting SDC is therefore a practical software engineering problem for accelerator qualification, deployment screening, regression testing, and incident diagnosis