The Challenge
Acme Bank had deployed AI models to assist with transaction monitoring and fraud detection. However, they faced growing concerns about model reliability:
- Regulatory pressure: MAS guidelines required explainability and continuous validation of AI systems
- Silent failures: Models occasionally missed obvious fraud patterns without any indication of degraded performance
- Audit requirements: Internal risk teams needed evidence that AI systems were performing as expected
The Solution
Acme Bank deployed Guardian to monitor their transaction AI systems in real-time.
Implementation
- API Integration: Connected Guardian to their existing ML pipeline with minimal code changes
- Baseline Establishment: Guardian automatically learned normal behavior patterns over a 2-week period
- Alert Configuration: Set up alerts for sandbagging detection, drift, and confidence anomalies
Key Capabilities Used
- Sandbagging Detection: Metacognitive probes identified when models were underperforming on specific transaction types
- Drift Monitoring: Automatic detection of behavior changes after model updates
- Compliance Dashboard: Audit-ready reports for regulatory submissions
The Results
Within 3 months of deployment:
- 96% detection accuracy for sandbagging behavior
- 40% reduction in undetected model degradation incidents
- Passed MAS AI governance audit with commendation
- 2 hours average time to detect and respond to model issues (down from 2 weeks)
What They Said
“Guardian gave us visibility into our AI systems that we didn’t know we were missing. The sandbagging detection alone has prevented several potential compliance issues.”
- Head of AI Risk, Acme Bank
Products Used
- Guardian: Real-time AI reliability monitoring
- Eval: Pre-deployment testing and validation