All case studies / Case study 07 / 12
Banque Orvelia
AI assistant evaluation platform
Proving an AI assistant is reliable before it goes live
In short
Before handing an AI assistant to its advisers, the bank wanted proof. The platform tests every version on hundreds of questions, measures how faithful answers are to their sources and blocks any release below the bar.
- Client
- Banque Orvelia
- Sector
- Retail banking
- Size
- 2,000 staff, 4 AI assistants in preparation
- −82%ungrounded answers from v1 to v3
- 600test questions replayed on every version
- Go / no-goautomatic before every release
The context
Banque Orvelia is preparing four internal assistants: procedures, lending, compliance and support. Risk management refuses any release without an objective measure of reliability: a made-up answer about a lending procedure can be costly.
The challenge
- 01
Checking thousands of answers by hand after every change is impossible.
- 02
Improving one thing can quietly break another.
- 03
Risk and IT must look at the same numbers.
- 04
Every release must be justified and archived.
Our approach
A reference set of questions, with correct answers approved by the business, is replayed automatically against every version of the assistant. An evaluator model (LLM-as-a-judge) scores each answer on faithfulness to sources, relevance and precision of the retrieved context. The dashboard compares versions, and a threshold blocks the release until quality is reached.
The solution, step by step
- 01
Reference questions
600 real questions with answers approved by business experts.
- 02
Replayed on every version
Every change to the assistant triggers a full evaluation, automatically.
- 03
Scored by an evaluator
Faithfulness, relevance, context precision: every answer is measured.
- 04
Compared version to version
You see at once what improves and what regresses.
- 05
Go or no-go
Release is only allowed above the thresholds agreed with risk.
Results
- −82%ungrounded answers from v1 to v3
- 0.94faithfulness score in production
- 2 hto evaluate a version, instead of two weeks
- 100%of releases documented
What we deliver
- Test sets built with the business
- Automated evaluation pipeline (RAGAS, LLM-as-a-judge)
- Version comparison dashboard
- Quality gates in CI/CD
- Archived reports for audit
Technologies
- RAGAS
- LangSmith
- Python
- FastAPI
- Docker
- GitLab CI
Timeline
6 to 8 weeks
Team
1 AI engineer, 1 MLOps engineer
Illustrative cases: clients and figures are fictional, representative of the solutions we deliver.