Reference engagement · chatbotBaseline suite
The generic chatbot baseline.
A retrieval-backed assistant answering product, pricing and policy questions on a public website. No named client: this is the baseline battery every conversational system runs before anything sector-specific is added.
1,240
adversarial prompts across 11 OWASP LLM categories
4.1% → 0.4%
unsupported claims after grounding fixes
9 days
artefact submission to signed Trust Proof
What we tested
Grounding and citation fidelity against the source corpus; prompt injection and jailbreak resistance; PII leakage from context; refusal behaviour on out-of-scope questions; tone and language consistency; latency under burst load.
Finding. The assistant answered policy questions from model memory when retrieval returned nothing, producing confident statements with no source — the failure mode behind most publicised chatbot incidents.
Result. Retrieval-empty answers now refuse and hand off; every answer carries a citation the user can open. Unsupported-claim rate fell below the 1% declared threshold and is re-tested on every release.
Suites: grounding · red team · robustness · loadS.AI.F 01 Validation
Bank · call centreVoice + LLM
A call centre where the AI listened, summarised and advised.
An AI assistant transcribing live customer calls, summarising them into the CRM, suggesting next-best actions and drafting the post-call note. Agents kept the decision; the model shaped what they saw.
6,800
recorded calls replayed through the pipeline
3 dialects
Gulf, Levantine and MSA Arabic plus English
11.2%
summary error rate on code-switched calls
What we tested
Speech-to-text accuracy by dialect, accent and line quality; summary faithfulness against human-written notes; next-best-action fairness across customer segments; PII redaction before storage; escalation behaviour on distress and complaint language; agent override tracking.
Finding. Transcription degraded sharply on code-switched Arabic–English calls, and the summariser silently dropped customer objections — so the CRM record showed agreement where the customer had refused. Vulnerability cues in caller speech never triggered escalation.
Result. Dialect-specific acoustic tuning, a faithfulness check that flags dropped objections for agent confirmation, and a vulnerability-detection rule routed to a human queue. Summary error rate on code-switched calls fell to 2.8%; complaint-handling evidence now satisfies the conduct requirement.
Suites: accuracy by cohort · fairness · grounding · redactionS.AI.F 01 + 02
Telecom · WhatsApp AI chatAgentic · transacting
A WhatsApp agent that could change a customer’s plan.
A WhatsApp Business assistant handling balance queries, plan changes, roaming add-ons and payment links for millions of subscribers — an agent with write access to billing, reached over a channel the operator does not control.
2,600
injection and social-engineering attempts
7
unauthorised actions reachable before fixes
100%
transactions now carrying a signed action log
What we tested
Identity binding between WhatsApp number and account holder; prompt injection via forwarded messages, media captions and quoted replies; authorisation limits on billing actions; idempotency of payment links; PDPL consent and retention on message history; behaviour after SIM swap or number recycling.
Finding. A forwarded message containing crafted instructions could make the agent apply a roaming bundle to the sender’s account without a confirmation step, and number recycling let a new owner read the previous subscriber’s chat history.
Result. Untrusted content is now isolated from instruction context, every billing action requires an out-of-band confirmation and writes a signed action log, and number-change events force a history purge and re-verification. The operator cleared its TDRA and PDPL review with the evidence pack unchanged.
Suites: red team · authorisation · privacy · agentic guardrailsS.AI.F 01 + 02 + 03
