All work
Case study · Tanla Platforms · My most important project
Wise Albert
A multi-agent intelligence system we launched with Indosat Ooredoo Hutchison. Internal teams and IOH stakeholders get insights, and act on them, by asking questions in plain language instead of raising data requests. It was also my first time building a native AI product.
- Role
- Senior AI PM, discovery to launch
- Timeline
- Oct 2025 – present
- Users
- CS, product, IOH network & analytics teams
- Stack
- LangGraph, LangSmith, Rill, ClickHouse, Neo4j
10 days → hoursQBR preparation; CS now reviews drafts instead of assembling them
2–3 days → ~0Dependence on the DB team; answers come inside the meeting
>90%Data accuracy at launch, with <10% hallucination
4.9sP50 latency, with <1% PII leakage
Wise Albert · redrawn illustration
Why did SDK activations drop this quarter?
Daily activations halved, 24K → 12K. The weekly impact summary now flags drops like this when they happen, not at the QBR.
SDK funnel · ClickHousesemantic layer · Rill✓ verifier passed
01Context
Wisely AI is our anti-scam and anti-spam platform at IOH in Indonesia. Over the same period it was protecting users, churn came down from 1.9% to 1.4%.
Coverage95M+subscribers covered
Scale15.6B+interactions processed
Protection45M · $715Musers protected, fraud loss averted
The product was working, but we couldn't prove its impact quickly or catch problems early.
02The problem
Working with teams internally and at IOH, I found problems on both sides:
DB dependenceEvery non-standard question took 2–3 days, so we couldn't answer IOH inside a meeting.
Manual QBRsThe CS team spent up to 10 days assembling each QBR by hand.
No monitoringDaily SDK activations halved, 24K → 12K, and nobody noticed until the QBR.
Siloed contextDiscussions and commitments were spread across mail, Jira, Confluence and dashboards, and updates to IOH got missed.
03Discovery & validation
Before finalising the architecture, I ran interviews and focus groups with IOH's network and data analytics teams and our own CS and product teams. I asked about past behaviour, not hypotheticals: "walk me through your last QBR prep," not "would you use an AI assistant."
Use cases surfaced from interviews and focus groups
16P0, each validated with the team that would use it
9
I RICE-scored them and mapped the top ones to the problems:
DB dependence2–3 days per question
Conversational dashboardPlain-language questions over governed metrics, answered in the meeting
Manual QBRsUp to 10 days each
QBR generatorDrafts the QBR with every number cited to its source; a person reviews it
No monitoringDrops found at the QBR
Weekly impact summaryAnomaly alerts push deviations like the SDK drop as they happen
Siloed contextMissed updates
Personal intelligence agentConnects Outlook, Jira and Confluence into a weekly priority digest and a tracker of open commitments to IOH
04Two prioritisation calls
Shared layer firstAll four features draw on the same metric definitions, so I built the shared metrics layer first. That made each later feature much cheaper.
Ship at 80%I shipped with the 80% of P0 metrics that were available, and ran the ARPU billing-data agreement with IOH in parallel instead of delaying the release.
05Architecture
ClickHouse for metrics, Neo4j for the knowledge graph, Rill as the semantic layer, and LangGraph agents traced in LangSmith, with governed web search for public benchmarks.
Three decisions mattered most:
Semantic layer instead of LLM-written SQL
Why
Numbers are computed from governed metric definitions, so they're exact.
Trade-off
Undefined metrics can't be answered. But "I can't answer that" is safer than giving a telco customer a wrong number.
RAG only for documents, never for metrics
Why
RAG covers mail, Jira and past QBRs; numbers come from the semantic layer. I chose RAG over fine-tuning because our facts change daily and every answer needs a citation.
Trade-off
Two retrieval paths to build and maintain, with routing that has to get the question type right.
Two models, routed by risk
Why
In our bake-off, no single model won everything: one was more accurate on numbers and hallucinated less, the other reasoned better. Numbers and anything touching PII go to the first, reasoning over aggregated data goes to the second, and a separate verifier checks every claim.
Trade-off
Extra latency from routing and verification.
06How I measured quality
The bake-off used a weighted scorecard across eight KRAs. The weights follow the risk profile of a decision-intelligence product: a wrong number costs more than a slow one.
Data accuracy25%
Response quality20%
Compliance & PII15%
Latency10%
Coverage of cuts10%
Visuals & insights10%
Cost (TCO)5%
Extensibility5%
07Impact
- QBR preparation went from ~10 days to a few hours. The CS team now reviews drafts instead of assembling them.
- DB-team dependency is close to zero. Questions that took 2–3 days are answered inside the meeting with IOH.
- Model quality at launch: over 90% data accuracy, under 10% hallucination, under 1% PII leakage, and P50 latency of 4.9 seconds.
- ~80% stickiness across 50 users at IOH, with task completion up from 70% to over 90%.
- Our own evaluation flagged web search and dashboard automation as gaps, and we've since shipped both.
Saying "I can't answer that" is safer than giving a telco customer a wrong number.
Designed and specified, not yet shipped: a Deep Research mode for long-form, multi-source analysis.
Next case study
Ace Club: loyalty for the players behind 80% of revenue