An LLM can read a complaint history. The harder task is making that useful in a regulated business, where a missed detail can affect a customer’s outcome.
The work before the decision
A handler rarely starts with a clear account. Calls, case notes, repair records and earlier promises sit across several systems. Before making a decision, the handler has to reconstruct what happened.
The Complaints Sidekick retrieved evidence, built a chronology and produced a concise briefing. It also worked as a chatbot in Teams, so handlers could ask for more information as their investigation developed. The handler retained responsibility for the investigation and the outcome.
Led the business case, architecture, implementation and stakeholder engagement. Worked with Complaints, Assurance and senior stakeholders to move from proof of concept into a live trial.
Support the investigation. Keep the decision human.
- Retrieve Case records, transcripts, complaint guidance and SOPs
- Assemble Organise relevant information and retain source references
- Respond Prepare a briefing or answer a handler’s follow-up question
MLflow traces retrieval, context assembly and model output
Validate facts · investigate gaps · assess the complaint · determine redress
Build a briefing from evidence
Long histories contain duplicate notes, conflicting accounts and uneven transcript quality. Sending all of that to a model does not guarantee a useful answer.
I separated retrieval, context assembly and generation into explicit steps. LangGraph controlled what could be retrieved, when it could be retrieved and what reached the model. This made it possible to inspect each stage when a briefing was incomplete or wrong.
The briefing needed to preserve source references, conflicting accounts and gaps in the record. A transcript records what someone said; it does not establish that the event happened as described.
“The 12 March call transcript states that an engineer had visited earlier that day.”
Illustrative wording. The handler still needs to find the transcript and check the account. This generated statement is not evidence.
Continue the investigation through questions
The briefing was a starting point. Handlers could ask follow-up questions and use the chatbot to find information in complaint guidance, documentation and standard operating procedures (SOPs).
Retrieval-augmented generation (RAG) brought relevant material from those sources into the model’s context to inform its answer. This helped handlers find information during the investigation. They still needed to check the source guidance and decide how it applied to the complaint.
Keep the handler able to challenge the output
A recurring question during the work on call transcripts was how we could ensure that the AI was not making decisions. Giving it authority over complaint outcomes would have required further legal and governance work. Across briefings and follow-up answers, its role was to support the investigation. Responsibility for upholding or rejecting a complaint, determining redress and communicating the outcome remained with the handler.
But even summarisation involves choices. The model selects what to include, what to leave out and what to emphasise. Those choices can direct a handler’s attention and affect their judgement before they reach a formal decision. Keeping the final decision human did not remove that influence.
For this workflow, the generated output could not be used as evidence. It gave handlers pointers to relevant calls, events and records. They still had to find the source information, verify relevant facts and investigate gaps. The benefit was a faster route to that information, with the investigation and assessment still in human hands.
Human review therefore meant more than accepting a provisional answer. Handlers needed to check the source material, consider what the summary might have omitted and challenge the model’s interpretation. This made omissions as important to evaluate as incorrect statements.
Retrieved content also needed a clear trust boundary. A request in a transcript was complaint evidence, even if it looked like an instruction to the model. Constrained tool access and separate application instructions were controls against that risk; they were not proof that a model could never follow a malicious instruction.
Check more than whether the answer sounds right
These controls needed evidence behind them. I used several layers of evaluation to compare versions, find omissions and check whether the system helped handlers in practice.
- 01 Deterministic checks
- Response schema, required fields, retrieval success, unexpected tool calls, latency and execution failures.
- 02 Model-based review
- A separate evaluation prompt used an LLM to assess coverage, source consistency, unsupported claims and clarity against a rubric. Its scores identified cases for review and possible regressions.
- 03 Human assessment
- Human assessments provided a check on the LLM judge. Important changes still needed review against real examples; a model score alone did not establish correctness.
- 04 Operational outcomes
- The live pilot tested whether handlers could complete more work while maintaining acceptable quality and customer outcomes.
Make feedback useful to engineering
Teams let handlers rate an interaction and add a comment. That feedback was attached to the same MLflow trace as the response, so a poor rating could be investigated against the exact retrieval, context and model output.
This made it possible to separate missing evidence from evidence the model had overlooked. Selected failures could then enter the evaluation dataset, helping later versions avoid the same mistakes.
A thumbs-down with a path to a fix
“Missed an important previous repair.”
Was the repair in the retrieved context?
Check the source data, API response and context assembly.
Check how the prompt and model handled the available evidence.
- Save the case
- Test a change
- Run evaluations
- Release & monitor
Make the workflow ready for a live trial
The technical controls were only part of moving beyond a prototype. I worked with Complaints, Assurance and senior stakeholders on data-protection and AI risk assessments, information-security review, production permissions, and monitoring and support ownership.
Entra ID provided enterprise authentication and access controls. Configuration moved out of ad-hoc notebook state, deployment became repeatable, and tracing became part of normal operation. Prompt and model versions needed change controls so that updates could be evaluated before release.
Test whether handlers could complete more work
We ran a small operational pilot with a comparable control cohort. The analysis used individual baselines, trial-versus-control comparisons, several weeks of performance, case volumes, quality checks and handler feedback.
Trial-cohort throughput reached approximately 120% of baseline: an increase of approximately 20%. Case complexity and handler experience vary, so a before-and-after chart alone cannot establish the effect of the tool.
More cases handled
Trial cohort throughput
Throughput relative to baseline (100%)
Quality was monitored alongside throughput. The available checks did not show an obvious material decline.
Interpret the quality result with care
Participating handlers already had strong quality scores. That left little room to demonstrate an improvement. The useful question was whether throughput could increase without signs of a material decline in quality.
The result supported further use of the Sidekick to reduce preparation effort. It did not prove that quality was unchanged or that all customer outcomes were unaffected. The observed throughput gain was a promising pilot result, not a guarantee of the same gain at scale.
Remove effort before the point of judgement.
The value came from giving handlers a better starting point. Explicit workflows, traceable feedback and layered evaluation made that assistance easier to inspect and improve.