A policy chatbot that cited every answer – reducing internal product questions by 74%.

The story

Runacres has broked commercial and personal cover across New Zealand for more than 40 years, placing risk with dozens of insurers as an NZBrokers member. That breadth is a genuine strength for clients, but operationally it meant the firm was carrying hundreds of policy wordings, endorsements, schedules and insurer underwriting guides, each with its own definitions, exclusions and sub-limits that shifted every renewal cycle. Brokers were constantly interrupting the product desk to confirm what a policy actually covered, and the same handful of senior people became a daily bottleneck that slowed quotes and tied up the people the business could least afford to have answering routine lookups. The deeper problem was advice risk. In a regulated advice environment, every off-the-cuff answer a broker gives a client is the firm putting its name to an interpretation, and a confident reply pulled from memory or an out-of-date wording can quietly become a coverage dispute months later. Runacres did not want a clever assistant that sounded authoritative; they wanted fast answers the team could trace to an exact clause, stand behind in front of a client, and defend if the Fair Insurance Code or a complaint ever came knocking. Anything that could hallucinate a limit or quote a superseded wording was a non-starter.

What we built

1. 01

   Built permissioned retrieval over current and superseded wordings, schedules and insurer guides, with version awareness so the agent always knows which wording is in force for a given policy period

2. 02

   Engineered every answer to quote the exact policy clause inline, with the insurer and document version shown, so the broker can verify the source before advising the client rather than trusting the model

3. 03

   Tuned the system to decline when retrieval confidence is low, routing the query to a senior broker instead of guessing, so the failure mode is a handoff rather than a wrong answer

4. 04

   Captured a full audit log of every question, answer and cited source for compliance and Fair Insurance Code review, giving the firm a defensible record of exactly what was said and why

5. 05

   Delivered the whole experience inside Microsoft Teams behind Azure AD, so brokers asked questions in the tool they already live in with no new login or workflow to learn

“The first thing a broker checks now is which clause the answer came from, and it's always right there on the page with the wording version next to it. That citation is the whole point - it's what lets us put our name to an answer in front of a client and sleep at night. The product desk has gone from fielding the same questions all day to actually working on placements.”

Results at a glance

• Routine product and desk questions fell by roughly three-quarters, freeing the senior product team from being a daily lookup service
• Broker time-to-answer dropped from around half an hour of hunting through PDFs to a cited reply in seconds
• No advice incidents were traced to a chatbot answer across the first nine months in production, with every reply backed by a verifiable clause
• Newer brokers ramped faster, confidently answering wording questions they would previously have escalated, because the citation was always in front of them