Written by Kiwi Dynamics Team 1What retail systems have you actually integrated with
Ask specifically which platforms and systems an agency has connected AI to before, Shopify, your courier's API, your inventory system, your helpdesk, not just "we work with e-commerce". A generic answer here is the first warning sign, because retail AI is only useful when it's actually wired into live order and stock data. An agency that can name the exact integrations they've shipped has done this before.
2Can you show a live production system
Ask to see something running in production, answering real customer questions or triggering real reorders, rather than a slide deck or a sandbox environment built for the pitch. Anyone can make a chatbot look good in a controlled demo, far fewer can show you one that's been handling real order volume for months. This is the single most revealing question on the list.
3How will we measure hours and dollars saved
Push for a specific answer on how success will be measured, hours of staff time saved, percentage of tickets resolved without escalation, cart recovery rate, not a vague promise of "efficiency". If an agency can't commit to a number before starting, they likely don't have a track record of hitting one. Success should be measured in your business's terms, not theirs.
4What happens to our data
Ask exactly where your customer, order, and inventory data goes, who can see it, and whether it's used to train anything beyond your own system. This matters more in retail than most industries because customer and payment data carries real compliance weight in both NZ and AU. A serious agency will have a clear, specific answer, not a shrug toward a generic privacy policy.
5Who owns the system once it's built
Ask whether you own the system, the code, and the workflow once it's built, or whether you're locked into paying the agency indefinitely just to keep it running. This question saves a lot of pain later if the relationship ends or your needs change. An agency confident in its own work is usually comfortable handing over ownership.
6How do you handle AI's mistakes
Ask directly what happens when the AI gets something wrong, misreads an order, misapplies a return policy, gives a customer a bad answer. A serious agency will already have an escalation path and a human-in-the-loop design for exactly this, because retail is full of edge cases a model wasn't trained on. If the answer is vague, the system probably hasn't been tested against real messiness yet.
7What's the smallest version we could ship first
Ask what the smallest, fastest version of this is that could go live first, rather than a six-month roadmap before anything ships. The best retail AI projects start with one well-scoped workflow, like order status replies, proven in production, then expand from there. An agency pushing a big bang launch instead of a fast first win is optimising for their invoice, not your result.
8How long until this is actually live
Ask for a real timeline to production, not to a demo or a proof of concept, and ask what's actually required from your team to get there. Retail businesses run on tight margins and tighter time, and a project that drags for months loses momentum and budget along the way. A credible agency can give you a specific number of weeks, tied to a specific first workflow.
9What does ongoing support look like
Ask what happens after launch, who monitors the system, who fixes it when a courier API changes or a new product category breaks a pattern, and what that costs. AI systems need light maintenance as your catalogue and customer base evolve, and an agency that disappears after go-live leaves you carrying that risk alone. Ongoing care should be part of the conversation from day one, not a surprise later.
10Why this workflow, and not another one
Ask the agency to justify why they're recommending this specific workflow for your business, rather than a generic package they sell to everyone. The right starting point depends on where your actual hours and dollars are leaking, and a good agency will point to your numbers, not a template.
A good agency will welcome every one of these questions, because the honest answers are exactly what separates a system that survives contact with real customers from one that never gets past the pitch deck. Kiwi Dynamics answers these questions the same way with every retail client: production systems, clear measurement, and one well-scoped workflow shipped before the next one starts.