Start with behaviour you can actually review
Test an AI brand assistant against specific conversations and separate factual accuracy, tone, task completion and escalation. A response can sound appropriate while giving the wrong advice. A single overall brand score hides that distinction. Before changing prompts or models, decide which failures should stop a release.
Nubank's 5 November research announcement describes using simulated customers to guide agent changes before live testing. Its banking results are not a forecast for a small consultancy. The useful lesson is methodological: evaluate alternatives, and validate the simulator that helps you choose them. The smaller workflow below is my proposed adaptation.
Turn brand language into observable decisions
Replace an instruction such as sound helpful with a decision a reviewer can recognise. Does the assistant ask for the missing requirement? Does it explain an unavailable service honestly? Does it preserve an approved price qualification? Define what an acceptable answer must contain without forcing every response into identical wording.
For a fictional design consultancy, useful situations include an enquiry outside its service area, an urgent request it cannot meet, and a prospect asking whether a portfolio concept was commissioned work. Write the approved facts beside each situation. These are illustrative test cases, not reports of JQ client conversations.
Calibrate the simulator against people
Have a person review a small reference set before using another model to judge hundreds of generated conversations. Include acceptable responses, obvious mistakes and difficult borderline cases. Ask the automated reviewer for a decision, the relevant passage and the rule it applied. Compare disagreements before accepting its scores.
Keep some reference examples outside prompt development so they can expose overfitting to familiar cases. A simulator that rewards smooth reassurance while missing an invented capability is unsuitable for this release decision. More simulated conversations will multiply that blind spot rather than repair it.
Change one component and preserve the evidence
Compare the current assistant with one candidate using the same scenario set and approved business information. Record the model, instructions, tool permissions and knowledge version. Keep factual failures separate from style preferences and identify whether a poor answer began with missing information, faulty retrieval or an unsuitable response rule.
In the fictional consultancy example, both versions might answer a delivery question correctly, but one could promise a date without checking availability. Retain the transcript and the approval criterion. It explains the rejected behaviour more clearly than a dashboard average and gives the next prompt revision a concrete target.
Approve a bounded pilot
Choose a limited deployment with a responsible reviewer and a way to pause it. Define when the assistant should ask a question or transfer the conversation. Repeat the relevant cases when the offer, knowledge source or tool permissions change. A successful offline test supports a pilot decision; it does not establish real customer satisfaction.
JQ's agent architecture and prompt engineering work can turn approved brand decisions into an evaluation pack. Start with one assistant and a short human-reviewed conversation set. That creates a useful basis for improvement before increasing autonomy.
Link Map
Sources and editorial review
Assigned to the 3–9 November 2026 editorial series. This edition was prepared and source-checked on 9 November 2026; assigned dates do not imply prior public deployment. Proposed workflows and illustrative examples are JQ AI SYSTEMS editorial analysis, not completed client experiments.
- Nubank: Screen before you serve, simulated customers and agent evaluation (5 November 2026; primary company research announcement).