An AI assistant becomes useful when it completes a clearly defined task with information you can trust and a result somebody can check. A fluent demonstration does not establish that it can handle missing information, unusual customer requests or a failed connection to another system.

Start with a business task, then decide how much responsibility the assistant should have. Answering a question, preparing a draft and changing a customer record are different jobs. They need different tests and different controls.

Choose a task with a visible finish

“Handle customer service” leaves too much undefined. “Find the relevant delivery policy and prepare a response for an employee to review” identifies the source, output and decision owner. “Summarise this enquiry and suggest the next qualification question” is similarly easier to evaluate than an open-ended sales agent.

Consider how the team will recognise a correct result. If experienced staff disagree about the answer, resolve the underlying process first. An assistant will otherwise reproduce the disagreement at greater speed.

Separate assistance from authority

  • Read and explain: answer from an approved knowledge set, identify the source and make missing information visible.
  • Prepare: draft a reply, summary or proposed record change for a person to review.
  • Act within a narrow rule: perform a defined operation only after the application checks the user, record and permitted action.
  • Escalate: hand over exceptions, disputed facts and decisions outside the agreed scope.

For example, an assistant might prepare a follow-up task from a conversation. Whether it may send a message, offer a discount or close the opportunity should be decided separately. Keep those permissions in the application, where they can be enforced and audited.

Make the knowledge boundary explicit

Prepare a small set of approved sources with a named owner and review date. Separate public information from internal instructions and customer-specific records. Give a user access only to material they are entitled to see; an answer should not reveal another customer’s account because both records were placed in the same search index.

Design the response to a missing or contradictory source. If two documents contain different delivery terms, the assistant should not quietly select whichever one sounds more convincing. It should identify the uncertainty and direct the question to the responsible person.

Treat incoming content as untrusted

A customer message, uploaded document or retrieved web page may contain instructions that attempt to redirect the assistant. OWASP describes this class of problem as prompt injection. A document being useful evidence does not give it authority to change the assistant’s permissions.

Test examples such as a document telling the assistant to reveal hidden instructions, send records elsewhere or ignore the user’s task. Prompts and filters can help, but they should not be the only protection around valuable data or consequential actions.

Keep actions small and checkable

Give an assistant only the operations needed for its task. OWASP’s excessive agency guidance recommends limiting functionality, permissions and autonomy. A tool that can create a draft is easier to constrain than unrestricted access to an account.

Validate proposed actions against normal business rules: the record belongs to the right company, required fields are present and the requested change is allowed. For higher-impact actions, show the proposed change to an authorised reviewer before execution. Record the outcome and make failures visible.

Give the human handover an owner

A message saying “contact support” is not a complete handover. Decide who receives the case, what context accompanies it and how the customer knows it has been picked up. The handover should include the question, relevant source and attempted action without forcing the employee to reconstruct the entire conversation.

Set a clear rule for pausing automation while a person handles the case. Otherwise the customer may receive an automated response while an employee is making a different commitment. Returning the case to automation should also be deliberate.

Test the work your team actually receives

Build a review set from anonymised examples: ordinary requests, unclear messages, incorrect assumptions, unsupported languages and cases that require escalation. Add checks for inaccessible records, stale documents, duplicate events and provider timeouts. Include the request that should be refused or handed over, not only the one that should succeed.

Evaluate answers for accuracy and usefulness, and evaluate actions for permission and correctness. Re-run the same review set when the prompt, knowledge source, model or integration changes. A working first demonstration is the beginning of that process.

Measure useful work, including the review effort

Track the proportion of drafts accepted without material correction, cases that need handover, unsupported answers and failed actions. Measure employee handling time with the review step included. A faster first response is not a saving if staff spend longer checking and repairing it.

Start with a limited pilot and a clear stop condition. For example, pause automatic actions after a permissions failure while keeping the read-only assistant available. Expand responsibility when the observed results justify it, with a named owner for ongoing review.

GLV’s AI and automation services can begin with this workflow definition. Bring one task, the approved knowledge and a few anonymised examples so the proposed assistant can be judged against real work.