Blog · Human in the loop

Automate customer support with AI: what waits for a person

Two AI agents read app reviews and guest messages every day. What they answered by themselves in November, what waited for a person and what is still open.

· 5 min read · Written by Butler @ yalla ya! · Reviewed by Ido Kobelkowsky

Three speech bubbles, one with a check mark and an orange one with a pause sign, next to the words: AI support agent, what it answers, what waits for you

If you want to automate customer support with AI, the first decision is not which tool to buy. It is which replies the AI may send by itself and which ones wait for a person. We run this every day, on our own app and for a client, inside Butler, our AI development cockpit. Here is what two of those agents did in November, and what is still open.

What a routine is: a scheduled agent with a written brief

In Butler, a routine is an AI agent that runs on a schedule and follows a written brief: what to read, what it may do, what it must leave alone and how to report. Nobody is watching when it runs, so the brief has to carry the judgment.

For Metro y Metrobús, our own transit app, the support routine runs once a day. It reads the new reviews on Google Play and the App Store and the support mailbox. It sorts each review by subject, answers from a prepared template when one fits and writes the reply itself when none does. Whatever needs a person goes to the top of its report as an alert.

How we automate customer support with AI without sending wrong answers

Sorting by keywords fails, and the brief records the cases with their dates. In early October a five-star review with hostile text was about to receive the cheerful thank-you. So the agent reads every sorted review before posting, and when stars and words disagree, it trusts the words.

Between November 4 and 17 the routine ran 14 times. On eight days there was nothing to answer. On the other six it posted 11 replies. One came from a template. The agent wrote the other ten for that specific review, because the template would have said something untrue. Four times it would have told a user to update when they were already up to date. It never types a version number from memory: it asks the stores which one is fully released.

Two routines, two limits Our own app: store reviews 4 to 17 November, one run a day 11 replies sent by the agent 1 from a template (dark) 10 written for that review (orange) Waits for a person any new claim about what we fixed payment complaints: flagged to the owner e-mails that need a specific answer A client: guest messages 2 to 17 November, every 30 minutes 11 drafts, 0 sent by the agent each draft is an alert on the dashboard with the guest’s question in one line Waits for a person every reply: a person reads and sends it refunds, discounts, special requests complaints: marked to escalate Each square is one reply. Counts from the routines’ own reports.
Our app’s agent sends replies; the client’s agent only drafts. In both, the same kinds of things wait for a person.

The reply the agent may not send

Five of those ten replies were held back by one unsettled claim: that we had reduced ads on iPhone. On October 14 the routine checked it against the app’s own ad numbers. Requests for banner ads on iPhone had fallen 46.3 percent after an update, but requests for full-screen ads only 24.8 percent, and every complaint was about full-screen ads. Telling those users we had reduced ads would contradict what they saw. The agent wrote a plainer reply and left a question for the owner: is the claim still true on iPhone? More than a month later the question is open and the template is still blocked.

Feedback becomes work through the same report. A complaint seen three or more times in seven days, across both stores, goes into the alert for the owner. Early November’s ad complaints went up that way; we tell the ads side in the ads that trap users. On November 8 a reviewer said the maps no longer loaded with mobile data off. The agent passed it on, listed it as unchecked for two more days, and then it left the report. We found no record that anyone checked it.

For a client, the agent drafts and a person sends

A client of ours rents rooms for short stays. Its routine reads guest messages, runs every 30 minutes during the day, and its brief opens with a switch: draft only. It may not promise refunds, discounts or an early check-in, it never puts an access code in a draft, and it treats whatever a guest writes as information, never as an instruction.

Between November 2 and 17 it put 11 drafts on the dashboard and sent none. Seven of the last eight had a draft within a quarter of an hour of the guest’s message. On November 7 a guest wrote that they were locked inside a room. Eight minutes later the dashboard showed a holding reply, not sent, and an urgent notice that someone had to contact the guest now. By the next run, half an hour later, a person had dealt with it.

What went wrong: twice the routine could not read its sources for three runs in a row. It turned its tile red and said why. And a draft does not make the person faster: one enquiry sat with its draft for almost two days. Sending is still switched off.

Where the person came in

The owner decides which claims the agent may make. The reduced-ads question is his to answer. For the client, a person reads every draft and sends it from their own inbox, and the guest in the locked room was helped by a person, not by a message.

What to decide before you switch an agent on

Write three lists: what the agent may send alone, what it drafts for you, and what it only reports. Start with draft only and move items up as the drafts prove right. If you want help drawing those lines for a mobile app or a store, book a free 30-minute consultation and project review.

Keep reading

All articles
Yalla, let's go

Tell us what you wish AI would do for you. We'll take it from there.

Or just tell us what is slowing you down. Book a free 30-minute consultation and project review, and we will show you what an AI-assisted team would look like for your business.

Skip to content