Strategy
Can Your Customers Tell It's AI? Where Automation Builds Trust and Where It Breaks It
11 min read
Most people can't spot a well-built AI assistant in a short chat, but customers rarely stay fooled, and the moment they find out shapes how they judge your business. Drawing on recent research and real failures, this guide shows where automation earns trust, where it loses it, and how to disclose AI without losing the sale.
✦Key Takeaways
- In a short chat, most people can't tell. In a 2025 Turing test, GPT-4.5 with a humanlike persona was picked as the human 73% of the time. Customers tend to find out later, usually when something goes wrong.
- Saying it's AI has a cost: in a field experiment with more than 6,200 customers, revealing a sales chatbot before the call began cut purchase rates by more than 79.7%.
- Being found out costs more. Across 13 experiments with more than 5,000 participants, AI use that came to light later caused the steepest fall in trust.
- Automation builds trust on jobs with a known right answer, such as order status, booking changes and proactive delay notices. It breaks trust when it poses as a person, invents policy or blocks the way to a human.
- Empathy is the surprise: AI-written replies made people feel more heard until they learned AI wrote them. For upset or grieving customers, let AI draft and a person send.
- In the EU, since 2 August 2026, people must be told they're dealing with AI by the first interaction unless it's obvious. Disclose once, in plain words, with what the assistant can do and a one-step route to a person.
In April 2025, users of the AI coding tool Cursor started getting logged out whenever they switched between devices. One of them emailed support and got a reply from "Sam", explaining that this was expected behaviour under a new login policy. There was no new policy. Sam was an AI support bot, and it had made the rule up. By the time a co-founder posted "We have no such policy" on Reddit, there were already reports of users cancelling their subscriptions.
What Cursor did next is the instructive part. It kept using AI for support, but, as The Register reported, every AI-written support email is now clearly labelled as AI. The bot's invented rule was a software problem. The human name on the email turned it into a trust problem.
So, can your customers tell it's AI? Often, at least at first, they can't. But that turns out to be the less useful question. The one that matters is what happens when they find out, because sooner or later many of them will, and usually at the worst possible moment. This guide covers what the research says about both, where automation builds customer trust, where it breaks it, and how to tell customers they're dealing with AI without losing the sale.
Can Customers Tell It's AI? Not in a Short Chat
In 2025, cognitive scientists Cameron Jones and Benjamin Bergen at UC San Diego ran a pre-registered, three-party Turing test. Participants chatted for five minutes with a real person and an AI at the same time, then decided which one was human. When GPT-4.5 was prompted to adopt a humanlike persona, participants picked it as the human 73% of the time, significantly more often than they picked the actual person. ELIZA, a chatbot from the 1960s, managed 23%.
People know they can't reliably spot AI, and it bothers them. In a Pew Research Center survey of 5,023 US adults in June 2025, 76% said it was extremely or very important to be able to tell whether pictures, videos and text were made by AI or by people. Yet 53% were not too or not at all confident that they could.
A support conversation isn't a five-minute test with an alert judge, though. Customers find out later, and in predictable ways: an answer that's confidently wrong, a loop they can't get out of, the same cheerful phrase three times in a row, a detailed reply that lands at 3am within seconds. Or a friend sends them a screenshot. Discovery tends to arrive just after the AI has let them down, which is the worst moment to learn you weren't talking to a person.
Why Finding Out Hurts More Than Being Told
If customers often can't tell, why tell them at all? Because there's solid evidence that disclosing AI has a cost, and stronger evidence that being found out costs more.
The cost of saying it's AI
In a field experiment published in Marketing Science, Xueming Luo and colleagues randomly assigned more than 6,200 customers to receive outbound sales calls from either a chatbot or a human worker. When customers weren't told, the chatbots sold as effectively as proficient workers and four times more effectively than inexperienced ones. But telling customers they were talking to a chatbot before the conversation began cut purchase rates by more than 79.7%. Customers who knew became curt, ended calls sooner and saw the bot as less knowledgeable and less empathetic, despite its objective competence.
The penalty was smaller when the disclosure came later in the call, a tactic that's now off the table in many places, as you'll see below. The more useful finding is where the penalty came from: customers' assumptions about bots, not anything the bot actually did.
The bigger cost of being caught
Oliver Schilke and Martin Reimann at the University of Arizona found the same disclosure penalty in a different setting. Across 13 experiments with more than 5,000 participants, people who said they had used AI in their work were trusted less than people who didn't mention it. But the steepest fall in trust came when someone had used AI quietly and others found out later. As the authors put it, "the cover-up may be worse than the crime."
Put the two findings side by side and the choice looks different. Disclosure is a small, predictable cost, paid once at the start of the conversation. Discovery is a larger cost, paid at the worst moment and often in public. Hiding the AI doesn't avoid the penalty; it postpones it and makes it bigger.
The law is settling the question anyway
In the EU, Article 50 of the AI Act has applied since 2 August 2026. AI systems that interact directly with people must be designed so that those people are told they're dealing with AI, unless that would be obvious to a reasonably well-informed person, and the information has to be given clearly at the latest at the first interaction. In California, a bot disclosure law in force since 2019 makes it unlawful to use a bot online to mislead people about its artificial identity in order to sell to them, unless the bot is disclosed clearly and conspicuously. Check the rules wherever your customers are, but the direction is plain: the late-disclosure tactic from that 2019 study is now a legal risk as well as a trust risk.
Where Automation Builds Customer Trust
None of this means customers dislike automation. What they dislike is being let down and being misled. Automation builds trust when it does the job the customer came to get done, faster than a person could, in situations with a right answer it can look up.
- Fast answers to simple, factual questions. Where's my order? Can I move my booking? How do I reset my password? Are you open on Sunday? The customer cares about the answer, not who gives it, and a correct answer now beats a human reply tomorrow.
- Being there at 11pm. Plenty of customer questions arrive outside office hours. An assistant that handles the routine ones overnight and books a callback for the rest gives better service than an inbox promising to get back to you.
- Telling customers before they ask. A message saying the delivery is running late, with a new time, stops the call before it happens. Proactive updates show that the business is paying attention, the opposite of what people fear from automation.
- Consistency. An assistant working from approved answers gives the same answer every time. Customers notice when the answer depends on who picks up the phone.
- Work the customer never sees. Routing a query to the right person, summarising the case so the customer doesn't have to repeat themselves, drafting a reply for a person to check. This kind of background automation rarely raises trust questions at all, because what the customer experiences is a faster, better-informed person.
The common thread: the AI is open about what it is, it handles tasks with a known right answer, and it gets out of the way when it can't help.
Where Automation Breaks Customer Trust
The failures follow patterns too. Most AI customer service stories that end up in the news involve at least one of these five.
1. Pretending to be a person
Human first names, stock-photo faces, typing indicators that pause for effect, sign-offs like "Cheers, Sam". None of it makes the assistant better at its job. It only makes the eventual discovery feel like a lie. Cursor's bot had a human name, and when it went wrong, some customers complained about the lack of transparency as well as the invented rule.
2. Inventing answers
Ask a language model about something it doesn't know and it will often produce a plausible answer anyway. In customer service, that means invented policies, prices, discounts and deadlines, delivered in the same confident tone as the correct ones. Most of the fix happens before launch: answer only from approved sources, and say "I don't know, let me get someone who does" when those sources don't cover the question. If your help content is out of date or contradicts itself, fix the knowledge base before you build the chatbot.
3. Handling moments that need empathy
This one is counterintuitive. In a study published in PNAS, Yidan Yin, Nan Jia and Cheryl Wakslak found that AI-generated responses made people feel more heard than responses written by people, and that the AI was better at detecting emotions. But when recipients realised a response came from AI, they felt less heard. Wakslak told USC News the two effects were similar in size, so they essentially cancelled each other out, and participants reported an uncanny-valley unease when told that an empathetic reply had come from AI.
So a bereaved customer, a furious one, or one worried about money or health is the worst place for visible AI, even if the AI would write the kinder message. Let AI draft and a person send, or let a person lead.
4. Locking the door to a human
A bot with no way out is the fastest way to lose a customer's goodwill. Klarna is the best-known company to go hard on AI customer service and then change course. In May 2025, chief executive Sebastian Siemiatkowski told Bloomberg that "cost unfortunately seems to have been a too predominant evaluation factor" and that "what you end up having is lower quality". Klarna started recruiting human agents again, and Siemiatkowski said it was "so critical" to be clear with customers "that there will be always a human if you want."
5. Making decisions customers can't challenge
Refusing a refund, closing an account, rejecting a claim. When an AI makes a call like that and there's no way to question it, customers don't blame the software. They blame you. Keep a person on decisions that cost the customer money or access, and make the route to appeal obvious.
How to Tell Customers It's AI Without Losing the Sale
The research suggests the disclosure penalty comes from assumptions: people expect a bot to know less and care less. So disclose in a way that answers those assumptions from the first message.
- Say it once, at the start, in plain words. For example: "Hi, I'm the AI assistant for [business name]. I can track orders, change deliveries and answer product questions. Type 'person' at any time to reach the team." It's honest, it shows what the assistant can do, and it shows the way out.
- Give it a role, not a human identity. "The [business name] assistant" is fine. A human first name, a face and a personality that implies a person are not.
- Make the route to a person real. One word or one button, available at every step, and the person who picks up should see the conversation so far. If the team only works office hours, say so and offer a callback.
- Label AI-written emails too. Labelling was Cursor's fix. If a person reviews and approves each AI draft before it goes out, a label is optional, but that person now owns what it says.
- Let it say "I don't know". A timely handover builds more trust than a confident guess. Test for this before launch with the awkward questions real customers ask; our guide to testing an AI system before launch shows how.
- Watch the moments of discovery. Review conversations where customers ask "are you a bot?", ask for a person or repeat themselves. Those are the places where trust is leaking.
A Five-Question Check Before You Automate a Customer Conversation
Before you hand any customer conversation to AI, run it through these five questions.
- Is there a right answer the AI can look up, or will it have to guess?
- If it gets the answer wrong, who pays, and how much?
- Is the customer likely to be upset, grieving or worried about money or health?
- Can they reach a person in one step, without repeating themselves?
- If a screenshot of this conversation went viral tomorrow, would you be comfortable?
If the answer can be looked up, a mistake would be cheap, the customer is unlikely to be distressed and a person is one step away, automate it and say so. The more answers point the other way, the closer a person needs to be to the conversation. And if the screenshot question makes you wince, fix that before anything else.
Conclusion
Can your customers tell it's AI? In a short chat, often not. But they find out eventually, and how they find out decides whether automation helps your reputation or hurts it. Told at the start, an AI assistant gets judged on whether it solved the problem. Discovered after a mistake, it gets judged as a deception.
The businesses that get this right neither hide their AI nor apologise for it. They use it where there's a clear right answer, say plainly what it is, and keep a person close for the moments that matter. If you'd like help deciding which customer conversations to automate and how to design the handover, talk to AI Native Agency about our AI automation and agent systems work.
Frequently Asked Questions
- Do I have to tell customers they're talking to an AI?
- In some places, yes. Since 2 August 2026, the EU AI Act has required AI systems that talk to people to tell them they're dealing with AI, unless it's obvious, at the latest at the first interaction. California has banned undisclosed bots that mislead people in order to sell to them since 2019. Even where no rule applies, disclosure is usually the safer choice, because being found out damages trust more than being told.
- Does telling customers it's AI reduce sales?
- It can. In one field experiment, revealing at the start of a sales call that the caller was a chatbot cut purchase rates by more than 79.7%, because customers assumed the bot knew less and cared less. Hiding it only delays the cost. A disclosure that shows what the assistant can do and how to reach a person is the practical way to counter those assumptions.
- Can customers tell if they're talking to a chatbot?
- Often not in a short conversation. In a 2025 study, GPT-4.5 prompted with a humanlike persona was judged to be the human 73% of the time after five-minute chats. Customers usually find out later, when the bot gets something wrong, traps them in a loop or ends up in a screenshot.
- Should I give my chatbot a human name?
- It's better not to. A human name and face don't make the assistant any more capable, and they make the moment of discovery feel like deception. A clear role, such as the assistant for your business, sets honest expectations from the start.
- When should an AI chatbot hand over to a person?
- When the customer asks, when approved information doesn't cover the question, when the customer is upset or vulnerable, and when a decision would cost them money or access. The handover should take one step and carry the conversation history with it, so the customer doesn't have to start again.
- Which customer service tasks are best suited to AI?
- Frequent questions with a clear, checkable answer: order tracking, booking changes, password resets, opening hours and product facts. Proactive updates, such as delay notices, also work well. Complaints, bereavement, billing disputes and anything involving vulnerable customers are better led by a person, with AI helping behind the scenes.