What Actually Breaks When You Put a Language Model in a Customer-Facing Flow

Search for what goes wrong with a language model in production and you get a remarkably consistent answer: hallucination, drift, latency, cost, observability gaps. The list is correct. It is also written almost entirely by companies selling observability platforms, which is why every version of it ends at the same place — a dashboard.

The dashboard is not the problem. The problem is that the list is organised by what the model does, and the things that actually hurt you are organised by what the customer does next. Those are different taxonomies. A model that hallucinates into a log file is a quality issue you fix on Thursday. The same hallucination said to a customer, in your brand voice, on a page with your logo on it, is a commitment somebody may hold you to.

This is the version of the list I would want before shipping: the failure modes in the order they actually turn up, what each one looks like from the outside, and the specific thing that catches it. It assumes you have already decided the automation is worth building — should you automate this is the arithmetic for that decision, and it is a different question from this one.

What breaks first when a language model starts talking to customers?

The first thing that breaks is not the model’s general knowledge — it is its account of your own policy, stated confidently, to somebody who then acts on it.

This is the failure that surprises people, because it is the opposite of what the testing predicted. In testing you ask the model hard general questions and it does well. In production, customers ask it narrow, specific, high-stakes questions about your terms: can I get a refund after 30 days, does this plan include that, will you waive the fee, can I apply for the discount retroactively. Those answers are not in the model’s training data in any reliable form. They are in your policy document, your CRM, and three exceptions your support lead carries in her head. The model does not know it does not know. It produces the most plausible-sounding policy, and a plausible-sounding refund policy is a very close relative of a real one.

The reason this matters commercially rather than just technically is that a statement made by your chatbot on your website can be treated as a statement made by you. That is not a hypothetical. In Moffatt v. Air Canada, 2024 BCCRT 149, decided by British Columbia’s Civil Resolution Tribunal in February 2024, a customer asked the airline’s website chatbot about bereavement fares and was told he could apply retroactively. He could not — the airline’s actual policy said otherwise. Air Canada’s position was that it should not be held liable for the information the chatbot provided. The tribunal disagreed, found the airline had not taken reasonable care to ensure its chatbot was accurate, held that this amounted to negligent misrepresentation, and awarded damages of $650.88.

The money is trivial and the money is not the point. The point is the principle the case settles: the chatbot is not a third party you can stand behind. It is a channel, and everything on it is yours. Whatever you would not let a new support hire say unsupervised in week one, the model should not be saying either.

The practical consequence is a scoping rule I would apply before writing a line of the prompt: separate what the model is allowed to explain from what it is allowed to state. Explaining is safe — here is roughly how our pricing works, here is what this feature does, here is where to find that setting. Stating is not — your refund will be processed, you qualify for that rate, we can do that for you. Anything in the second category should come out of a system that knows the answer, with the model doing the wording rather than the deciding.

Why does it get worse after launch instead of better?

It gets worse because the questions change. The model is the same on day 60 as on day 1; the input distribution is not.

Every pre-launch test set is written by people who already know how the product works. Real users do not, and the gap shows up in three specific ways within the first two months:

• The questions get shorter and worse-formed. Internal testers write full sentences. Customers write “refund???” and “not working”. A model that was reliable on well-formed questions has to guess at intent, and guessing at intent is where it starts inventing.
• The long tail arrives. The 20 scenarios you tested cover most of the volume and almost none of the risk. The edge cases — the customer who is angry, the one asking about a plan you discontinued, the one describing a situation your policy genuinely does not cover — arrive at a low rate and a high cost.
• Your own product changes and the prompt does not. You change a price, retire a plan, alter a delivery window. The people who make that change do not think of the prompt as a place where the old value is written down, because it does not look like a config file. So the model keeps confidently quoting a price you no longer charge.

That third one is the same class of defect as the automation failure nobody catches — the workflow that runs green and does nothing. Nothing errors. Every dashboard is green. The system is doing its job perfectly against an instruction that stopped being true in March.

The fix is boring and it works: treat the prompt as production configuration, not as copy. It goes in version control, it has an owner, and any change to pricing, policy or plan structure has “update the assistant’s context” on the same checklist as “update the pricing page”. If your release checklist does not name it, it will not happen.

What happens when the provider updates the model?

Your system’s behaviour changes underneath you, at a time you did not choose, and nothing in your monitoring says so.

This is the failure mode that is most specific to building on someone else’s model and least like anything in normal software. You did not deploy. Your code did not change. But the thing your product’s tone, refusal behaviour and formatting all depend on has been retrained, and the prompt you tuned against the old behaviour is now slightly mistuned against the new one. Usually it is small. Occasionally it is not — a model that used to refuse a category of request now answers it, or one that used to answer now hedges, or the output format your parser depends on gains a preamble.

Three things make this survivable:

Pin the version. Use a dated or pinned model identifier rather than the floating alias that always points at the latest. The floating alias is convenient exactly until the day it is not. Pinning turns an unannounced change into a change you schedule.

Keep a golden set. Thirty to fifty real questions with the answer you consider correct, run on demand, diffed by hand. Not an evaluation framework, not a score out of ten — a list you can re-run in ten minutes and read with your own eyes. Most teams skip this because it feels unscientific. It catches more than the scored version does, because you notice the thing you were not measuring.

Read the deprecation notices. Providers publish them. Somebody has to be subscribed, and if nobody is named, nobody is.

Why did the bill triple in a month?

Because the cost of a language model feature scales with conversation length and retry behaviour, not with the number of customers — and both of those grow quietly.

The budget you build before launch is almost always per-request: tokens in, tokens out, multiplied by expected volume. The bill you get is per-conversation, and a conversation resends the entire history on every turn. A support thread that goes eight turns instead of three does not cost you 2.6× more, it costs considerably more than that, because each turn carries everything before it. Add a retrieval step that pastes in documents, and the context grows faster still.

Then there is the failure case nobody budgets: a request errors, something retries it, and the retry also errors. Retry loops are cheap when the unit cost is a database read and expensive when the unit cost is a model call. The first surprising bill I would expect any team to get is not from traffic. It is from a loop.

What actually controls this:

The last row is the one with the most money in it and the least attention. A lot of production spend is a frontier model doing work that a smaller model does adequately — routing, classification, extraction. The model that talks to the customer should be your best one. The model that decides which queue the customer belongs in does not need to be.

What stops a customer talking your assistant into something?

The thing that stops it is not a better prompt. It is not giving the model the ability to do the damaging thing in the first place.

Two related risks sit here and they are worth naming separately. The first is prompt injection: a user, or a document the model reads, contains instructions that the model follows as though they came from you. The OWASP Top 10 for LLM Applications keeps prompt injection at the top of its list, and the reason it stays there is structural rather than a matter of engineering effort — instructions and data arrive through the same channel, the context window, and there is no equivalent of the parameterised query that cleanly separates them. You cannot prompt your way out of an architectural property. You can only limit what a successful injection is able to reach.

The second is excessive agency: the model has more tools, permissions or autonomy than the job requires, so an influenced model can take a harmful action rather than merely say a wrong thing. OWASP’s mitigations are the ordinary ones from access control — least privilege on every tool, human approval for consequential actions, spending and rate limits — which is the correct and slightly deflating answer. This is not an AI problem with an AI solution. It is a permissions problem, and I have written the longer version of it in deciding what an AI agent is allowed to touch.

The test I would apply to any tool before connecting it: assume the model has been successfully persuaded to call this with the worst plausible arguments. What is the damage? If the answer is “reads a public FAQ”, connect it. If the answer is “issues a refund”, “changes an address”, “sends an email as us”, or “cancels the order”, it does not get called directly — it gets proposed, and something else confirms.

Why the customer-facing part is a brand decision, not a support decision

Because the failure is not evenly distributed across your customers. It is concentrated on the ones already having a bad day.

People do not open a support chat when things are going well. The population your assistant talks to is skewed towards confusion, frustration and urgency, which means a wrong answer lands on the exact customer least able to absorb it, at the exact moment they are deciding what they think of you. That asymmetry is why I treat this as a brand question. Your brand is whatever your worst touchpoint is, and an assistant that is excellent 95% of the time and confidently wrong to an angry customer 5% of the time is not a 95% system in the customer’s memory.

Two design consequences follow, and both cost conversion rate on paper:

The escalation path has to be visible from the first message, not offered after three failures. Hiding the route to a human raises containment and lowers the thing containment is a proxy for. A customer who wanted a person and was made to negotiate with a bot for four turns has already had the experience you were trying to prevent.

The assistant should be able to say it does not know. Explicitly, in the prompt, with examples. Models default to answering because answering is what the training rewards, and “I am not certain — let me get someone who is” is a sentence you have to ask for. It is also, in a support context, a genuinely good answer, and customers respond to it far better than product teams expect.

The order I would instrument it in

Before launch, in this order, because each one catches something the previous one cannot:

1. Log the full exchange, with an identifier. Input, output, model version, prompt version, tools called, timestamp. Without this you cannot investigate anything — you are relying on a customer screenshot. The general principle and the specific fields are in what to log when an AI agent acts on your behalf.
2. A one-click way for a support human to flag an answer as wrong. The people who spot the failures are the ones cleaning up after them. If flagging is a form, it will not get used.
3. A daily read of a sample. Twenty conversations, read by a person, every day for the first month. Nothing automated finds what this finds, and it is the single highest-value habit in the list. It ends when you stop being surprised.
4. Alerting on the shape of the traffic, not just errors. Escalation rate, average turns per conversation, spend per day, refusal rate. All four move before anything errors.
5. A kill switch that a non-engineer can reach. One toggle, documented, that routes everything to the existing human path. If turning it off requires a deploy, it will stay on through the incident.

Item five is worth being blunt about. The reason to build it is that at some point you will want it at an inconvenient hour, and the decision to use it should not depend on who is awake.

What to do in the first hour after it says something wrong

Assume the answer it gave is now a commitment, and work backwards from that.

The sequence I would follow, in order:

Honour it if it is small. If the model promised a discount, a waiver or a delivery date that you could reasonably absorb, absorb it. Arguing with a customer about whether your own website meant what it said is a losing position commercially and, per the Air Canada decision, not a strong one otherwise. The cost of honouring one wrong answer is almost always less than the cost of the argument.

Find out how many others got the same answer. This is what the logging is for. One customer with a wrong answer is an incident; four hundred with the same wrong answer is a policy change you did not make. Search the logs for the topic, not for the exact wording — the model will have said it differently every time.

Turn off the specific capability, not the whole system, if you can. Narrow the blast radius before you debug.

Fix the source, not the symptom. The instinct is to add a line to the prompt saying “never say X”. That works for the exact phrasing and fails for the next one. If the model got the policy wrong, the question is why the policy is not being supplied to it as a retrievable fact rather than a remembered one.

Write down what class of question caused it, and add it to the golden set. The failure you already had is the cheapest test case you will ever get.

The summary table

FAQs

Is a retrieval setup enough to stop the model getting our policy wrong? It removes the largest cause and not the whole problem. Retrieval fixes the case where the model was remembering instead of reading. It does not fix a policy document that is ambiguous, out of date, or contradicted by an exception your team applies in practice but never wrote down. Retrieval makes the model as accurate as your documentation, which is usually a large improvement and rarely a complete one.

Should the assistant be able to complete transactions or only answer questions? Answer questions first, and add actions one at a time, each with a confirmation step, in increasing order of blast radius. The reason is not that the model is incapable — it is that an action taken wrongly is discovered later and costs more than a sentence said wrongly. If you want a general rule for which actions qualify at all, the classes in deciding what an AI agent is allowed to touch apply unchanged.

How do we measure whether it is working? Not by containment. Containment rises when you hide the human option, which is the failure you are trying to avoid. Better measures are resolution without a follow-up contact within a week, escalation rate read alongside satisfaction rather than instead of it, and the number of answers your support team flags as wrong per hundred conversations.

Does a smaller or cheaper model make the failures worse? For the failures in this post, mostly not — they are architectural rather than capability-limited. A stronger model hallucinates a policy slightly less often and is not meaningfully harder to talk into something, because prompt injection is a property of how instructions and data share a channel. Model choice is a cost and latency decision. Scope, retrieval and permissions are the safety decisions.

Why do these systems seem fine in testing and degrade in production? Because your test set was written by people who know how the product works, and it stops representing the real traffic within weeks. The questions get shorter, the long tail arrives, and your own product changes without the prompt changing with it. The related question of why the model sounds equally confident whether it is right or wrong is covered in why ChatGPT gives wrong answers.

The honest position

A language model in a customer-facing flow is a good idea more often than the sceptics say and a much larger commitment than the vendors say. The technology is not the risky part — the risky part is that you have connected a fluent, confident, non-deterministic system to the channel where your customers form their opinion of you, and then measured it on containment.

Most of what makes it work is unglamorous and predates all of this: scope it to what it can be right about, give it the facts rather than trusting it to remember them, restrict what it can do as opposed to what it can say, read the transcripts, and keep a way to turn it off. The parts that fail are rarely exotic. They are the same parts that fail in any automation, which is why the discipline that keeps the automation you should delete from quietly rotting applies here without modification.

If you are deciding whether to ship one, the useful question is not “is the model good enough”. It is: what is the worst sentence this can produce, and are we willing to honour it? If the answer is no, the scope is wrong, and no amount of prompt engineering fixes a scope problem.