You don't need a full language model on every customer message
Running generation on every inbound message wastes compute and adds latency where it hurts. The case for a tiered AI pipeline that spends tokens where they're needed.
The most expensive thing you can do with an AI system is run a full generation pass on every input. It’s also, surprisingly, one of the most common things production AI systems do.
The cost shows up in two places: the bill, and the latency. Both get worse as volume grows.
Why everything ends up at the generator
Most AI systems for customer-facing applications are built around a single stage: take the message, send it to a language model, get a reply. Simple to implement, easy to understand, fast to prototype.
The problem is that not every customer message has the same complexity. “What are your opening hours?” is categorically different from “I need a refund for an order that arrived broken and the driver was rude.” The first question has a correct, retrievable answer. The second requires judgment, context, and probably a human.
Running both through the same generation pipeline treats them identically. The routing is the same, the compute cost is similar, the latency is similar. The system is spending the same resources on a FAQ lookup as on a complaint that needs care.
The cascade model
A tiered pipeline separates the work by complexity.
Stage 1: Classification. A fast, lightweight model reads the incoming message and categorises it. Is this a factual question? A booking inquiry? A complaint? An out-of-scope request? This stage runs in milliseconds and produces a category plus a confidence score. It doesn’t generate a reply — it decides what kind of reply is needed.
Stage 2: Guardrail check. Before any AI-generated reply exits the system, it passes through a filter that checks for harmful content, off-brand language, or responses that reference information the system shouldn’t reveal. This stage is also fast and cheap. It’s not generating — it’s validating.
Stage 3: Generation. Only messages that classified as AI-handleable, and that need a crafted response rather than a static answer, reach the full generation stage. By the time a message gets here, the system already knows what category it belongs to and has the relevant context loaded.
The result is that most messages never hit the generation stage. FAQ-class questions get matched to static answers. Low-confidence messages route to the human queue. Only the genuinely complex, AI-handleable messages spend the compute.
What this changes in practice
The cost reduction depends on what percentage of your volume is FAQ-class. For a typical Indonesian FnB operation — opening hours, menu inquiries, booking confirmations, directions — that percentage is high. The majority of incoming messages are answerable from a small, stable knowledge base. Running full generation on every one of them is spending premium compute to produce answers that a lookup handles just as well.
The latency reduction matters more than the cost in high-volume periods. At 2pm when the lunch rush is generating a hundred simultaneous WhatsApp messages, a pipeline that classifies in 50ms and retrieves in 100ms is meaningfully faster than one that generates in 800ms for each message. The customer waiting for confirmation that their booking was received feels the difference.
The confidence score as a routing primitive
The classifier’s confidence score is useful beyond cost optimisation. It’s also the mechanism for routing to humans cleanly.
When a message lands outside the trained distribution — unusual phrasing, an edge case the system hasn’t seen, a question that mixes categories — the confidence score drops. A system that monitors confidence as a routing signal can automatically move those messages to a human queue before attempting a generation that would likely produce a low-quality reply.
This is the circuit breaker pattern applied to AI: the system decides not to try, rather than trying and failing. Low confidence doesn’t mean the AI is broken — it means the AI correctly identified that a particular input falls outside what it can handle reliably.
Building the pipeline in the right order
The instinct when building an AI customer service system is to start with generation — get a model replying to messages, see what the output looks like, iterate. That instinct makes prototyping fast but makes production harder.
A tiered pipeline is easier to build in the other direction:
- Build the human queue first. Everything routes to humans initially.
- Add the classifier. Now some messages get a category; humans still handle everything.
- Add static matching for the high-confidence, factual categories.
- Add generation for the messages that need crafted responses.
- Add guardrails before any generated reply exits the system.
Each stage adds coverage without removing the safety net of the stage before. Generation never becomes the fallback — humans are. The system can be deployed incrementally, and each stage can be evaluated independently before the next is added.
The log is the product
A tiered pipeline also makes the audit trail richer. Every message has a record: which stage handled it, what classification it received, what confidence score it produced, whether it hit the guardrail, how long each stage took.
That record is the primary mechanism for improving the system over time. Patterns in misclassified messages tell you where to expand the training data. Patterns in low-confidence routing tell you where the knowledge base is thin. Patterns in guardrail triggers tell you what edge cases users are actually hitting.
A single-stage generation pipeline produces a reply. A tiered pipeline produces a reply and evidence. Over months, the evidence is more valuable than any individual reply.
The concrete implementation of this pattern — classifier, guardrail, generator, and the confidence-based routing between them — is documented in the Channelflow case study, built for an Indonesian FnB client handling hundreds of daily messages across WhatsApp and Instagram.