You’ve probably seen this in interviews. People can name tools and terms like RAG, MCPs, or tool calling, but applying them to a real business problem gets fuzzy. They also struggle to explain what it takes to get an AI system safely into production and make it scale.

In this guide, you’ll learn a practical five-layer framework for AI system design. You’ll work through the full journey, from scoping the problem to improving performance in production, using a customer support agent as your anchor.

Scope and assumptions set the direction

The first layer is scope and assumptions. Before you start designing the solution, you need to be really clear on what you’re solving and what you’re assuming. Good system design means making those assumptions explicit, not pretending they don’t exist.

Start with four practical questions. Who is the user, and what problem are they dealing with? What outcome are you trying to improve? Which parts of the workflow should the AI handle? And how much control should the system have, especially when it needs to escalate to a human?

For the customer support example, the user is a customer who gets stuck with repetitive requests that still need a human to step in when required. The goal is to resolve more standard questions via the help centre, while also looking up orders, creating tickets, and escalating when the system isn’t confident. Higher-risk actions, like issuing a refund, should require extra control.

Next, make your assumptions explicit across four buckets: data, the types of inputs, operating constraints, and the risk of actions. In the example, you might have a few thousand help centre articles in English, mostly short requests, and thousands of customers each day, with answers expected within a few seconds. Looking up an order is low risk, but refunds are not.

Architecture takes shape with Know, Act, Remember, Coordinate

The next layer is architecture, guided by CARC: know, act, remember, coordinate. This is where you decide what the system must know, what it’s allowed to do, what it needs to remember, and how all the parts fit together. If your scope decisions were solid, CARC naturally turns them into a workable blueprint.

Know means identifying the information the agent must retrieve before it answers. In customer support, a lot of what’s needed sits in unstructured documents, like return policies, shipping rules, cancellation policies, and troubleshooting steps. A first version can use a simple RAG pipeline: chunk the documents, create embeddings, then store them in a vector database.

You then tailor retrieval to the questions you expect. Some questions are semantic, like “Can I send this back?”, so vector search helps. Other questions include exact identifiers like SKUs or order-related IDs, so you may need a BM25-style keyword search. A hybrid retriever can cover both, followed by a reranker so only the most relevant chunks get passed to the model.

Act means defining the tools the agent is allowed to use. In the example, you’d include tools for looking up an order, creating a ticket, checking refund eligibility, and escalating to a human. Notice the difference: eligibility is about gathering information, while issuing a refund is higher-risk, so it needs policy checks or human approval.

For each tool, design four things. First, permissions should follow least privilege, like read-only access restricted to the authenticated customer. Second, blast radius controls what happens if something goes wrong, with refunds and cancellations gated by deterministic policy and approvals. Third, failure handling covers timeouts and API issues, with retries and fallback paths that can end in escalation. Fourth, trust boundaries treat user messages, retrieved documents, and tool outputs as untrusted data, so instructions inside documents don’t quietly give the system extra power.

Memory and coordination keep responses consistent

The memory part covers what the system needs to store across a conversation and over time. In AI systems, memory typically splits into four types: working memory, semantic memory, episodic memory, and procedural memory.

Working memory is the current task context, like conversation history, retrieved information, and intermediate tool results. Semantic memory is persistent facts about the user or account, such as preferences or relevant transaction details. Episodic memory is about what happened before, like previous escalations or a history of refunds. Procedural memory describes how the system should behave based on past patterns.

You don’t have to use all four types right away. In the first version, working memory is needed, episodic memory can help you avoid repeating actions, some semantic memory may help with user context, and procedural memory is often unnecessary at the start.

Coordinate ties everything together. A simple starting point is one agent with access to help centre articles via hybrid retrieval, plus a small set of scoped tools. When a request comes in, you authenticate the user, load retrieval knowledge and memory, and then let the agent answer, call a tool, ask for more information, or escalate. High-risk actions must go through policy checks or human approval, and the loop needs limits for retries, steps, and timeouts so it can’t run forever.

Observability gives you a feedback loop in production

Once the system is working, you still need to understand what it’s doing once it goes live. That’s observability. If something goes wrong, you shouldn’t just see that the final answer was bad. You need to trace where it went off track so you can fix the next version.

Observability focuses on three questions: what should be logged, what evals you should build from those logs, and which evals should run online versus offline. At a high level, you log important inputs and outputs, plus decisions made across every component.

For the customer support system, logging should include what went into the model, what came out, what was retrieved, which tools were called, the arguments passed, how many retries happened, how long each step took, and what it cost.

Then build actionable evals. These are metrics that show how well your system is performing in a specific area, so you can improve it. Examples include context precision and context recall for retrieval quality, groundedness or faithfulness for answer quality, and tool selection accuracy and argument accuracy for tool use. For the full flow, you’d track escalation behaviour, resolution rate, latency, cost per resolution, and customer satisfaction.

Some evals can be checked with code, like whether the right tool was called and whether arguments were valid. For subjective parts, like usefulness or brand voice, you use AI judges with solid rubrics. Observability is your feedback loop: log what happened, find where the system fails, then update the architecture.

Production ops and optimisation handle scale, safety, and cost

Production and ops turn your design into something you can run reliably. You start by revisiting your assumptions and asking what needs to change once real users depend on it. The five key dimensions are scale, latency, reliability, freshness, and security.

Scale often means keeping the agent service stateless, storing conversation state externally, and running multiple replicas behind a load balancer. Latency needs a clear budget, split across retrieval, model calls, and tools; then you optimise the slowest part first. Timeouts also stop one slow dependency from holding up the whole request.

Reliability needs retries, fallback mechanisms, queues for traffic spikes, and graceful degradation when something fails. Freshness matters because help centre articles can change, so you need an ingestion schedule that detects updates and refreshes embeddings and indexes. Finally, security and auditability include authentication and authorisation, encryption both in transit and at rest, and keeping secrets out of prompts and code. You also need an audit trail for sensitive actions, so you can explain what the system accessed, which tools it used, and who approved risky steps.

Optimisation is the last layer. Once the system is working, you improve it based on real usage patterns and edge cases. Optimisation usually targets latency, cost, quality, and specific failure patterns. First, you identify bottlenecks, then consider fixes like faster retrieval, fewer retrieved candidates, a lighter reranker, or even a smaller model if that’s the slowest stage. For cost, you can use ideas like model routing, prompt caching, and semantic caching for repeat questions. For quality, you adjust models or tool flows based on what your evals show.

In interviews, this is where the trade-offs matter. If your support system expands into things like billing, returns, security, or troubleshooting, one agent might end up carrying too much context and too many tools. If that happens, specialist agents behind an orchestrator can make sense, but only if you can clearly justify the added complexity with real benefits.

Conclusion

This five-layer framework helps you move from vague terms to clear design decisions. You learn how to scope the problem, shape the architecture with CARC, add observability, prepare for production, and then optimise based on what you actually see.

Practise the framework on different support-style scenarios, so you can explain your choices confidently in interviews and in real work.

Frequently Asked Questions

What is the first step in AI system design according to this framework?

Define scope and assumptions. Be clear about who the user is, the outcome you’re aiming for, which parts the AI will handle, and the agency level, including when the system escalates to a human.

Why is the CARC framework important for architecture?

It helps you spell out what the system must know, what it can act on, what it needs to remember, and how those pieces coordinate into an initial architecture that fits your assumptions.

What should you include in observability for an AI system?

Log the important inputs, outputs, decisions, retrieved context, tool calls and arguments, retries, step timings, and costs. Then build evals on top of those logs so you can improve specific parts, not just measure them.

How do you ensure production readiness and reliability?

Design for scale with stateless services and externalised conversation state, set a latency budget, and put in retries, fallbacks, queues, and graceful degradation. Add security, encryption, secret handling, and audit trails for sensitive actions.

How do you approach optimisation without overcomplicating the system?

Start with the bottleneck, then improve latency, cost, quality, and failure patterns based on evals and logs. Add extra complexity, like specialist agents, only when you can clearly justify the benefit.