LLM Agent Architecture: Memory, Planning, Tools, and Self-Challenge

An effective LLM agent architecture separates four concerns: memory supplies relevant verified state, planning proposes bounded next steps, tools perform typed operations, and self-challenge looks for missing evidence or unsafe assumptions. Deterministic policy and human approval remain outside the model so that critique improves decisions without becoming its own authority.

Start with a state machine, not a personality

A production agent should have observable states such as intake, clarify, plan, execute, verify, wait for approval, complete, and failed. Each state defines allowed transitions and tool access. This is easier to test than an unconstrained loop that continues until the model feels finished. A maximum step count, time budget, and cost budget provide additional brakes.

Keep the model’s role explicit. It can interpret intent and propose actions, but authorization belongs to software and people. The execution layer checks permissions, validates parameters, records receipts, and rejects stale state. This architecture reduces the chance that persuasive text is mistaken for permission.

Memory as typed, sourced state

Split memory by purpose. Working memory holds the current goal and recent observations. Episodic memory stores selected prior outcomes. Semantic memory stores durable facts with provenance. User preferences should be separate from instructions embedded in retrieved documents. Every item needs a source, timestamp, scope, and update rule.

Retrieval is part of the attack surface. Documents, tickets, and web pages may contain instructions that conflict with policy. Present retrieved content as quoted evidence, not executable commands. Prefer small, relevant chunks and show the agent which fields are authoritative. Delete or expire transient material rather than allowing memory to grow forever.

Planning with checkpoints and replanning rules

Plans should be short hypotheses, not promises. Generate the next few steps, identify dependencies, and mark which step needs approval. After each tool result, compare observed state with the expected state. Replan only when a defined mismatch occurs. This avoids both rigid execution of a bad plan and endless plan rewriting.

For complex work, separate planner and executor contexts. The planner sees the goal and summaries; the executor receives one scoped step and only the tools it needs. That separation reduces prompt size and limits accidental authority. It also creates cleaner traces for debugging.

Typed tools and transactional execution

Tool schemas should be narrow, versioned, and validated. Read operations can be available earlier than writes. For a consequential write, use prepare, review, and commit phases. The prepare call returns a diff or preview; the commit call requires an approval token bound to that exact payload. Idempotency keys prevent retries from creating duplicate effects.

Normalize tool errors and expose only actionable details. Retries should have caps and backoff. If the state may have changed after a timeout, read it before retrying. A model must never assume that silence means success. Receipts and postcondition checks are part of completion.

Self-challenge without self-approval

Self-challenge asks a second pass to find unsupported claims, untested assumptions, missing alternatives, and policy risks. The critic can cite evidence and recommend revise, escalate, or stop. It should not grant permissions or waive controls. Using the same model can still catch omissions, but an independent prompt, model, or deterministic checker offers stronger diversity.

Trigger critique where it earns its cost: before irreversible actions, after conflicting evidence, and before presenting high-stakes conclusions. Avoid recursive debate with no stop rule. One or two structured critique rounds are usually easier to evaluate than an open-ended “think harder” loop.

Evaluation and operational controls

Test the architecture with complete traces. Score final correctness, evidence use, tool selection, permission handling, recovery, and the ability to stop. Include injection attempts in retrieved data, stale approvals, duplicate requests, partial tool failures, and misleading success messages. Observe each state transition and retain sanitized receipts.

Deploy behind feature flags, canaries, and rollback. Track recurring failure categories rather than only aggregate success. A strong architecture does not make an LLM infallible; it makes mistakes visible, bounded, and recoverable.

Implementation worksheet

Create a one-page architecture record with the state machine, memory stores, tool inventory, approval service, and observability path. For every transition, name the actor, required evidence, timeout, retry rule, and rollback. Walk one normal trace and four failures: injected retrieval, denied permission, uncertain tool timeout, and stale approval. The design is ready for implementation only when each case reaches an explicit safe state and produces enough evidence for an operator to explain what happened.