Agentic AI Systems: Tool-Calling, Planning, and Execution

A single prompt-response call can summarize a document, classify a ticket, or draft an email. It cannot investigate a production incident, reconcile a spreadsheet against a database, or answer "why did revenue drop in the EU region last week" by actually going and looking. The moment a task requires more than one piece of information gathered in sequence, with each step depending on what the previous step found, a single completion stops being the right tool. That's the line where an agent starts being worth the complexity it adds.

We've built and shipped a few of these — a document analysis agent that reads contracts and pulls structured obligations out of them, and an internal SQL assistant that answers ad-hoc data questions without a human writing the query. Neither started as "let's build an agent." Both started as a single well-crafted prompt that worked fine on the demo case and fell over the first time the real input required two lookups instead of one. This post is about what it actually takes to go from that single prompt to something that reliably finishes a multi-step task — the loop structure, the tool-calling mechanics, the state you have to carry between steps, and the failure modes that show up only once real traffic hits it.

Why a single completion isn't enough

Ask a model "what's our current AWS spend trend and is anything anomalous," and without tools it will either refuse (it doesn't know your AWS bill) or, worse, confabulate a plausible-sounding answer. The task decomposes into: query the cost data for the last 30 days, compute a baseline, compare the current week against it, and only then produce a written answer. Each of those sub-steps needs a real system call, and the result of one determines what the next one should even be — you don't know which service to drill into until the first query tells you where the anomaly is.

That dependency chain is the whole reason agents exist. An agent, in the sense we mean here, is a loop: the model proposes an action, something executes that action against the real world, the result comes back into the model's context, and the model decides what to do next — including whether it's done. Everything else in this post is detail on top of that one idea.

Architectures: chain-of-thought, explicit planning, and ReAct

Before tool-calling APIs existed in their current form, people got multi-step behavior out of models with prompting patterns alone. It's worth understanding these because you'll still reach for pieces of them even with modern tool-calling support.

Chain-of-thought is the simplest: ask the model to "think step by step" before answering. It improves reasoning quality on a single completion, but it doesn't let the model touch anything outside its own context. It's a reasoning aid, not an agent architecture. If your task doesn't need external data or side effects, chain-of-thought prompting alone might be all you need — don't reach for an agent loop just because the problem sounds complicated.

Explicit planning loops ask the model to write out a full plan up front — "step 1: fetch X, step 2: compute Y, step 3: compare against Z" — and then execute the plan step by step, usually with a separate call per step. This works well when the task structure is genuinely knowable in advance, like a fixed ETL pipeline or a document template with a fixed set of fields to extract. It falls apart when step 2's outcome should change what step 3 even is. A rigid plan commits to a shape before you have the information to know if that shape is right, and you end up either over-planning (steps that never get used) or re-planning mid-execution, at which point you've basically reinvented the next pattern.

ReAct (Reason + Act) interleaves reasoning and action at every step instead of separating planning from execution. The model reasons about what to do next, takes one action, observes the result, and then reasons again — with the new observation in context — about what to do after that. It's a tight loop rather than a batch plan. This is why ReAct became the default: it handles the case explicit planning can't, where you genuinely don't know step 3 until step 2 comes back, without giving up the benefit of chain-of-thought's improved reasoning at each hop.

Here's the loop shape:

Notice the loop has an explicit exit condition — the model has to decide it has enough information, not just keep going until it runs out of ideas. That decision point is where most of the operational risk in agent systems lives, and we'll come back to it under error handling and guardrails.

In practice most production agents are ReAct with light planning bolted on: the model reasons and acts step by step, but you nudge it toward a rough plan up front ("first check X, then Y") in the system prompt so it doesn't wander on ambiguous tasks. Pure ReAct with no planning guidance at all tends to be slower and more exploratory than it needs to be for well-understood task types.

Tool-calling mechanics

"Tool calling" is really just structured output with a contract. You describe a function to the model — a name, a natural-language description, and a JSON schema for its parameters — and the model, instead of replying with prose, replies with a structured request to call that function with specific argument values. Your code is the one that actually executes it; the model never touches your database or your filesystem directly.

The important part isn't the JSON schema — that's mechanical. The important part is the description, because that's the only signal the model has for deciding when to reach for a given tool versus another one, or versus just answering from what it already knows. A tool named search with the description "searches" gets called at the wrong times and skipped at the right ones. A tool named search_customer_records with the description "Search the customer database by name, email, or account ID. Use this when the user asks about a specific customer's order history, account status, or contact details. Do not use this for product or inventory questions" gets called correctly far more often, because you've told the model not just what the tool does but when it's the right tool among several plausible ones.

The round trip looks like this on the wire, in Anthropic's Messages API shape: you send a request with a tools array (name, description, input_schema) alongside the conversation. The model's response comes back with stop_reason: "tool_use" and one or more tool_use content blocks, each carrying a name and an input object matching your schema. You execute the corresponding function, then send a follow-up request where the user turn contains a tool_result block referencing that tool_use_id. The model sees the result and continues — either calling another tool or replying with text and stop_reason: "end_turn".

Two things trip people up here. First, tool input is a plain object, not a hand-typed string — parse it as JSON, don't string-match against it, because escaping can differ across models and versions. Second, the model can request multiple tool calls in a single turn if the task allows it (fetch three independent records in parallel, say); you're expected to execute all of them and return all their results in a single follow-up message, not one at a time. Splitting parallel results across separate messages quietly trains the model to stop asking for parallel calls, which costs you round trips you didn't need to spend.

Multi-step reasoning: checking before moving on

The tempting mistake is to treat every tool result as automatically good and just feed it back in. A tool call can succeed at the transport level and still return something useless — an empty result set, a record that doesn't match what was asked for, a value that's technically present but clearly wrong (a account balance of null where the schema promised a number). The model needs to evaluate the observation, not just consume it.

This is where the "reason" half of ReAct earns its keep. After a tool result comes back, the next thought shouldn't just be "what's the next action" — it should first be "did that action actually get me what I needed." Prompting for this explicitly helps a lot: something like "after each tool result, state whether it answered your question before deciding on the next step" turns a silent pass-through into a checked step, and it's cheap — one extra sentence of reasoning per hop.

In our document analyzer, this check step is what catches malformed extractions before they propagate. If a "extract effective date" tool call returns a string that isn't parseable as a date, the wrong behavior is to hand that string to the next step (say, a "compute renewal window" calculation) and let it silently produce garbage. The right behavior is for the model to notice the mismatch, re-invoke the extraction with a narrower prompt, or flag that field as unresolvable. That only happens if you've built the "check before proceeding" step into the loop rather than assuming the model will do it unprompted — it doesn't always.

State management across steps

An agent loop is stateless at the transport level — every request to the model is really a full replay of everything so far. What you actually have to manage, as the loop grows, is what goes into that replay:

• Conversation history — the original request, every intermediate assistant turn, every tool call and its result. This is the ground truth of what happened.
• A scratchpad of decisions — not everything the model reasoned about needs to be replayed verbatim forever. For long tasks it helps to explicitly summarize "what we've established so far" as its own artifact, separate from the raw tool-result blobs, so later steps can reference conclusions without re-deriving them.
• Intermediate tool outputs — often large (a full database result set, a full page of extracted document text). These are the first thing to prune when the context window gets tight, because their informational value to the next step is usually much smaller than their token cost — the model needed the full result to decide what came next, but it rarely needs the full result to re-derive a summary of it three steps later.

The practical failure mode is a loop that runs 15 tool calls deep on a complex analysis task and accumulates enough raw tool output that the context window is mostly noise by the end — the model starts losing track of the actual question because it's buried under six paragraphs of raw JSON from steps four through seven. Two things help: compact tool results before they go back into context (return "12 rows matched, top 3 shown" instead of a raw 40-row dump when the full set isn't decision-relevant), and periodically ask the model to write a running summary of what's been established, then drop the older raw tool outputs from the replayed history once they've been folded into that summary. You're trading perfect replay fidelity for a context budget that survives long tasks — which is the right trade, because the alternative is truncation happening to you at an arbitrary point instead of one you chose.

Error handling and self-correction

Tool calls fail. A downstream API times out, a query returns a syntax error because the model guessed at a column name that doesn't exist, a file path doesn't resolve. The question is what the loop does next, and "nothing" is a real answer some engineers accidentally ship — the tool result comes back with an error and the next prompt just... continues, with the model quietly working around a failure it never actually understood.

The right handling depends on what kind of failure it is:

• Transient failure (timeout, rate limit, 503) — retry, with a cap. Two or three attempts with backoff, then treat it as a hard failure rather than retrying forever.
• Malformed request (bad SQL, invalid parameter) — feed the error message back to the model as the tool result, with is_error: true, and let it retry with corrected arguments. This is the case models actually handle well, because the error message itself usually contains the fix ("column cust_id does not exist, did you mean customer_id?" is something the model can act on directly).
• Wrong tool for the job — if a tool keeps failing in a way that suggests it's the wrong approach entirely, not just a malformed call, the model should be able to fall back to a different tool or a narrower version of the task, rather than hammering the same call with cosmetic tweaks.
• Genuinely stuck — after some bounded number of attempts, the correct behavior is to stop and say so. "I couldn't retrieve the Q3 numbers because the reporting API returned an authentication error" is a completely acceptable final answer. A loop that has no concept of "give up gracefully" will either error out ungracefully or, worse, fabricate an answer to have something to say.

The critical piece of infrastructure here isn't clever retry logic — it's a hard step limit. Every agent loop needs a maximum iteration count enforced in your harness code, not requested of the model. Without it, a model that's slightly confused about whether it's made progress can loop indefinitely, and each iteration is a paid API call. We set this explicitly on every agent we've shipped, typically in the 8–15 range depending on task complexity, and log loudly when it's hit — hitting the cap regularly is a signal the task decomposition or tool descriptions need work, not something to quietly paper over by raising the limit.

Grounding: keeping the model from acting on its own hallucinations

The scariest failure mode isn't a bad answer — it's a bad action. A model that hallucinates a fact in its final text response is wrong but harmless. A model that hallucinates a customer ID and then calls a delete_account tool with that hallucinated ID is a production incident.

The fix isn't better prompting — it's a validation layer between the model's tool call and the tool's actual execution, one that doesn't trust the model's stated inputs at face value. Concretely:

• Never let a tool call directly trigger a destructive or irreversible action. Anything that deletes, sends externally (an email, a payment), or mutates a record that can't be trivially rolled back should route through a confirmation step — either a human approval gate for anything above a risk threshold, or at minimum a re-verification against ground truth (does this customer ID actually exist and match what the user described) before execution.
• Validate tool arguments against your own data, not the model's claim about it. If the model says "the invoice ID is INV-4471," check that INV-4471 exists and belongs to the customer in question before acting on it, rather than trusting that the model correctly extracted it from an earlier tool result.
• Separate read tools from write tools explicitly, and give write tools a much higher bar for automatic execution. Our document analyzer only ever reads; it has no tool capable of modifying the source documents or downstream systems, specifically so a hallucinated extraction can produce a wrong report, never a wrong write.
• Log every tool call with its arguments before execution, independent of whether it succeeds. When something does go wrong, "what did the model actually try to do" needs to be reconstructable without relying on the model's own narration of events, which can itself be wrong.

None of this is about distrusting the model's competence generally — it's about not letting a probabilistic text generator be the last checkpoint before an irreversible side effect. That checkpoint should be deterministic code that you wrote and can audit.

Cost optimization for multi-step loops

Every hop in an agent loop resends the accumulated conversation history, so cost scales worse than linearly with the number of steps if you're not careful — a 10-step loop isn't 10x one call's cost, it's closer to the sum of ten increasingly large calls. A few levers matter in practice:

• Batch independent tool calls into a single turn where the model can determine they're independent up front, rather than looping through them one dependency-chain hop at a time. If the task needs three unrelated lookups, one turn with three parallel tool_use blocks costs one round trip instead of three.
• Keep intermediate prompts short. The system prompt and tool schemas get resent on every hop; a system prompt bloated with examples and edge-case instructions that only matter for the final answer is dead weight on every intermediate step. We've had good results splitting an overly long "how to format the final answer" section out of the main system prompt and only injecting it once the loop signals it's about to produce a final response.
• Prune tool outputs as described above — this is a cost lever as much as a context-management one. A 5,000-token raw database dump that gets replayed across the next six hops of the loop is paying six times for information that was only decision-relevant once.
• Use prompt caching for the stable parts. Tool definitions and a large fixed system prompt don't change hop to hop; caching them means each subsequent request in the loop only pays full price for the new tool result and reasoning, not for re-processing the same tool schemas every time. This is close to free money if your loop structure keeps the cacheable prefix (tools, then system, then the stable early conversation) actually stable across turns — reordering tools or editing the system prompt mid-loop kills it.
• Right-size the model per step, not just per agent. Not every hop in a loop needs your most capable model. A step that's purely "does this tool result look like an error" is a much smaller decision than "synthesize a final answer from six sources" — if your harness structure allows it, routing cheap classification-shaped sub-decisions to a smaller model and reserving the expensive model for the steps that need real judgment can meaningfully cut cost on high-volume agents. This only pays off once you have enough loop volume to justify the added complexity of a two-model setup; for most first agents, one capable model for the whole loop is the right starting point.

Evaluation: how do you know it's actually working

"The agent answered correctly on the three examples I tried" is not an evaluation strategy — it's a demo. Before an agent goes anywhere near production traffic, you need at least a small labeled task set and three numbers you track over time:

• Task success rate — did the agent's final answer actually satisfy the request, judged against a held-out set of tasks with known correct answers (or, for open-ended tasks, judged by a human or a separate LLM-as-judge pass against a rubric). This is the number that matters most, and it's worth being honest that it usually starts lower than you'd like — our SQL assistant's first eval pass came in under 70% on a set of realistic analyst questions, mostly on ambiguous column-name guesses, before tool descriptions and a schema-lookup tool fixed most of it.
• Step efficiency — how many tool calls did it take to reach the answer, versus the minimum plausible number. A model that takes 9 hops for a task that should take 3 is burning cost and latency even when it eventually gets the right answer, and a rising step count over time on similar tasks is often the earliest signal that a tool description has gone stale or a new edge case has appeared in traffic.
• Cost per completed task — not cost per API call, cost per finished task, because a task that fails after 8 hops and then gets manually completed by a human has cost you the compute for those 8 hops plus the human's time. This number is what actually tells you if the agent is worth running versus doing the task manually or with a simpler workflow.

Track these against a fixed eval set every time you change a tool description, a system prompt, or the underlying model — agent behavior is sensitive to all three in ways that are easy to miss without a regression check. A "quick prompt tweak" that improves one failure case can silently regress step efficiency on ten others if you're not measuring it.

Worked example: a document analysis agent

Here's the shape of the document analyzer we mentioned earlier, walked through end to end. The task: given a contract PDF, extract obligations, flag anything with a near-term deadline, and answer follow-up questions about specific clauses.

Walking the loop:

1. User request: "Analyze this contract for renewal risk — flag anything with a deadline in the next 60 days."
2. Agent's first thought: it doesn't know the document's structure yet, so the first action is a search call for section headers matching "renewal," "termination," and "notice period" — cheap, cast a wide net before committing to anything narrower.
3. Observation: three matching sections come back with their page locations. The agent reasons that section 7 ("Renewal and Termination") is the primary target and the other two are probably cross-references.
4. Second action: extract on section 7, asking specifically for renewal date, notice period, and auto-renewal clause language — a narrow, typed extraction rather than a full-text dump, because the agent already knows from the check-before-proceeding step what fields it needs.
5. Observation: extraction returns a renewal date, a 30-day notice period, and confirms auto-renewal is active. The agent checks this against today's date — this is the validation step, done in your harness code around the extraction result, not trusted blindly from the model's own arithmetic — and confirms the notice deadline is inside the 60-day window the user asked about.
6. Third action: summarize on the two cross-referenced sections, to check whether either modifies the base renewal terms (a common contract pattern — an addendum that overrides a base clause).
7. Observation: one cross-reference turns out to be irrelevant (a different renewal clause for a different service tier); the other confirms the base terms apply unmodified.
8. Final answer: the agent has enough grounded, tool-verified information to answer — auto-renewal is active, notice deadline falls inside the requested window, here's the exact date and the clause it came from — without ever having guessed at a date or clause it hadn't actually retrieved.

Every fact in the final answer traces back to a specific tool call and its result. That traceability is the entire point of building this as a tool-calling loop instead of asking one prompt to "just read the contract and tell me" — the second approach produces answers that sound just as confident and are much harder to verify after the fact.

Tool-calling setup in Spring Boot

Here's the shape of the tool definition and response-handling side of this in a Spring Boot service, using the Anthropic Java SDK. This shows one tool (extract_contract_field) being declared with its schema, and the service loop that inspects the model's response for a tool-use block, executes the corresponding Java method, and sends the result back.

package com.example.agent;

import com.anthropic.client.AnthropicClient;
import com.anthropic.client.okhttp.AnthropicOkHttpClient;
import com.anthropic.models.messages.*;
import org.springframework.stereotype.Service;

import java.util.List;
import java.util.Map;

@Service
public class ContractAgentService {

    private final AnthropicClient client = AnthropicOkHttpClient.fromEnv();
    private static final String MODEL = "claude-opus-4-8";

    // Tool schema: name, description (critical for correct triggering),
    // and a JSON schema for the input the model must supply.
    private Tool extractFieldTool() {
        return Tool.builder()
                .name("extract_contract_field")
                .description(
                        "Extract a specific structured field (renewal_date, "
                        + "notice_period_days, or auto_renewal_flag) from a "
                        + "named contract section. Call this only after you "
                        + "have located the relevant section via search_sections. "
                        + "Do not call this to retrieve full section text.")
                .inputSchema(Tool.InputSchema.builder()
                        .properties(JsonValue.from(Map.of(
                                "section_id", Map.of(
                                        "type", "string",
                                        "description", "Section identifier returned by search_sections"),
                                "field", Map.of(
                                        "type", "string",
                                        "enum", List.of("renewal_date", "notice_period_days", "auto_renewal_flag"))
                        )))
                        .putAdditionalProperty("required", JsonValue.from(List.of("section_id", "field")))
                        .build())
                .build();
    }

    public String runTurn(List<MessageParam> history) {
        MessageCreateParams params = MessageCreateParams.builder()
                .model(MODEL)
                .maxTokens(4096)
                .tools(List.of(extractFieldTool()))
                .messages(history)
                .build();

        Message response = client.messages().create(params);

        // A tool_use stop reason means the model wants us to execute
        // something before it continues — never treat content[0] as
        // the final answer without checking stopReason first.
        if (response.stopReason().isPresent()
                && response.stopReason().get().equals(StopReason.TOOL_USE)) {
            return handleToolUse(response, history);
        }

        return extractText(response);
    }

    private String handleToolUse(Message response, List<MessageParam> history) {
        for (ContentBlock block : response.content()) {
            if (block.isToolUse()) {
                ToolUseBlock toolUse = block.asToolUse();
                String toolName = toolUse.name();
                String toolUseId = toolUse.id();

                // Validate before executing — never trust the model's
                // arguments as ground truth for anything with a side effect.
                String result = executeAndValidate(toolName, toolUse.input());

                // Append the assistant turn (with the tool_use block) and
                // the tool_result in the next user turn, then loop again.
                history.add(MessageParam.builder()
                        .role(MessageParam.Role.ASSISTANT)
                        .content(response.content())
                        .build());
                history.add(MessageParam.builder()
                        .role(MessageParam.Role.USER)
                        .content(List.of(ContentBlockParam.ofToolResult(
                                ToolResultBlockParam.builder()
                                        .toolUseId(toolUseId)
                                        .content(result)
                                        .build())))
                        .build());

                return runTurn(history); // continue the loop with the new observation
            }
        }
        return extractText(response);
    }

    private String executeAndValidate(String toolName, JsonValue input) {
        if (!"extract_contract_field".equals(toolName)) {
            return "Error: unknown tool " + toolName;
        }
        // Real extraction logic against your document store goes here.
        // Any downstream write or destructive action would additionally
        // require a confirmation gate before this method is ever reached.
        return ContractExtractor.extract(input);
    }

    private String extractText(Message response) {
        return response.content().stream()
                .filter(ContentBlock::isText)
                .map(b -> b.asText().text())
                .findFirst()
                .orElse("");
    }
}

The parts worth calling out: the tool description states not just what the tool does but when to call it ("only after you have located the relevant section"), the response handling checks stopReason before assuming there's a final answer, and executeAndValidate is the seam where you'd insert argument validation or a human-approval gate before anything with real side effects runs — that seam existing at all is the difference between an agent you can trust with write access and one you can't.

Common mistakes

Giving the agent too many tools. A tool set of 20 overlapping options confuses the model's selection more than it helps — it's genuinely hard for a model to reliably pick between search_customers, find_customer, lookup_account, and get_customer_by_id when they're all plausible for the same request. Keep the tool set as small as the task allows, and if two tools serve near-identical purposes, merge them into one with clearer parameters rather than letting the model guess between them.

No hard step limit. We covered this above, but it bears repeating as its own mistake because it's the one that turns a bad answer into a runaway bill. Enforce the cap in your harness, not as a suggestion in the prompt — a prompt instruction to "stop after a few steps" is not a control, it's a hint the model can ignore under pressure.

No validation layer before side-effecting tool calls. The single most consequential mistake on this list. If a tool can delete, send, or pay, and the only thing standing between a hallucinated argument and execution is the model's own judgment, you will eventually ship a bad action. Build the validation as deterministic code outside the model's control.

Trusting tool results without a sanity check. An empty result set, a null where a value was expected, or a value with an implausible magnitude should trigger the agent to reconsider, not proceed as if everything is fine. This is cheap to add — a sentence in the prompt asking the model to state whether the observation actually answered its question — and expensive to debug later if you skip it.

Treating the agent as done when it's actually stuck. A loop that hits its step limit or repeatedly fails the same tool call needs to say "I couldn't complete this and here's why," not silently produce its best guess dressed up as a confident final answer. Distinguish "I finished" from "I gave up" explicitly in how you prompt for the final response, and surface that distinction to whoever's reading the output.

Final takeaway

An agent is not a smarter prompt — it's a loop with state, a tool contract, and a stopping condition, and most of the engineering effort goes into the parts that aren't the model call at all: validating tool arguments before they execute, deciding what to keep in context as the loop grows, capping how long it's allowed to run, and measuring whether it actually finished the task rather than just producing something that reads like it did. Get the tool descriptions and the guardrails right before you worry about which planning pattern sounds most sophisticated — ReAct with a well-scoped tool set and a hard step limit will outperform an elaborate multi-agent architecture with vague tools every time. Start with the smallest loop that solves the real dependency chain in your task, measure success rate and cost per completed task from day one, and add complexity only where the eval numbers tell you it's actually needed.

Next in this AI engineering series: Retrieval-Augmented Generation at Scale, where we'll go deeper into the retrieval tool this agent likely leans on most: vector search over your own data.

Subscribe to get system design and backend engineering posts in your inbox every two weeks.