Building Effective AI Agents — Summary Notes

Agentic Systems · Practical Guide

Building Effective AI Agents

Simple, composable patterns beat complex frameworks. Here's what dozens of teams taught us.

Notes from building agents with customers · Updated reference from Dec 2024

Over the past year, we worked with dozens of teams building LLM agents across industries. The most successful implementations weren't using complex frameworks or specialized libraries — they were built with simple, composable patterns.

Workflows vs. Agents

All of these variations sit under one umbrella term: agentic systems. But there's an important architectural distinction between the two families.

Workflows

LLMs and tools orchestrated through predefined code paths. Predictable, consistent, easy to reason about.

Agents

LLMs dynamically direct their own processes and tool usage, keeping control over how they complete a task.

When — and When Not — to Use Agents

Find the simplest solution possible and only increase complexity when needed. Sometimes that means not building an agentic system at all.

Agentic systems trade latency and cost for better task performance — decide consciously whether that tradeoff makes sense.

Choosing the right level

Workflows: predictability and consistency for well-defined tasks.
Agents: flexibility and model-driven decision-making at scale.
Often enough: a single LLM call optimized with retrieval and in-context examples.

Frameworks: Helpful, With Caveats

Frameworks simplify standard low-level work — calling LLMs, defining and parsing tools, chaining calls.

Claude Agent SDK Strands Agents SDK by AWS Rivet Vellum

The tradeoff

Extra layers of abstraction can obscure prompts and responses.
That makes systems harder to debug.
They make added complexity tempting when a simpler setup would do.

The advice

Start with LLM APIs directly — many patterns take only a few lines of code. If you do use a framework, understand the underlying code; wrong assumptions about what's under the hood are a common source of errors.

The Building Block: Augmented LLM

An LLM enhanced with retrieval, tools, and memory. Today's models actively use these capabilities — generating their own search queries, selecting tools, and deciding what information to retain.

Two things to focus on

Tailor these capabilities to your specific use case.
Give your LLM an easy, well-documented interface.

One implementation path is the Model Context Protocol, which connects developers to a growing ecosystem of third-party tools through a simple client.

Five Workflow Patterns

01 · Pattern

Prompt Chaining

Decompose a task into a sequence of steps, where each LLM call processes the output of the previous one. Programmatic checks (gates) can verify intermediate steps stay on track.

Use when

The task decomposes cleanly into fixed subtasks. You trade latency for higher accuracy by making each call easier.

Examples

Generate marketing copy, then translate it.
Write an outline, check it against criteria, then write the document.

02 · Pattern

Routing

Classify an input and direct it to a specialized follow-up task. This separates concerns and lets you build more specialized prompts.

Use when

There are distinct categories better handled separately, and classification is accurate — by an LLM or a traditional model or algorithm.

Examples

Split customer service queries (general questions, refunds, technical support) into different downstream processes, prompts, and tools.
Send easy or common questions to smaller, cost-efficient models like Claude Haiku 4.5 and hard or unusual ones to more capable models like Claude Sonnet 4.5.

03 · Pattern

Parallelization

Run LLM calls simultaneously on a task and aggregate their outputs programmatically. Two key variations:

Sectioning: break a task into independent subtasks run in parallel.
Voting: run the same task multiple times to get diverse outputs.

Use when

Subtasks can be parallelized for speed, or multiple perspectives are needed for higher confidence. For complex tasks, LLMs generally perform better when each consideration gets its own focused call.

Examples

Sectioning: one model instance processes user queries while another screens them for inappropriate content or requests — this tends to beat handling both in one call. Also useful for automated evals, where each call evaluates a different aspect of performance.
Voting: several prompts review a piece of code for vulnerabilities; multiple prompts rate content as inappropriate with different vote thresholds to balance false positives and negatives.

04 · Pattern

Orchestrator–Workers

A central LLM dynamically breaks down tasks, delegates them to worker LLMs, and synthesizes their results.

Use when

You can't predict the subtasks needed — for example, how many files a coding change touches and what changes each requires. It looks like parallelization, but the key difference is flexibility: subtasks aren't pre-defined; the orchestrator determines them from the specific input.

Examples

Coding products that make complex changes across multiple files each time.
Search tasks that gather and analyze information from multiple sources.

05 · Pattern

Evaluator–Optimizer

One LLM call generates a response while another provides evaluation and feedback in a loop.

Use when

Evaluation criteria are clear and iterative refinement adds measurable value. Two signs of a good fit: responses demonstrably improve when feedback is articulated, and the LLM can provide that feedback itself. It mirrors the iterative writing process a human goes through.

Examples

Literary translation, where nuances may be missed initially but an evaluator can critique them.
Complex search tasks needing multiple rounds of searching and analysis, with the evaluator deciding whether more searches are warranted.

Agents

Agents are emerging in production as LLMs mature at understanding complex inputs, reasoning and planning, using tools reliably, and recovering from errors.

They start from a command or an interactive discussion with a human. Once the task is clear, they plan and operate independently, returning to the human for information or judgment. During execution they need ground truth from the environment at each step — tool results, code execution — to assess progress, and can pause for feedback at checkpoints or blockers. Work often ends on completion, but stopping conditions like a maximum number of iterations are common.

Implementation is often straightforward: typically just LLMs using tools based on environmental feedback in a loop. That makes a clearly designed toolset and documentation crucial.

Use when

Problems are open-ended, the number of steps can't be predicted, and no fixed path can be hardcoded. The LLM may run for many turns, so you need some trust in its decision-making — ideal for scaling tasks in trusted environments.

Watch out

Autonomy means higher costs and the potential for compounding errors. Test extensively in sandboxed environments with appropriate guardrails.

Examples

A coding agent resolving SWE-bench tasks, which involve edits to many files based on a task description.
The "computer use" reference implementation, where Claude uses a computer to accomplish tasks.

Combine and Customize

These patterns aren't prescriptive. They're common building blocks developers can shape and combine for different use cases. As with any LLM feature, success comes from measuring performance and iterating.

Add complexity only when it demonstrably improves outcomes.

Success in the LLM space isn't about building the most sophisticated system. It's about building the right system for your needs. — From the original post's summary

Three Core Principles

1Maintain simplicity in your agent's design.
2Prioritize transparency by explicitly showing the agent's planning steps.
3Carefully craft your agent–computer interface (ACI) through thorough tool documentation and testing.

Frameworks can help you start quickly, but don't hesitate to reduce abstraction layers and build with basic components as you move to production. That path leads to agents that are powerful, reliable, maintainable, and trusted by their users.

Where Agents Pay Off

A · Customer Support

Familiar chat interfaces plus tool integration — a natural fit for open-ended agents:

Interactions follow a conversation flow while needing external information and actions.
Tools pull customer data, order history, and knowledge base articles.
Actions like issuing refunds or updating tickets can be handled programmatically.
Success is clearly measurable through user-defined resolutions.

Several companies have shown viability through usage-based pricing that charges only for successful resolutions — a sign of confidence in their agents' effectiveness.

B · Coding Agents

Capabilities have evolved from code completion to autonomous problem-solving. Agents are particularly effective because:

Code solutions are verifiable through automated tests.
Agents can iterate using test results as feedback.
The problem space is well-defined and structured.
Output quality can be measured objectively.

In one own implementation, agents now solve real GitHub issues in the SWE-bench Verified benchmark from the pull request description alone. Automated testing verifies functionality, but human review remains crucial for aligning solutions with broader system requirements.

Prompt Engineering Your Tools

Tool definitions deserve as much prompt engineering attention as your overall prompts. The same action can often be specified several ways — a diff versus rewriting a whole file, code inside markdown versus JSON — but some formats are far harder for an LLM to write. A diff requires knowing how many lines change before writing code; JSON requires escaping newlines and quotes.

Choosing a tool format

Give the model enough tokens to "think" before it writes itself into a corner.
Keep the format close to what the model has naturally seen in text on the internet.
Avoid formatting overhead — accurate line counts, string escaping, and similar burdens.

Designing a good agent–computer interface

Put yourself in the model's shoes: if the tool isn't obvious from its description and parameters, it won't be for the model either. Good definitions include example usage, edge cases, input format requirements, and clear boundaries from other tools.
Refine parameter names and descriptions as you would write a great docstring for a junior developer — especially important with many similar tools.
Test how the model uses your tools: run many example inputs and iterate on the mistakes you see.
Poka-yoke your tools — change the arguments so mistakes are harder to make.

While building a SWE-bench agent, more time went into optimizing the tools than the overall prompt. The model kept making mistakes with relative filepaths after moving out of the root directory; changing the tool to always require absolute filepaths fixed it — and the model used it flawlessly.

Action Checklist

Start with the simplest solution; add complexity only when it demonstrably helps.
Try a single optimized LLM call with retrieval and in-context examples first.
Pick workflows for well-defined tasks; pick agents when flexibility is needed.
Begin with LLM APIs directly, and understand the code under any framework.
Invest as much in the agent–computer interface as in the human–computer one.
Expose the agent's planning steps for transparency.
Test agents in sandboxes, add guardrails, and set stopping conditions.
Measure performance and iterate — the right system beats the sophisticated one.