How Do Agents Actually Work?

Cover image for article: How Do Agents Actually Work?

There's a lot of mystery around AI agents. This post breaks down how they actually work by building one from scratch in Ruby using RubyLLM. The result is a coding agent that can read your files, edit your code, run shell commands, and evaluate whether what it did made sense.

The entire thing lives in about 10 files of Ruby.

The Core Loop

At its core, an agent is just a loop:

  1. Perceive: What can I see?
  2. Plan: What should I do?
  3. Act: Execute (safely)
  4. Remember: What happened before?
  5. Reflect: Did it work?

That's it. Every agent you've ever used (Cursor, Copilot, Amp, Devin) runs some version of this loop. The differences are in how sophisticated each component is, not in the fundamental architecture.

Here's how each one works with actual code.

Setting Up

The agent is built on RubyLLM, a gem that handles chat sessions, tool registration, and conversation history. We use OpenRouter to access models like qwen/qwen3-coder.

The entry point is straightforward:

RubyLLM.configure do |config|
  config.openrouter_api_key = ENV.fetch("OPENROUTER_API_KEY", nil)
  config.default_model = "qwen/qwen3-coder"
end

Agent.new.run

And the agent itself is a REPL:

class Agent
  def initialize
    @memory = Memory.new
    @chat = RubyLLM.chat
    @chat.with_tools(...)
    @chat.with_instructions(...)
  end

  def run
    loop do
      print "> "
      user_input = gets.chomp
      break if user_input == "exit"

      response = @chat.ask user_input do |chunk|
        print Rainbow(chunk.content).italic
      end
    end
  end
end

Every time you type something, @chat.ask sends your message plus the entire conversation history to the LLM. The LLM either responds with text or calls a tool. If it calls a tool, RubyLLM executes it, feeds the result back, and the LLM decides what to do next. This inner loop continues until the LLM produces a final text response to show you.

That's the core loop. Everything else is just tools.

Perception: What Can I See?

The agent perceives the world through tools. It can't see your filesystem, your terminal output, or your code unless it explicitly asks for it. Perception is active, not passive.

Two tools handle this: ReadFile and ListFiles. I borrowed these from Radan Skorić's excellent post on building a coding agent in Ruby:

class ReadFile < RubyLLM::Tool
  description "Read the contents of a given relative file path."
  param :path, desc: "The relative path of a file in the working directory."

  def execute(path:)
    File.read(path)
  rescue => e
    { error: e.message }
  end
end

class ListFiles < RubyLLM::Tool
  description "List files and directories at a given path."
  param :path, desc: "Optional relative path. Defaults to current directory."

  def execute(path: "")
    Dir.glob(File.join(path, "*"))
       .map { |f| File.directory?(f) ? "#{f}/" : f }
  rescue => e
    { error: e.message }
  end
end

Notice two things. First, the actual implementation is trivially simple: File.read and Dir.glob. Second, the descriptions matter more than you'd think. The LLM uses these descriptions to decide when to call each tool. A vague description means the agent won't pick the right tool at the right time.

Also notice the error handling pattern: catch exceptions and return them to the LLM. Don't crash. Let the agent try to recover. This pattern shows up throughout. Errors are data, not dead ends.

Action: Execute Safely

Action tools let the agent modify the world. We have two: EditFile and RunShellCommand, both also adapted from Radan's post.

class EditFile < RubyLLM::Tool
  description <<~DESCRIPTION
    Make edits to a text file.
    Replaces 'old_str' with 'new_str' in the given file.
    If the file doesn't exist, it will be created.
  DESCRIPTION
  param :path, desc: "The path to the file"
  param :old_str, desc: "Text to search for - must match exactly"
  param :new_str, desc: "Text to replace old_str with"

  def execute(path:, old_str:, new_str:)
    content = File.exist?(path) ? File.read(path) : ""
    File.write(path, content.sub(old_str, new_str))
  rescue => e
    { error: e.message }
  end
end

The edit approach is string substitution: find exact text, replace it. This works surprisingly well because LLMs are good at generating precise string matches from context. It also doubles as file creation: pass an empty old_str with a new path and you get a new file.

Shell commands are where safety gets real:

class RunShellCommand < RubyLLM::Tool
  description "Execute a linux shell command"
  param :command, desc: "The command to execute"

  def execute(command:)
    puts "AI wants to execute: '#{command}'"
    print "Do you want to execute it? (y/n) "
    response = gets.chomp
    return { error: "User declined" } unless response == "y"

    `#{command}`
  rescue => e
    { error: e.message }
  end
end

This is the only tool with a human-in-the-loop gate. The agent proposes a command, you review it, and you decide whether to run it. You really don't want your agent running destructive commands without your sign-off.

With just these four tools (ReadFile, ListFiles, EditFile, RunShellCommand), you already have a functional coding agent. Radan's original implementation stopped here and it works. The agent can read your code, understand it, and make changes.

But it's missing something critical. It has no memory across sessions, no ability to decompose complex tasks, and no way to evaluate whether its own actions actually worked. That's what we built on top.

Memory: What Happened Before?

The chat session gives the agent short-term memory. It remembers everything from the current conversation. But what about across sessions? What if you want the agent to remember that your project uses RSpec, not Minitest? Or that you prefer tabs over spaces?

We added a persistent key-value store backed by a JSON file:

class Memory
  def initialize(path = "memory.json")
    @path = path
    @store = load
  end

  def save(key, value)
    @store[key] = value
    persist
  end

  def recall(key)
    @store[key]
  end

  def list_keys
    @store.keys
  end
end

Two tools wrap this: Remember and Recall.

class Remember < RubyLLM::Tool
  description "Store a piece of information for later recall. Use descriptive snake_case keys."
  param :key, desc: "A descriptive key (e.g., project_test_framework)"
  param :value, desc: "The value to store"

  def execute(key:, value:)
    @memory.save(key, value)
    "Stored '#{key}'"
  end
end

class Recall < RubyLLM::Tool
  description "Retrieve stored information by key. Pass 'list' to see all keys."
  param :key, desc: "The key to recall, or 'list' for all keys"

  def execute(key:)
    return @memory.list_keys if key == "list"
    @memory.recall(key) || "No memory found for '#{key}'"
  end
end

A subtle but important design decision: both tools share a single Memory instance, injected via the constructor:

@memory = Memory.new
@chat.with_tools(
  Tools::Remember.new(@memory),
  Tools::Recall.new(@memory),
  # ...
)

Since memory persists to memory.json, the agent accumulates knowledge across sessions. Ask it about your project today, and it can recall what it learned yesterday.

Planning: What Should I Do?

Give an agent a complex task without planning and things fall apart quickly. "Build me a REST API with auth" becomes a mess of half-finished files and confused tool calls. The agent needs a way to decompose big tasks into small, sequential steps.

We added a Plan model:

class Plan
  Step = Struct.new(:id, :description, :status, :output, :error, keyword_init: true)

  def add_step(description)
    # Appends a step with status "pending", saves to plan.json
  end

  def next_pending_step
    @steps.find { |s| s.status == "pending" }
  end

  def mark_complete(id, output:)
    # Sets status to "complete"
  end

  def mark_failed(id, error:)
    # Sets status to "failed"
  end

  def complete?
    @steps.all? { |s| s.status == "complete" }
  end
end

Three tools expose this to the LLM:

  • MakePlan: takes a goal and an array of step descriptions, creates the plan
  • ViewPlan: returns the current plan with all step statuses
  • UpdateStep: marks a step as complete or failed

The plan serializes to plan.json, so it survives restarts. The agent can pick up where it left off.

But here's the key insight: planning alone isn't enough. Without reflection, the agent will happily mark steps as "complete" even when they're broken.

Reflection: Did It Work?

This is arguably the most important component.

After every action, the agent must call Reflect, a structured self-evaluation:

class Reflect < RubyLLM::Tool
  MAX_RETRIES = 2

  param :step_id
  param :action_taken
  param :expected_result
  param :actual_result
  param :success          # boolean
  param :analysis
  param :next_action

  def execute(step_id:, action_taken:, expected_result:,
              actual_result:, success:, analysis:, next_action:)
    reflections = load_reflections
    retry_count = count_retries(reflections, action_taken)

    if !success && retry_count >= MAX_RETRIES
      next_action = "ESCALATE: Max retries (#{MAX_RETRIES}) reached. Ask the user for help."
    end

    reflection = {
      timestamp: Time.now.iso8601,
      step_id:, action_taken:, expected_result:, actual_result:,
      success:, analysis:, next_action:,
      retry_count: success ? 0 : retry_count + 1
    }

    reflections << reflection
    save_reflections(reflections)
    reflection
  end
end

Three things make this powerful:

Structured parameters force honest evaluation. The LLM can't just say "it worked." It has to state what it expected, what actually happened, and whether those match. This forces deterministic verification: run the tests, check the output, compare.

Retry cap with auto-escalation. After 2 failed attempts at the same action, the agent stops retrying and asks for help. Without this, agents get stuck in infinite loops doing the same broken thing over and over.

Reflection gates plan progression. The UpdateStep tool checks reflections.json before allowing a step to be marked complete. No reflection? No progress. This is enforced in code, not just in the prompt:

# Inside UpdateStep#execute
reflections = JSON.parse(File.read("reflections.json"))
has_reflection = reflections.any? { |r| r["step_id"] == step_id }
return { error: "Must call Reflect before updating step" } unless has_reflection

This is a behavioral contract enforced in code. The system prompt tells the agent to reflect, but the code requires it.

Wiring It All Together

The system prompt in agent.rb orchestrates the cognitive workflow:

@chat.with_instructions <<~PROMPT
  You are a coding agent. Follow this workflow:
  1. Before starting any task, use Recall to check what you already know.
  2. For complex tasks, use MakePlan to break them into small steps.
  3. Execute each step one at a time.
  4. After each action, verify the result deterministically. Then call Reflect.
  5. If Reflect shows success, use UpdateStep to mark it complete.
  6. After completing all steps, use Remember to store what you learned.
PROMPT

This is where the five components come together. The LLM orchestrates itself through the loop (perceive, plan, act, reflect, remember), guided by the system prompt but enforced by the tool contracts.

The full tool registration:

@chat.with_tools(
  Tools::ReadFile, Tools::ListFiles,         # Perception
  Tools::EditFile, Tools::RunShellCommand,   # Action
  Tools::Remember.new(@memory),              # Memory (write)
  Tools::Recall.new(@memory),                # Memory (read)
  Tools::MakePlan, Tools::ViewPlan,          # Planning
  Tools::UpdateStep,                         # Planning (gated)
  Tools::Reflect                             # Reflection
)

When It Fails

These are the common failure modes worth knowing about:

Context window overflow. The chat session sends the full conversation history every turn. Long tasks eat through context fast and the agent starts forgetting things mid-task. Memory helps here. It offloads important facts to disk so the agent can recall them without relying on the conversation window.

Hallucinated verification. Without structured reflection parameters, agents will claim success without checking. "I updated the file." Did you? Did you read it back? Did the tests pass? Forcing expected_result vs actual_result catches this.

Infinite retry loops. An agent that fails at something will often try the exact same thing again. And again. The MAX_RETRIES cap with escalation is essential to break these cycles.

Plan drift. Agents sometimes wander off-plan, doing things that weren't in any step. The planning tools create accountability. If it's not in the plan, why are you doing it?

Wrapping Up

We covered a lot of ground here, but the core idea is pretty simple. An agent is a loop that gives an LLM the ability to see, do, and learn. Everything we built serves one of those purposes.

Perception lets the agent read files and explore the codebase. Planning gives it structure so it doesn't try to do everything at once. Action lets it make real changes. Memory lets it carry knowledge across sessions. And reflection forces it to actually verify its own work before moving on.

There's obviously a wide gap between this and a polished product. But all of that is layered on top of what we built here. The foundation doesn't change.

If you're curious about agents, I'd encourage you to build one yourself. You'll learn more from wiring up the loop and watching it work (and fail) than from any blog post, including this one.

The code is on GitHub.