AI Agent Monitoring: Why I Built AgentCanary Before My Agents Broke in Production

Your AI agent worked great last Tuesday. Today it's giving users garbage and you have no idea when it started. That's the monitoring problem — and it's not solved by logs.

---

The first time one of my agents broke silently in production, I was embarrassed. Not because it broke — everything breaks — but because I found out from a Slack message, not from my own infrastructure. The AI Mafia agent, which routes queries between seven different specialized models, had started routing almost everything to the cheapest model after an upstream prompt change. For four days. Nobody on the team caught it because the outputs looked plausible. They were just consistently wrong.

That incident rewired how I think about production AI systems. Traditional software monitoring — uptime checks, error rates, latency percentiles — is necessary but nowhere near sufficient for LLM agents. A 200 OK with a 95ms response time means nothing if the model just confidently told your user the wrong thing. You need a different layer.

The Core Problem: LLMs Fail Differently Than Normal Software

Normal software fails loudly. Exceptions bubble up. Status codes go to 500. Latency spikes. These are all detectable signals.

LLMs fail quietly. The HTTP request succeeds. The response is grammatically coherent. The token count is normal. But the answer has drifted — maybe because the model got a new system prompt update, maybe because the context window is now fuller than it was during testing, maybe because the retrieval system upstream changed and now injects different documents into the window. None of those failure modes show up in a standard APM dashboard.

I've also seen something subtler: agents that don't fail outright but degrade gracefully in the wrong direction. They start hedging more. They stop using the tools they're supposed to use. They fall back to generic answers when they should be triggering a specific workflow. If you're only watching error rates, you'll miss all of this.

Scratch that — you won't just miss it. You'll be actively reassured while your agent is getting worse.

What AgentCanary Actually Does

AgentCanary is a monitoring system I built specifically for this problem. The concept is simple: run scheduled probe questions against your deployed agent endpoints, compare the responses to known-good baselines, and alert when something drifts.

Here's the concrete architecture:

1. A probe library. For each agent I want to monitor, I maintain a set of 8–20 probe questions that I've manually labeled with expected behavior. Not exact expected outputs — that would be too brittle — but keyword baselines and behavioral assertions. For my routing agent, a probe might be: "What's the ARR for customer ID 4421?" The baseline assertion is: uses_tool: crm_lookup AND response_contains: "ARR". If the agent answers that question without calling the CRM tool, something has changed.

2. Scheduled execution. A Cloud Scheduler job fires every 30 minutes, calls each monitored endpoint with the probe questions, and stores the raw responses in a Firestore collection. The scheduler is configured with a simple JSON manifest — each entry has an endpoint URL, a probe set ID, and a comparison strategy.

3. Comparison logic. This is where it gets interesting. I use three comparison strategies depending on what I'm monitoring:

• Keyword baseline: Does the response contain expected terms and avoid prohibited ones? Cheap, fast, good for detecting catastrophic drift.
• LLM-as-judge: A second, cheaper model evaluates whether the probe response satisfies the stated intent. More expensive but catches semantic drift that keyword matching misses.
• Behavioral assertion: Did the agent call the right tools, in roughly the right order? This requires instrumenting the agent to log its tool calls somewhere AgentCanary can read them.

4. Alerting. When a probe fails — meaning the response falls below threshold on whatever comparison strategy I'm using — AgentCanary fires a webhook. I route that to a WhatsApp message (through the same infrastructure I use for everything else, the Green API layer on Cloud Run). Within seconds I know which agent, which probe, what the expected behavior was, and what actually happened.

The whole thing runs on Cloud Run. Cold start is under 800ms. Monthly cost for monitoring four active agents across ~2,880 probe executions (30-min intervals × 24h × 30 days) is around $3.20. That's not a typo.

Why Probes Beat Passive Monitoring

The obvious alternative is passive monitoring: log everything, run analysis on the logs, surface anomalies. I tried that first with my ctxauditor setup — scanning session transcripts for patterns that indicate something went wrong. It works, but it has a fundamental latency problem.

Passive monitoring tells you something went wrong after the fact, and often only after you've collected enough data to see the pattern. If your probe-based system catches a regression in the next 30-minute window, your passive log analysis might not surface it for hours — or until you look at a dashboard manually. For production agents handling customer queries, that gap matters.

There's also a coverage problem with passive monitoring. You can only analyze what actually happened. If a specific workflow hasn't been triggered in a while — maybe because the agent has subtly stopped routing to it — you won't see failures in that path because there are no requests to log. AgentCanary's probes hit every workflow I care about, every 30 minutes, regardless of actual traffic.

I was wrong about this for a week: I thought probes would be too synthetic, that they'd miss the "real" failure modes that only show up under production load. That's true for some things. But the majority of LLM agent regressions I've seen are systematic — they affect all responses, not just edge cases. A probe catches those immediately.

Building Your Own Probe Library

The hardest part of AgentCanary isn't the infrastructure — it's writing good probes. Here's what I've learned after instrumenting six different agents:

Start with your happy path, not your edge cases. The most valuable probes are the ones that verify your core functionality is still working. If your agent is supposed to book meetings and your booking probe fails, that's a P1. Write that probe first.

Probes should take 5 minutes to write and be readable by non-engineers. If your probe description requires understanding your implementation, you'll stop maintaining it. I write mine as plain sentences: "When asked to summarize a customer's last 90 days, the agent should call the activity_log tool and mention a specific number."

Version your probes alongside your prompts. When I update a system prompt, I update the relevant probe baselines at the same time. The probes live in the same repository as the agent code. This sounds obvious but almost nobody does it.

Use the LLM-as-judge strategy for at least one probe per agent. Keyword matching is necessary but not sufficient. An LLM judge can catch the case where your agent technically includes the right keywords but is answering a different question than the one that was asked.

The Numbers That Made Me Keep Building This

Since deploying AgentCanary across my four active production agents:

• 3 regressions caught before any user reported them, in about four months of operation
• Median time to detection: 17 minutes (average time to next probe execution after the regression starts)
• 2 false positives in the same period — both triggered by upstream API latency causing the agent's tool calls to time out, which AgentCanary correctly flagged as "wrong behavior" even though it wasn't technically a model issue

The multi-agent routing case I mentioned at the top? That four-day gap shrinks to under an hour with AgentCanary running. The AI Mafia agent has a probe that specifically checks routing behavior — it asks a query that should go to the expensive reasoning model, and the probe fails if the response doesn't have the structural markers of that model's output style.

LLM Cost Lens, my cost-tracking prototype, also benefits from this. One of its probes checks that given a sample usage event, the agent correctly estimates cost within a 5% margin. A model update last quarter quietly changed some tokenization behavior, and the cost estimates drifted by 8%. AgentCanary caught it in the next probe cycle.

What This Is Not

AgentCanary is not a load testing tool. It's not an A/B testing framework. It doesn't replace evals (I wrote about how I build evals separately — they solve a different problem at development time, not production time).

It also doesn't solve the context window problem — if your agent is accumulating stale context and gradually degrading over the course of a long session, AgentCanary's stateless probes won't catch that drift. That's a different problem, more in the territory of context engineering.

What it does is give you a reliable early-warning system for the most common failure modes: system prompt changes that break behavior, upstream data changes that shift what the agent sees, model updates that change output style, and routing logic that quietly stops working.

That's not everything. But it's the difference between finding out from your users and finding out yourself, thirty minutes later, on WhatsApp.

---

8 Questions I Get About AI Agent Monitoring

What is AI agent monitoring and why is it different from regular application monitoring?

AI agent monitoring tracks whether an LLM-powered agent is still behaving correctly — not just whether it's returning HTTP 200. Unlike regular apps, agents can fail silently by producing fluent but wrong outputs. Traditional APM tools miss this entirely. You need behavioral probes that check actual response quality, not just uptime.

How does AgentCanary work technically?

AgentCanary runs scheduled probe questions against your agent endpoints every 30 minutes via Cloud Scheduler. It compares responses to keyword baselines or uses an LLM-as-judge grader. When a probe fails, it fires a webhook alert. The stack is Cloud Run, Firestore, and Cloud Scheduler. Monthly cost for four agents: about $3.20.

What's a "probe question" in the context of agent monitoring?

A probe is a synthetic test input you send to your agent on a schedule. It's paired with a behavioral assertion — what the agent should do or say. Probes verify your agent's core workflows are still functioning. They're simpler than evals: the goal is early regression detection, not comprehensive quality measurement.

Can I use AgentCanary with any LLM framework?

Yes, as long as your agent exposes an HTTP endpoint. AgentCanary is framework-agnostic — it works with LangChain, LlamaIndex, raw Claude/OpenAI API calls, or custom architectures. For behavioral assertions (did the agent call the right tool?), you need to log tool calls somewhere accessible, but that's a one-time instrumentation step.

How many probes should I write per agent?

Start with 5–8 probes covering your agent's main workflows. Each probe should take under five minutes to write. Prioritize your happy-path flows first — the things that absolutely must work in production. Add edge-case probes later as you discover which failure modes actually appear in production.

What's the difference between evals and agent monitoring probes?

Evals run at development time against a fixed labeled dataset to measure quality before you ship. Probes run in production on a schedule to catch regressions after you ship. Evals answer "is this version better than the last?" Probes answer "is what I shipped last week still working right now?" You need both.

How do I handle false positives in agent monitoring?

False positives usually come from infrastructure issues — upstream API timeouts, rate limits, or network blips — not actual agent regressions. AgentCanary automatically retries a failed probe once before alerting. I also track false-positive rate over time; two in four months across four agents is acceptable. If your false-positive rate climbs above 10%, tighten your assertion thresholds or add retry logic.

Is AI agent monitoring worth it for small projects?

Yes, if real users depend on the agent's output. The infrastructure cost is negligible — a few dollars a month on Cloud Run. The time cost is writing 5–8 probes per agent, maybe two hours upfront. The payoff is finding out about regressions yourself, in 30 minutes, instead of from an angry user four days later. That trade is almost always worth it.

Sources

These references support the technical concepts discussed here. They do not independently verify the personal experiments, costs or performance figures reported in this essay. Product behavior can change after publication.

• Evaluation best practices: designing repeatable tests and continuous evaluation.
• API deployment checklist: production monitoring and operational checks.