Webhook Delivery Patterns for Async Agent Outputs

Picture an agent wired to Stripe's payment_intent.succeeded event. It works flawlessly in every test run, then goes live and starts charging customers' cards through duplicate API calls, two or three times per transaction, because nobody built in a check for Stripe's own retry behavior. Nobody notices until Monday morning, staring at a weekend's worth of duplicate charges and one missing idempotency check. That single omission is the entire thesis of this piece in miniature.

The standard webhook contract was never built for this. A sender fires an HTTP POST, expects an acknowledgment within 5 to 10 seconds, and treats silence as failure, triggering a retry. That contract assumes the receiver does something quick: write a row, flip a flag, forward a message. When an AI agent sits behind that endpoint, the assumption collapses. The agent might need to retrieve documents, query several systems, invoke a language model, validate what comes back, ask a follow-up question, or coordinate multiple tools before it can act, and none of that work is bounded in time. A single LLM call alone can take 8 to 12 seconds, which already eats most of the sender's patience before the agent has done anything else.

So the sender assumes failure and retries while the agent is still mid-task, leaving two, three, or more copies of the same job running concurrently. That is not an edge case. The gap between handling a webhook and building a reliable, webhook-driven agent architecture is exactly where most production failures live, because a misfired handler in an agent system does not just log an error. It takes action, often action that costs money or sends something irreversible downstream.

The two jobs a webhook does in an AI system: trigger vs. callback

Every webhook inside an AI system is doing one of exactly two jobs. It is either a trigger, an external event landing on the system to wake an agent up, such as a payment succeeding, a CI run failing, or a dispute opening, or it is a callback, a completion notice arriving after asynchronous AI work has finished, such as a batch job wrapping up, a transcription becoming ready, or a delegated task closing out.

Webhooks appear at four distinct boundaries in a real system, pointing in two directions: events coming into the AI system, and events going out of it. The trigger/callback split cuts across both directions rather than living on just one side.

Teams spend weeks debating model choice, framework, and vector store, then bolt the whole system onto the outside world with a single route handler that treats every incoming request the same way. That is a mistake, because triggers and callbacks carry different latency profiles, different ownership over retry logic, and different idempotency demands. A trigger has to survive an unpredictable burst of external events; a callback has to survive the agent's own unpredictable processing time. Confusing the two means applying the wrong pattern to the wrong problem, which is precisely the setup that produces the Stripe scenario above. Together, triggers and callbacks form the event layer of an AI system, the delivery infrastructure between the AI application and everything it connects to.

Five delivery patterns

Once you know whether you're building a trigger or a callback, the next decision is mechanical: which of five delivery patterns fits the job. They form a rough progression, each one solving a limitation the previous one runs into.

The platform holds scoped credentials and calls the endpoint directly, giving real-time observability and an immediate result. It only works when processing is genuinely fast, and it breaks the moment an LLM enters the path, since nothing about language model inference is fast in the way this pattern assumes.

Pattern 2 is the inbound HMAC-signed webhook, still a trigger motion, and by far the most common pattern in production. An external event arrives, its signature gets cryptographically verified, and only then does the agent act. A payment clears, a CI run fails, a dispute opens, or an agent run begins. The plumbing here is not new. It shares delivery semantics with any webhook system built over the last decade; what changes is that a model call now is in the middle of the handler. This pattern runs straight into the risk of the handler timing out the moment that handler tries to do real work synchronously.

Pattern 3 answers that problem from the other side: an outbound async request/response, the callback pattern for teams that cannot expose internal APIs. The platform posts to a webhook URL the team controls, the team returns a fast acknowledgment, and the actual result gets posted back later. The subscriber's internal systems never sit exposed on the public internet, and receipt is separated from processing structurally, not just by convention.

Pattern 4 is the provider-initiated callback, where the model provider itself calls back into the receiver's system. As model workloads shifted from quick synchronous completions to long-running jobs, polling "are we done yet" stopped being a workable strategy, so OpenAI, Google, and Anthropic each shipped webhooks for batch runs, fine-tuning jobs, background responses, and agent sessions. The three providers arrived at nearly identical design choices independently, a signal of what correct architecture actually looks like. The next section unpacks that convergence in more detail. Claude (Managed Agent) adds an auto-disable behavior, in which sustained consecutive delivery failures over a period of time lead Anthropic to switch the endpoint off until it is re-enabled manually.

Pattern 5 is the MCP Task primitive, best described as call-now, fetch-later. MCP originally assumed synchronous request-response interactions, a limitation that becomes obvious once agent sessions stretch out in duration. The spec dated 2025-11-25 introduced Tasks, where a request cannot return immediately, the server hands back a taskId, and the client polls through a defined sequence of states: working, input_required, completed, failed, cancelled. Work through 2026 pushed this further toward stateless, horizontally scalable servers running over Streamable HTTP, formalizing call-now, fetch-later as a protocol-level guarantee rather than something each team reinvents on its own. Delivery payloads are kept thin, containing an event type and a resource ID (or a result-file pointer) rather than the full result, so you call back into the API to hydrate the full object, avoiding the risk of serving stale data on a retry. Delivery is at-least-once with explicit duplicate warnings, so you dedupe on a delivery ID.

Decoupling receipt from processing: why the handler should do almost nothing

Everything that actually requires thought happens asynchronously, outside the request-response cycle.

Ignoring this rule makes the failure mode arithmetic. Stripe's webhook delivery timeout is 20 seconds. An LLM call on its own can eat 8 to 12 seconds of that budget. Adding document retrieval, a second model call for validation, or a tool invocation leaves no headroom left anywhere in the stack. The sender times out, assumes failure, retries, and the system lands right back in the duplicate-processing trap from the opening section.

The thin-payload convention that model providers adopted for provider callbacks (covered in more depth in the next section) is this exact same principle, just applied one layer up. Sending an event type and a resource pointer, nothing heavier, forces the receiver to hydrate the full object on its own schedule. It nudges every downstream implementation toward the same shape: acknowledge fast, do the real work later.

This is where dedicated infrastructure earns its keep. Hookdeck operates as a managed queue that absorbs the acknowledgment and durably delivers events into a handler on the receiver's own timeline, while Trigger.dev runs the AI task itself asynchronously, with durable functions built to run for hours or days without timing out, suited to work like long reasoning chains or video transcription. Neither is doing anything conceptually exotic. Both exist because the queue is the boundary that makes the acknowledge-fast, process-later rule survivable at scale. Accept the payload, verify the signature, return 200 OK (or 202 Accepted), and push the event onto a queue, with all heavy work happening asynchronously. The reason for 202 rather than 200 is that it semantically signals that the work is accepted but not yet complete, which is honest about what the handler actually did.

Idempotency: treating at-least-once delivery as a guarantee, not an edge case

Webhooks are documented, across every major provider, to guarantee at-least-once delivery. Duplicates are not a failure of the system; they are a specified, expected behavior of it. That single fact has to reshape how a receiver is built, because the queue-based architecture from the previous section keeps the handler from timing out while creating a new one: retries will happen, and without a defense against them, the agent will process the same event more than once, taking real action each time.

Revisit the Stripe scenario with that framing in mind. The duplicate charges were not evidence that Stripe's retry logic was broken. An agent processing Stripe payment_intent.succeeded events worked perfectly in testing but processed every payment two or three times in production because nobody accounted for Stripe's retry logic (one missing idempotency check, discovered Monday morning after a weekend of duplicate API charges). That reframing moves idempotency out of the category of defensive extra credit and into the category of required infrastructure.

The mechanics are not complicated, but they have to be exact. Use the provider's delivery ID as the idempotency key. Before doing any processing, record that delivery ID in storage; if it is already present, acknowledge the request and exit without touching anything else. That check has to be atomic. Two concurrent deliveries of the same event, arriving milliseconds apart, must resolve to exactly one processing run, not a race where both slip past the check before either has written its record.

Ordering is a separate problem from deduplication. Providers explicitly warn that lifecycle events can arrive out of sequence, so the correct posture is to treat every event as an instruction to go check current state. This connects directly back to the thin-payload convention: always re-fetch from the origin system rather than trusting the webhook body as ground truth. Zendesk sends a full, rich ticket payload; Jira sends the complete issue object, mirroring its own REST API response with no way to trim which fields show up. Both still require a follow-up GET request to know what is actually true right now, because the payload is a notification, not a record.

Retry logic, dead-letter queues, and circuit breakers for agent workloads

Idempotency stops duplicate delivery from causing duplicate harm. It does nothing for the separate problem of delivery genuinely failing, or the agent being unavailable when an event arrives.

Exponential backoff with jitter governs how retries are spaced. A naive retry scheme, fixed interval, fired immediately, produces a thundering herd: an agent endpoint that just recovered from an outage gets hit by every queued retry at once and falls back over. Jitter spreads that load out across time instead of letting it land in a single spike, and backoff prevents a retry storm from compounding whatever caused the original outage.

Dead-letter queues catch what backoff cannot save. Once an event exhausts every retry attempt, it should land in a DLQ rather than vanish silently. That gives an engineer something to inspect, a root cause to trace, and the option to redrive the event once the underlying bug is fixed. Silent discard is a worse outcome for an agent than for almost any other kind of system, because the action the agent was supposed to take simply never happens, and there is no error anywhere to say so.

Circuit breakers close the loop by pausing delivery to an endpoint that keeps failing, rather than continuing to hammer it and risk taking down other parts of the infrastructure along with it. Anthropic builds exactly this behavior into its managed agent product: after roughly twenty consecutive delivery failures, the endpoint is auto-disabled until someone manually re-enables it, which is a provider-side circuit breaker protecting Anthropic's own delivery infrastructure. It does not, however, protect the receiver's upstream dependencies, so that circuit logic still has to be built on the receiving end too.

None of this holds together if the middleware layer hides what is actually happening. If an integration layer silently retries, or quietly absorbs a rate-limit error, while an LLM sits in the loop, the agent's orchestration logic loses the signal it needs to decide to retry the call, switch to a different task, or back off entirely. Errors need to surface. An agent that never sees its own failures cannot route around them.

Push, pull, and change-notification: choosing how agents get triggered in the first place

Every pattern and safeguard above assumes an event has already arrived. How an agent finds out something happened in the first place comes down to three triggering models.

Polling is the oldest and least efficient: an agent or scheduler asks "anything new?" on a fixed interval, burns compute during every quiet period between checks, and can still miss events that land in the gap between one check and the next. Webhooks flip that around. The source system pushes a notification the moment an event happens, idle cost drops to effectively zero because no event means no compute spent checking, and latency falls to under a second. A monitor confirms that a real, meaningful change has occurred and hands the agent a diff, a summary, and an importance score, so the agent only runs when there is confirmed work waiting, and starts already holding the context it needs.

The efficiency gap between polling and event-driven triggering is not marginal. Event-driven architectures run roughly 90% more efficiently than polling and cut API costs by eliminating the wasted work done during every quiet interval, not merely by delivering events a little faster. Cost, in other words, should scale with change, not with the passage of time, and that is the architectural principle that separates an agent built to run at production scale from one that merely works in a demo.

A scheduled agent that re-fetches a page on a timer pays for every single run, including the overwhelming majority where nothing changed, and it has to keep its own copy of the previous state just to notice when something moved. A change-notified agent skips all of that bookkeeping, because the diff is already sitting in its hand when it wakes up. A scheduled job fires a heartbeat webhook at the agent's own endpoint, the agent receives it and checks its internal state, unresolved tickets, pending reviews, data that has gone stale, giving it a lightweight way to catch problems that no external event was ever going to announce, and this heartbeat pattern is a practical complement worth keeping in the toolkit.