Designing durable state for a human-in-the-loop agent

A human-in-the-loop agent has one property that's easy to design for on a whiteboard and easy to get wrong in a container: the run doesn't finish in one shot. It does some work, then waits sometimes seconds, sometimes however long it takes a person to review an approval. In that gap, ordinary infrastructure events happen anyway: deploys roll out, containers get rescheduled, processes restart. If a run's state lives only in memory, one of those events landing mid-approval doesn't just slow things down it erases the run. The person about to approve it is left approving something that, as far as the system is concerned, no longer exists

Why in-memory state isn't enough

The simplest version of an agent loop keeps everything in a single process: which tools have run, what's pending, what's waiting on a human. It's a reasonable starting point, and it's exactly the design that falls over first in production, because production containers restart for routine reasons autoscaling, deploys, failed health checks not just crashes. None of those are exceptional events from the platform's point of view. All of them are catastrophic for a run whose only record of being “awaiting approval” lived in a variable that just got garbage collected. Worse, this failure mode is silent: nothing throws, nothing alerts. The run simply stops existing, and the audit trail, if there is one has no record of why.

The design: checkpoint before the gate

The Agentic Integration Service treats the approval gate as a durability boundary, not just a control-flow branch. Before a run pauses for human approval, its state which tools have already run, what the pending action is, and the full audit trail so far is checkpointed to Postgres. A restart doesn't lose the run; it just means the next process to pick up that run_id reads the same state back and resumes from exactly where it paused, instead of re-deriving it or, worse, silently dropping it. The ticketing write that follows approval is idempotent for the same reason: a resumed run might re-attempt a step it's not certain completed, and an idempotent write means that's safe rather than a duplicate ticket. The audit log is append-only across this whole sequence, so the full run including the restart is replayable after the fact, not just the happy path.

Where this design still has edges

This pattern solves the specific failure mode it was built for single-instance restarts losing in-flight approvals without claiming to solve every durability problem an agent could have. Two edges worth naming honestly: a single Postgres instance is the right amount of durability for this workload's scale, but it's a ceiling, not a foundation, if the system ever needed multi-region failover. And the current design assumes one approver acts on a given pending action extending it to support multiple approvers racing to act on the same run would need real concurrency control at the checkpoint layer, not just at the tool-call layer. Neither is a problem today; both are the honest answer to “what would you change with more scale.”