createAgent() a store and pass a sessionId to send(), and the
run survives a crash, a restart, an abort or a human approval that takes
days. Nothing here depends on a particular host: any CheckpointStore
(memoryStore(), SqliteStore, the Cloudflare KV store the
cloudflare-worker build wires up, or your own three-method implementation)
and any ApprovalStore work the same way.
See also: Pause, restart, resume, a short end-to-end tutorial.
store option does the wiring: store.checkpoints holds the run’s
checkpoints and store.approvals holds approval pauses, and
agent.approvals.resolve() keeps checkpointing the run under its
sessionId. stream() takes a sessionId too. Sessions are checkpointed
too: agent.session({ id }) checkpoints every turn and agent.resume(id) (or
session.resume()) finishes an interrupted one - see
Durable sessions.
Underneath, createAgent() runs AgentExecutor.execute() with a
sessionId and a checkpointStore; the behavior described below is the
executor’s, and the advanced section shows the
executor calls.
What is checkpointed, and when
A checkpoint is the run’s whole transcript plus its step count, usage,businessState and a status. send() (and stream()) write it under the
sessionId:
A run that rejects (a provider error, a throwing hook, a
PropagatingToolError, a process crash) keeps its last 'in-progress'
checkpoint. When it rejects (rather than crashes), the checkpoint also gets the
error as lastError and the count of failed attempts as attempts, so
agent.pending() can tell a run that
keeps failing from one a crash interrupted. The next checkpoint the run saves
(it made progress) drops both.
While a sub-agent runs, what it has spent so far is saved on
the lead’s checkpoint after each of its model calls and tool results
(delegatedInFlight). A sub-agent is not checkpointed itself: when the lead
crashes mid-sub-agent and is resumed, the task call runs again and the
sub-agent starts over, but the model calls of the interrupted attempt stay in
the resumed run’s usage (as one more delegated run).
What happens when you send again with the same sessionId
Calling send() again with the same sessionId (or execute(), on the
executor) depends on the stored status:
- Unfinished run (
'in-progress': it crashed, was aborted, or a store write failed). The run resumes (agent.resume(sessionId)does the same without new input). If its last model turn has tool calls without a result, exactly those calls run first, through the normal tool path (argument validation, hooks,needsApproval,toolConcurrency), without calling the model again. Then the loop continues, with the remainingmaxStepsbudget;usagecontinues from the checkpointed totals (the same holds afteragent.approvals.resolve(), from the snapshot).- New
inputis appended as a user message after those tool results. It is never placed between a tool-call turn and its results, so the transcript stays valid for every provider. This is what lets a user interrupt a run (abort it) and redirect it with a new message. inputthat re-sends the message the run started from (the natural “retry the same request after a crash”) is treated as a retry and is not appended again.input: []also just resumes.
- New
- Finished run (
'finished'). The session continues as a conversation: the stored messages are kept andinputbecomes the next user turn. Pass only the new message(s); if you re-send the stored history followed by new messages, the history is recognised and not duplicated.steps,usageandtoolCallson the result count from zero for the new run;messagesis the whole conversation. - Paused awaiting approval (
'awaiting-approval').send()throwsSessionAwaitingApprovalError(withsessionIdandapprovalId) and does not call the model or any tool: new input must not bypass a pending decision. Resolve it withagent.approvals.resolve()first, then send the new message.
input are ignored when it is added to a stored
session (the session already has its system prompt). To end a session and
start over under the same id, call store.checkpoints.delete(sessionId).
Approvals in the middle of a tool batch
When one model turn asks for several tools and some of them need approval, the calls that need none run and are recorded, and the run pauses once on every call that needs approval (approvalIds, one approval record each; see
Several calls in one step).
Calls a steer kept from starting, and the turn’s handoff calls, are recorded
in the approval snapshot (remainingToolCalls). Once the last paused call is
decided, agent.approvals.resolve() (the resumeAfterApproval() export
underneath):
- records each paused call’s result, in call order - the tool’s result if
approved, or a structured
{ error, note }rejection result if rejected; - runs the remaining calls through the same batch logic: they can run,
fail validation, or pause the run again on another approval (a new
approvalId; resolve it the same way, as many times as needed); - calls the model once every call of the turn has exactly one result.
approvalStore for any later approval unless
you pass a different one in the options.
Recovering after a crash
A run that was interrupted (the process crashed or was killed mid-turn) or that is paused on an approval stays instore.checkpoints until someone
finishes it. agent.pending() lists those runs across every session and
send(msg, { sessionId }) id, so a restarted process can find them:
agent.resume() resolves with the finished ExecutionResult (or null when
nothing was pending); it does not return a stream. To watch a resumed run’s
events, stream it instead: agent.stream([], { sessionId }) resumes an
unfinished kind: 'run' checkpoint without new input (see
above) and
returns the usual AgentRun. A paused run streams with
agent.approvals.streamResolve().
pending() reads the shared checkpoint store, so it also lists runs that
are still running right now in another process or replica (as
'in-progress', without lastError) - not only crashed ones. See below for
why resuming one of those is safe.
Each entry is { sessionId, kind, checkpointId, status, approvalId?, approvalKind?, step, lastError?, attempts? }:
sessionIdis whatagent.resume()takes: theagent.session()id for a session turn (kind: 'session', checkpointed as<id>.turn-<n>), or thesend(msg, { sessionId })id (kind: 'run', also forks).statusis'in-progress'(interrupted) or'awaiting-approval'(decideapprovalIdwithagent.approvals.resolve();agent.approvals.list()shows the call).lastErroris set when the last attempt at a session turn, or at asend(msg, { sessionId })run (kind: 'run'), failed;attemptscounts the attempts that failed in a row. Withretryable: false(a 400 from the provider, for example) it will fail again: fix the cause beforeagent.resume(), or drop it withagent.session({ id }).discardPending()(a run:store.checkpoints.delete(id)). An'in-progress'entry withoutlastErrorwas interrupted (a crash), or is still running.
'in-progress'
too. Resuming it is safe: the run holds a lease (see Run leases),
so agent.resume() fails with LOUSHO_SESSION_BUSY before anything runs
instead of running its tool calls a second time.
A checkpoint that cannot be read (a write cut short by a power loss on a store
without durable writes) is listed as 'in-progress' with lastError.code
LOUSHO_CHECKPOINT_CORRUPT (retryable: false), the error send() and
resume() fail with; drop it with discardPending() (or clear()), or
store.checkpoints.delete(id) for a run.
Recovering automatically
agent.recover() runs that recipe for you, in the background: every
intervalMs (default 30 seconds) it lists agent.pending() and calls
agent.resume() on each 'in-progress' run. Start it once per process, next
to the code that serves traffic:
'in-progress': resumed, from its last checkpoint, so completed steps never run again (a tool that was mid-call runs again: make it idempotent).'awaiting-approval'(a tool approval or anask_question): left alone; a human decides it withagent.approvals.resolve().- A run whose
lastErrorhasretryable: false: left alone, since it would fail the same way;agent.pending()still lists it. - A run that failed recently: skipped until its backoff passes,
intervalMsdoubled for each failed attempt in a row, up to 15 minutes. - A run that failed
maxAttemptstimes in a row (default 10): quarantined. The loop stops resuming it and callsonQuarantined(run)once.agent.pending()still lists it, andagent.resume()still runs it. Saving progress resets the count. A failure that kills the process records no attempt, so it is not counted. WithretryWithDiagnosis: true, a run (not a session turn) first gets one last try: a fork<id>.heal, resumed from the same step, with a note saying why it failed. If the fork finishes, the original is deleted andonRecoveredreports the original’s id. If it also reaches the cap, both are quarantined. The note quotes the error as data and caps its length, because the error can hold tool output. The store must keep a history. - During a provider outage: after 3 provider failures in a row, across any runs,
the loop pauses and calls
onOutage(true). It then tries one run at a time, at growing intervals of up to 15 minutes, and carries on as normal, callingonOutage(false), once one gets past the provider. Nothing is quarantined for provider failures while it’s paused, so an outage doesn’t use up healthy runs’ attempts. A bad API key doesn’t count as an outage, since waiting won’t fix it. - A run still running in another process, or one whose crashed process’s
run lease has not expired yet: its resume is turned away
with
LOUSHO_SESSION_BUSYbefore anything runs, silently, and a later sweep tries again once the lease is free. So a crashed run is finished at most one lease expiry plus oneintervalMsafter the crash, and every replica can runrecover(): the leases make sure only one of them resumes a run. The loop also never starts a second resume of a run it is already resuming.
onError(error, run?) hears about a sweep that could not list the store
(run undefined) and a resume that failed (run set); the loop keeps going
either way. onRecovered(run, result) gets each run it finished. Aborting
signal (or stop()) stops the polling and aborts the resumes in flight
(their checkpoints stay for the next start); done resolves once everything
settled. The loop keeps the process alive until it is stopped. Like
pending(), recover() throws LOUSHO_CONFIG_MISSING_CHECKPOINT_STORE
without a checkpoint store. With a custom store that has no leases
(acquireLease), nothing stops it from resuming a run another process is
still running: run it in one process only.
For an overview of recovery alongside the progress guard and improveAgent(), see Self-healing and self-improvement.
Run leases
While a checkpointed run executes (a session turn, asend(msg, { sessionId })
run, an approval resume), it holds a lease on its checkpoint id in the
checkpoint store: an owner id and an expiry 30 seconds out, renewed every 10
seconds. A second process, replica or store object that tries to run or resume
the same id meanwhile - a send() that would resume the pending turn first,
agent.resume(), agent.approvals.resolve() - fails with LOUSHO_SESSION_BUSY
before any tool runs or any model is called (a resolved approval stays
pending). Two replicas behind a load balancer therefore never run one turn’s
tool calls twice.
This holds for replicas on one host sharing one SQLite file on a local disk, or
for a store you implement over a shared database; SQLite on a network
filesystem or across hosts is not supported. See
Sharing a SQLite file.
A lease whose holder died is taken over once it expires. Expiry is judged by
the clock of the process that wants the lease against the expiry the holder
wrote with its own clock, so the file and SQLite stores wait one more lease
period (30 seconds) past it: a run left by a crashed process on another
machine is taken over 30 to 60 seconds after its last heartbeat, and replicas
whose clocks disagree by less than about 50 seconds never take over each
other’s live runs. Keep replica clocks in sync (NTP); a larger skew can let
two replicas run one turn.
The file and SQLite stores take a lease over at once, without waiting, only
when its holder is provably gone:
- a run of the same process (the same thread and SDK instance) that no longer runs, so the usual “restart and resume” path does not wait;
- on Linux, a process that has exited, on the same host, the same boot and the same PID namespace (the lease records the boot id, the PID namespace and the process start time, so a reused process id does not count as alive or dead by mistake);
- on macOS and Windows, a process id of the same host that no longer exists.
LOUSHO_SESSION_BUSY at its next checkpoint instead of overwriting the new
holder’s progress, and a session turn’s late result is not added to the
transcript. A tool call the stalled run was already making when it lost the
lease is not undone: keep tools idempotent where a stall is possible.
The file, SQLite and in-memory stores take leases atomically. KV has no
compare-and-swap: KVStore writes the lease, waits 50 ms and reads it back,
which serializes requests at one location but not across edge locations within
KV’s propagation delay; its fenced saves are a read then a write. A custom
CheckpointStore takes part by implementing acquireLease(id, owner, ttlMs)
(an atomic “take or renew unless another owner holds an unexpired lease”,
resolving true when taken) and releaseLease(id, owner); without them runs
execute without leases. It can add renewLease(id, owner, ttlMs) (renew only
the owner’s own lease) and saveIfLeased(id, checkpoint, owner) (save only
while the owner holds the lease, atomically); without them a run renews its
lease with acquireLease right before each save, which leaves a short window.
During a rolling upgrade from a release without run leases, the old replicas
neither take nor respect leases, so a turn can still run on an old and a new
replica at once until every replica runs the new release. A SQLite database
opened by this release is migrated (the leases table) and is then refused by
older releases; roll back by restoring a copy of the file.
Schedules use the same mechanism: fireSchedule() (and so startSchedules())
claims each schedule’s fire time in the agent’s checkpoint store first, so
replicas that fire the same tick run it once.
agent.pending() needs a checkpoint store with list(): the file, SQLite,
in-memory and KV stores have one (KVStore lists only when its namespace
binding has list, as a real one does). On a custom CheckpointStore,
list({ status? }) returns { sessionId, status, checkpoint } for the
latest checkpoint of every id; listCheckpoints(entries, options) does the
filtering and ordering for you. Without it agent.pending() throws
LOUSHO_CONFIG_INVALID; without a checkpoint store at all, it throws
LOUSHO_CONFIG_MISSING_CHECKPOINT_STORE.
Guarantees
- Every tool call gets exactly one result, in call order, after any sequence of crashes, aborts, pauses and resumes. (An aborted call gets a “cancelled” result; a rejected call gets a rejection result.)
- A recorded result is never re-executed. Each result is checkpointed
by its
toolCallIdas soon as its call settles - even one that finished behind a slower earlier call of the same turn - so a resumed run only runs calls that have no recorded result. - A checkpointed model turn is never generated again. If the process dies after the model answered, the resumed run executes that answer’s tool calls instead of asking the model again (no extra cost, no different plan).
- The provider always receives a valid transcript: tool results directly follow their tool-call turn, and new user input comes after them.
- An approved call runs at most once. Deciding an approval keeps the
paused run’s checkpoint: it is re-saved as
'in-progress'with the approved call marked as started (startedToolCallIds) before the call runs, and saved again with its result before the run continues. If the process dies while the approved call runs, the turn is not lost:pending()lists it, and resuming it does not run the call again. The call gets an error result (kind: 'interrupted') telling the model it may or may not have taken effect, so the model can check or ask instead of silently repeating it - a human approved it once, not twice. - A stale approval never comes back. An approval continues its run only
while that run’s checkpoint still waits on it. Once the checkpoint was
deleted (
checkpoints.delete(id),discardPending(), a prune) or replaced by a newer run under the same id, resolving it fails withLOUSHO_APPROVAL_ORPHANEDand withdraws it, without running the tool or touching the newer run.discardPending()withdraws the dropped turn’s approvals at once.
Run a job once (webhook redelivery)
A webhook or queue delivers at least once, so the same payload can arrive twice. Sending it again with the samesessionId is not a no-op: once
the first run is 'finished', the second send() is a new turn of that
conversation and calls the model again (in a consumer test it re-ran seven
model calls). To run a job once, derive the sessionId from the delivery id
and look at the checkpoint before sending:
deliveryId, or a
unique-insert in your own database) and keep the run lease on (see
Run leases) so a second process fails with
LOUSHO_SESSION_BUSY instead of running the job twice. Tools that run again
on resume must still be idempotent, so also pass the call’s toolCallId as
the idempotency key of every side effect (see the next section).
At-least-once tools: make side effects idempotent
A tool that was running when the process died has no recorded result, so it runs again on resume. Tool execution is therefore at-least-once, not exactly-once. For tools with side effects (charging a card, sending an email), either gate them withneedsApproval or make them idempotent.
Each call’s toolCallId (the model’s id for the call, unchanged when that
same call is re-run on resume) is passed to execute, so it works as an
idempotency key against a re-run of one call:
toolCallId is set for every tool run, including tools that route through
sandboxExecute.)
The built-in file tools are safe to re-run. write_file writes the same
content again; an edit_file whose edit is already in the file (old_string
gone, a non-empty new_string present) succeeds and says the edit was already
applied, instead of telling the model that an applied edit failed. With
WorkspaceCheckpoints, each change’s backup is stored before the file is
written and a re-run keeps the first backup, so rewind() restores the
original content after a crash too (see
Workspace tools). The shell tool is not:
a command that was running when the process died runs again.
toolCallId (and ctx.approval.id for an approved call) identifies one call
the model made. It only protects against that call running twice. If the
process dies before the call is recorded, the model may make the call again
as a new call with a new id and a new approval, and the id no longer
matches. A side effect that must happen once per business event (one claim,
one invoice) needs an idempotency key from your own data, such as the claim id
in the arguments, checked or passed on to the system you write to:
Checkpoint history
A checkpoint record is overwritten on every save, so the run’s earlier steps are gone once it moves on. A store can also keep a bounded history per session: everysave() appends the record to a ring (the newest historyLimit saves,
default 50; the oldest are dropped), which is what replaying or forking a run
from an earlier step is built on.
Which stores keep one:
For example, with the in-memory store:
history(sessionId, { limit? })returns{ step, savedAt, status, checkpoint }entries, newest first (stepischeckpoint.stepIndex,savedAtan ISO timestamp,statusis'in-progress'for a checkpoint without one). The executor saves after every model turn and tool batch, so a session has one entry per save; the entries hold full checkpoints, so lowerhistoryLimitfor long transcripts. An unknown session gives[].historyis optional onCheckpointStore: a custom store without it keeps working, andgetCheckpointHistory()returnsundefinedfor it instead of throwing.KVCheckpointStorekeeps one index key per session (<prefix><sessionId>#history, the entry ids oldest first) and one key per entry (<prefix><sessionId>#history/<id>), so no value grows with the history; the latest checkpoint stays at<prefix><sessionId>, so a store written before history existed still loads. A save writes the checkpoint, then the entry, then the index, then deletes the entries it dropped: a crash between writes leaves an unlisted entry at worst (it expires with the TTL), never an unreadable store. KV has no transactions, so two concurrent saves of one session can lose one index update (that entry is missing fromhistory()), and an entry can be missing for a while at another edge location (eventual consistency;history()skips it). Route a session’s requests to one location when that matters. Each save costs onegetand twoputs more;historyLimit: 0turns it off.- Agent Forge’s file store writes the history as one file per session
(
.lousho/agents/<id>/checkpoint-history/<session>.json, replaced atomically); a file damaged by a crash reads as no history and is rebuilt by the next save. SqliteStorekeeps the history in a newcheckpoint_historytable, added when an existing database file is opened;prune()also removes history entries older than its cutoff.
Fork and replay
agent.fork(sessionId, { fromStep, patch? }) starts a new session from a step
of another session’s history: “what if the tool had returned something else
at step 1?” It takes the newest history entry of fromStep, applies patch,
saves the result as the 'in-progress' checkpoint of the new session over
store.checkpoints and returns { sessionId, step, checkpoint }. Resume the
fork like any unfinished run with agent.resume(fork.sessionId):
AgentExecutor.fork() is the same operation on the executor; see the
advanced section.
agent.fork() forks one run by its checkpoint history. To branch a
multi-turn agent.session() conversation, use session.history() and
session.fork({ fromStep }), which work from the session’s saved transcript
with any store (finished turns keep no checkpoint history); see
Forking a session.
patchis applied in this order:messages(messages)rewrites the transcript;businessStatereplaces it;toolResult: { toolCallId, result }replaces that call’s result in place (same position, call id and tool name, so the transcript stays valid for every provider), or records it when the call has no result yet;appendInputqueues a user message, which is sent after any tool calls still pending.- The fork keeps the source checkpoint’s step count and usage, so the rest of
the
maxStepsbudget applies to it. Recorded tool results are not re-run; tool calls without a result run first, as on any resume (so a fork of an'awaiting-approval'entry goes throughneedsApprovalagain; the oldapprovalIdis not carried over). newSessionIdnames the fork; the default is<sessionId>.fork-<n>, the firstnwith no checkpoint. AnewSessionIdthat already has a checkpoint is refused.- A step the history does not have (the session is unknown, the step never
ran, or it was dropped past
historyLimit) throws anSDKErrorwith codeLOUSHO_CHECKPOINT_NOT_FOUND, whose message lists the steps that are kept. A store withouthistory()cannot fork (ConfigurationError). compareTrajectories(a, b)takes two checkpoints or two transcripts and returns each run’s steps (one per assistant turn: its text and tool calls with their results),divergedAt(the first step that differs, tool call ids aside) anddrift, the tool-order, argument, step-count and finish-reason differences in the same shape aslousho eval --drift.
Resuming with a changed agent
Every checkpoint and approval snapshot carries an agent fingerprint: the model id, each tool’s name with a hash of its JSON input schema, and a hash of the system instructions (plus a short SHA-256 over all of them, computed with Web Crypto). Functions and other values that cannot be serialized (a tool’sexecute, hooks, callbacks) are not part of it, so changing only code never
counts as a change. The parts are stored one by one so a mismatch can say what
changed.
When a run is resumed (agent.resume(id), session.resume(), a later
execute() on an unfinished sessionId, agent.approvals.resolve() and the
streamed forms), the resuming agent’s fingerprint is compared with the saved
one, before any model call or tool runs. onAgentDrift on createAgent() (and
on AgentExecutor.execute() / the resumeAfterApproval() export) decides what a
difference does:
A pending tool call whose tool no longer exists (the model’s last turn called
it and it has no result yet, or the human approved it) always rejects with
LOUSHO_RESUME_TOOL_MISSING,
whatever the option.
- Checkpoints and snapshots written before this existed have no fingerprint and resume exactly as before, with no warning.
- A run of a dynamic agent saves the
ctxand the model it chose in the checkpoint (runConfig), and a crash resume uses them: the saved model, and tools and instructions resolved again with the savedctx(before, it resolved withinput: []and nometadata). - The fingerprint covers the agent’s own tools, model and instructions, not
the tools that
subagentsandskillsadd. A sub-agent that pauses for an approval has its own fingerprint in its nested snapshot. On resume, each paused level (the lead, the sub-agent, and a sub-agent of that sub-agent) is compared with its current definition fromsubagents, under the lead’sonAgentDrift. With'warn'theagent.driftevent carriessubagent. When a sub-agent’s check rejects, the lead’s approval and its'awaiting-approval'checkpoint are put back, so fixing the sub-agent and resolving again works. A remote sub-agent is not compared: its pause is kept on the remote. - A run resumed after an approval continues with the instructions it started with (they are in its transcript), so its later checkpoints record those.
Advanced: the executor API
createAgent() is built on AgentExecutor, which stays available when you
want to own the wiring: the provider, the ToolRegistry, the stores and the
resume calls are all explicit (see The executor API). Each snippet below says which createAgent()
call it corresponds to.
AgentExecutor.execute() with a sessionId and a checkpointStore is
agent.send(message, { sessionId }):
SessionAwaitingApprovalError on a paused session, as send() does:
resumeAfterApproval() function is what
agent.approvals.resolve() calls:
AgentExecutor.fork() is agent.fork(); execute() with input: [] on the
fork’s sessionId is agent.resume(fork.sessionId):
Compatibility with stored data
- Checkpoints written before
statusexisted are treated as'in-progress'(before this change, finished runs deleted their checkpoint, so only unfinished runs were ever stored). - Sessions saved before a store kept a history have an empty history until their next save.
- Approval snapshots written before
remainingToolCallsexisted resume as they used to: the calls after the paused one do not run. Each of them gets an error result saying it was not run, so the transcript stays valid.
Limits
- A
sessionIdidentifies one logical conversation. Concurrentagent.send()/agent.stream()calls with the samesessionIdon one agent store run one after another, in call order, in this process. Two processes (or two store objects over the same data) on the same id at the same time are refused withLOUSHO_SESSION_BUSYby the run lease (best effort on KV, see Run leases); with a custom store that has no leases, the last save wins. - A run id (
sessionIdofsend()/stream()) is 1-200 characters of letters, digits,_,-and., not starting or ending with., on every store, and may not end with.turn-<n>or.paused(a session’s own checkpoint keys) or.fire(a schedule’s fire claim). Anything else fails withLOUSHO_SESSION_ID_INVALID. A session id (agent.session({ id }), the routes’sessionId) has a stricter rule: 1-128 characters ofA-Za-z0-9_-, with no.. - If you call
resumeAfterApproval()(the function behindagent.approvals.resolve()) without thecheckpointStore, the session keeps its'awaiting-approval'mark and latersend()(orexecute()) calls keep throwingSessionAwaitingApprovalError; pass the store (or delete the checkpoint).agent.approvals.resolve()passes it for you. - Workers KV is eventually consistent across locations - see Deployment for the caveat.
- The spend of a sub-agent still running when its lead crashed is kept for in-process sub-agents only, as of its last model call or tool result: a remote sub-agent reports its usage when it ends, so an interrupted one’s calls are not counted, and a background sub-agent still running after the lead paused or finished does not write the lead’s checkpoint. Their limits are checked against a sub-agent’s usage only once it returns.