AgentExecutor.execute() and FlowExecutor.execute() emit spans through a
TraceExporter. Span names and attributes follow the
OpenTelemetry GenAI semantic conventions,
so GenAI-aware observability backends can recognise them without custom
mapping.
Stability. The GenAI conventions are inDevelopmentstatus (not stable). Attribute names have changed between versions of the spec before (for examplegen_ai.systembecamegen_ai.provider.name) and may change again. This SDK follows the spec as retrieved on 2026-10-01 from the semantic-conventions-genai repository (where the conventions now live). Pin your SDK version if dashboards depend on these names.
Quick start
Pass anexporter to createAgent(); every run of the agent (send(),
stream(), session turns, resume() and runs continued by
agent.approvals.resolve()) is traced, sub-agents included. To keep traces
on disk and browse them with npx lousho traces, use fileTraceExporter()
(see Local traces). The vendor/ model prefix chooses the
provider; with OpenRouter use openrouter/<vendor>/<model> (e.g.
openrouter/openai/gpt-4o-mini).
onSpanStart and onSpanEnd is an exporter. The run does not
wait for them: a hook that throws, or an async one whose promise rejects, is
reported once per exporter with console.warn and otherwise ignored.
Advanced: the executor API
With the executor, pass the exporter toAgentExecutor.execute():
createOtelTraceExporter() from
lousho/otel (needs the optional peer dependency
@opentelemetry/api and a registered TracerProvider). It carries span kind
and error status over to OpenTelemetry. See examples/tracing
(npm run example:tracing:console, npm run example:tracing:otel, which also prints metrics).
Span tree
handoff and approval.wait are marker spans: startTime === endTime, so a
consumer can mark the handoff target as the run’s active node (lousho.handoff.from
/ lousho.handoff.to) and a paused call’s tool node as waiting
(lousho.approval.id / lousho.approval.kind / lousho.approval.expires_at)
without holding a span open across the wait.
Each send() / stream() call is a trace of its own unless you parent it.
To put several runs in one trace, pass a span id of yours as parentSpanId;
the run’s invoke_agent span becomes its child, and its cost rolls up into it:
agent.approvals.resolve()) is a trace of its
own: its invoke_agent span holds the approved call’s execute_tool span
(first, as the call runs before the model is called again), then the chat
spans of the rest of the run. A rejected call has no execute_tool span.
A flow run (FlowExecutor.execute) produces:
parentSpanId
(withSpan hands the span to its callback); the flow span then becomes its
child.
Attributes
Constants for every name live insrc/execution/semconv.ts (exported as
GenAiAttr, GenAiOperation, GenAiMetric, ErrorAttr, FlowAttr, SdkAttr, LegacyAttr).
Agent run: invoke_agent {agent name}
Summing token usage over spans
The totals oninvoke_agent spans count tokens that the run’s chat spans
already carry, and a lead run’s total also counts its sub-agents’ runs. A
backend that adds up gen_ai.usage.* over every span of a trace (some
OpenTelemetry LLM cost dashboards do) therefore counts each token at least
twice, and once more per level of sub-agent nesting: a lead with one sub-agent
whose model calls used 33 + 7 input tokens shows 40 on the chat spans, 7 on
the sub-agent’s invoke_agent span and 40 on the lead’s, 87 in all.
To get the true total, sum chat spans only (gen_ai.operation.name = chat),
read the top-level invoke_agent span alone, or use the
gen_ai.client.token.usage metric. If the backend cannot filter, pass
createOtelTraceExporter({ agentUsageTotals: false }): the usage attributes
then stay on chat spans only (lousho.cost_usd stays on every span).
Model call: chat {model} (CLIENT)
Tool call: execute_tool {tool}
Guardrail check: guardrail {name}
One span per guardrail check, under the run’s invoke_agent span (input and
output checks) or the checked call’s execute_tool span (tools checks). The
span carries ids and outcome only — never the checked text or the trip
reason, which may hold user data (a trip is already reported by the
guardrail.tripped event).
Flows
The GenAI spec defines no convention for flow nodes, so node spans use thelousho.flow.* namespace. The run span uses the spec’s workflow convention.
Errors
A failed span getserror.type and an error span status (OpenTelemetry
ERROR, message = the error message). error.type is the thrown error’s
name (_OTHER when a non-Error is thrown). A tool call that returns an
error to the model (isError) is marked error.type = tool_error.
Metrics
createOtelTraceExporter() also records the OpenTelemetry GenAI client
metrics through @opentelemetry/api’s metrics API, using the global
MeterProvider (metrics.getMeter(...)), or the meter option:
Both use the spec’s recommended bucket boundaries. They follow the
gen-ai-metrics page of semantic-conventions v1.40 (Development stability):
the newest conventions split the token histogram into per-type counters,
which this SDK does not emit yet. Agent and flow spans are not counted in the
duration metric, only the model and tool calls inside them.
MeterProvider, OpenTelemetry’s no-op meter discards
the records. The metrics API lives in @opentelemetry/api, so nothing is
recorded (and nothing is loaded) when that optional peer is not installed.
The cost is an estimate from the price table (see registerModel), not a bill.
Message and argument content is opt-in
Prompts, model output and tool arguments/results are sensitive and large, so thegen_ai.* content attributes are never recorded by default. Opt in
per agent with createAgent({ exporter, captureContent: true }).
The exception is a local collector: when LOUSHO_OBSERVE points at loopback
(1/true, or a URL whose host is 127.0.0.1, localhost or ::1;
a localhost port that an SSH tunnel forwards to a remote daemon counts as
loopback too, so set captureContent: false if that daemon must not get content),
createAgent defaults captureContent to true, so the daemon’s trace view
shows the prompt, the model output and the tool arguments and results that
explain a failure. Only the daemon gets that content, and it stays in the
daemon’s db on your machine: the agent’s own exporter (fileTraceExporter()
files, an OTLP or vendor exporter, the built server’s OTel export) still sees
every span without it, unless the agent sets captureContent: true or the
OTel content variable below is true. Turn it off with captureContent: false;
a collector beyond loopback keeps the opt-in default. The default applies to
agents built with createAgent.
Secrets in captured content
Captured content goes throughredactContent before it is recorded. Unset or
true (the default) runs a built-in scrubber over message text, system
instructions, tool-call arguments and tool results, and replaces API-key shapes
with [redacted]:
The same scrubbing (or your function) runs on the error text recorded on a span,
error and the error status message, so a token in a thrown error’s message is redacted too; error.type (the class name) is kept. redactContent: false records error text as is.
Each shape needs a long token after its prefix (16+ characters for sk-, lsv2_,
gh*_, github_pat_, Stripe and Bearer; 10+ for Slack; 35 for Google; JWT parts of 8+), so
words like sk-learn or task- are left alone. Bearer is case-insensitive, so a long
word after “bearer” in prose can be redacted too. The scrubber catches common key formats,
not every secret: a password typed into a prompt is still recorded, so set
captureContent: false if content must never reach the local db.
Pass a function to use your own redaction instead of the scrubber. It gets each
piece of plain text (before JSON encoding) and returns what to record; if it
throws, that text is recorded as [redacted] and the run carries on:
redactContent: false records content as is (and also records the deprecated
input/prompt/args/result attributes, see below).
Advanced: the executor API
With the executor, opt in per run:captureContent). When
captureContent is not set, the OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
environment variable (the name the spec uses as its example) turns it on.
An explicit captureContent: false always wins.
With it on, content is recorded as JSON strings (the spec’s fallback for
spans) in the spec’s message schema:
Deprecated attribute names
Before the GenAI conventions, spans used ad-hoc names and the span namesagent.run, llm.generate and tool.call. The old attribute names are still
emitted next to the new ones so existing dashboards keep working, but they are
deprecated and will be removed in a future major version. Span names
changed (to what the spec requires); update queries that matched on them.
With
createOtelTraceExporter({ legacyAttributes: false }) the OpenTelemetry
spans carry only the GenAI names, not the deprecated ones next to them (fewer
attributes per span); the default true keeps both.
The deprecated content attributes (input, prompt, args, result) are
off by default: redactContent defaults to true (a function counts as true
here). Pass redactContent: false to record them as before; captureContent
governs only the gen_ai.* content attributes.
Local traces
fileTraceExporter() from lousho/traces writes each run to a
file, and lousho traces shows the files in the terminal. No OpenTelemetry
backend, no extra package.
--dir D reads another directory (default .lousho/traces),
--limit N lists N runs (default 20), --json prints the summaries or the
spans as JSON. An id prefix that matches several runs lists them and exits 1.
Colour is off when NO_COLOR is set or the output is not a terminal. The
token counts and the cost are the sums of the run’s chat spans, sub-agents’
included (background ones too); the cost is blank (-) when a model’s price is
unknown.
Files. One file per run at <dir>/<YYYY-MM-DD>/<traceId>.jsonl (the run’s
local start date; the trace id is the id of its invoke_agent span), so old
traces can be deleted by day. Each line is one finished span, written as it
ends:
startTime and endTime are milliseconds since the epoch; attributes are
the span attributes above, unshortened. A span that never ended (the process
died) is not in the file. If the directory cannot be written, the exporter
warns once and the run goes on. listTraces({ dir, limit }) and
readTrace(idOrPrefix, { dir }) read the files from code.
The files hold what the spans hold. By default that is no message or tool
content: the deprecated input, prompt, args and result attributes are
recorded only with redactContent: false (see
Deprecated attribute names), and
captureContent: true adds the gen_ai.* content. Keep .lousho/ out of
version control; projects made by lousho init ignore it.
lousho dev and lousho chat write these files when you pass --traces
(into .lousho/traces) or --traces=<dir>, for a spec file, an agent
directory or a module, so a spec-file agent needs no code to be traced. It is
opt-in on purpose: the files persist span timings, token counts and costs,
and prompt and tool content only if the agent sets redactContent: false or
captureContent: true, so nothing is written to disk unless you ask. The spec file format has no
traces setting.
The observe daemon
lousho observe start runs a local collector on http://127.0.0.1:7777
(loopback only, no auth needed). When LOUSHO_OBSERVE is set, every agent
built on this SDK reports spans to it automatically — 1, true, on or yes
emits to the default URL, an http:///https:// URL emits there, and
0, false, off, no, none, disabled or empty (any case) turn reporting
off. Any other value is treated as a typo: one warning naming LOUSHO_OBSERVE
is printed per process and reporting stays off. LOUSHO_OBSERVE_PROJECT labels
the rows; LOUSHO_OBSERVE_TOKEN sends a bearer token for daemons bound
beyond loopback (--require-token plus lousho observe token create).
LOUSHO_OBSERVE_PROJECT also decides where an agent lands in
Lousho Studio: set it to a workspace id (shown in Studio’s
Settings → Workspaces) and the agent’s observed card appears under that
workspace; unset, the project name is the package name and the agent appears
under Elsewhere. Studio reads the same collector - it attaches to a
running daemon or starts one itself on loopback (see
The collector) - so a deployed agent that
reports to Studio’s collector shows up with no Studio-specific code.
Security. Because the loopback daemon takes no token, it refuses requests
a web page in your browser could forge. On every /api/* route:
- a daemon bound to loopback answers only loopback
Hostnames (localhost,*.localhost,127.x.y.z,[::1]), any otherHostgets 403, which defeats DNS rebinding; - a write (anything but
GET/HEAD/OPTIONS) whoseOriginis not a loopbackhttp(s)origin (includingOrigin: null), or that carriesSec-Fetch-Site: cross-site, gets 403; - a write that has a body must send
Content-Type: application/json, anything else (text/plain, form posts) gets 415.
Origin and always
send JSON, so they are unaffected; a custom client that writes to the daemon
must send application/json too. On a daemon that requires a token, the 401
check comes first.
One SQLite file (default ~/.lousho/observe.db, WAL mode) holds the fleet’s
observations, sessions, eval scores, prompt versions and shared context.
npx lousho-studio renders it (see Lousho Studio):
Traces and Trace (trace trees with latency waterfalls), Threads, Monitoring
(a live fleet monitor with heartbeat-derived status — a crashed agent’s open
sessions flip to interrupted — over per-agent charts of runs, p95 latency,
error rate, cost and tokens), Scores, Datasets, Evaluators, Annotation
queues, Prompts and Context Hub. The Studio’s nav shows whether it can reach
the daemon. Alerts on down transitions,
error-rate thresholds and score regressions (see
Continuous score) POST to --webhook URLs or an
alerts.json next to the db.
lousho improve --watch posts to the same alerts.json webhooks:
improve.pending, improve.applied, improve.rolled_back,
improve.guard_failed, improve.paused and improve.canary. details holds
{ agent, dir, from, to, reason, cycle }, plus card, the local path of a
pending proposal. A regression the monitor or a scheduled guard saw but did
not roll back (automatic rollback off, or refused) is improve.guard_failed,
posted once per revision. The reason is plain text: links, images and markup are
stripped. Each event is also one stderr line, and a webhook that fails never
fails the cycle.
The sweep also records each agent.score_regression in the db’s
score_regressions table, with or without webhooks. A running
lousho improve --watch polls it every tick and starts a cycle with
trigger: 'score' for a regression of the agent’s current revision, instead
of waiting for new failed runs; every budget, the cooldown and the kill switch
still apply. The daemon never writes to an agent directory. A watcher lists
itself in watchers.json beside the db while it runs (informational: the sweep
records every regression, once per agent and revision, across daemon restarts).
A score cycle learns from the revision’s failed and low-scored runs; a
regression with neither ends idle. The score_regressions table is a schema
migration, so once this daemon has opened a db, a process on an older lousho
refuses it with LOUSHO_STORAGE_FAILED; upgrade every process that shares the
db. See Running it continuously.
Insights averages runs/day over the days an agent was active in the window
(from the later of the window start and the agent’s first sighting), so an
agent first seen today with 12 runs reports 12/day, not 12/7. The per-bucket
series (GET /api/v1/insights series, hour or day buckets per agent) carries
runs, errors, cost_usd, input_tokens, output_tokens,
latency_ms, plus p95_latency_ms (over the bucket’s finished runs; no row
when none finished) and error_rate (errors / runs). Those two are stored per
bucket, not summed. A db written by an older daemon lacks them for past
buckets: run lousho observe rebuild-rollups once to backfill.
The daemon also serves two stores agents can use at runtime:
- Prompt registry.
createAgent({ prompt: 'name@version' })(or'name@label') orLOUSHO_PROMPTfetches instructions from it at build (@latestand labels resolve on the daemon; an unreachable daemon falls back toinstructions; with noinstructions, the run fails withLOUSHO_PROMPT_UNRESOLVED). A failed fetch is not cached for good: runs in the next 30 seconds resolve at once (usinginstructions, or failing), then the next run retries it, so a daemon that was briefly down at build recovers without a restart. That one run waits for the retry fetch, up to 2.5 seconds. Every run logs the exact version used.lousho prompt pushpublishes a new version; a push whose text and variables equal the latest version’s mints nothing (200 { unchanged: true, version }). A label (prod,staging) points at exactly one version:lousho prompt label mira prod 3(orPUT /api/v1/prompts/:name/labels/:labelwith{ version }) moves it,push --labelsmoves it onto the pushed version, andGET /api/v1/prompts/:name/:reftakes an integer,latestor a label. A label ref takes the same path as a version: unresolvable means theinstructionsfallback orLOUSHO_PROMPT_UNRESOLVED, with the same 30-second retry window. - Context hub.
hub()/hubTools()let agents publish and query namespaced key-values scopedprivate(the writing agent only) orfleet(every agent on the daemon).
POST /playground/run, see
Agent Forge) report as agent
playground; a run started from a registry prompt carries the prompt’s ref
(name@label, name@3) in its root span’s lousho.prompt.ref attribute.
Reporting is fire-and-forget and never blocks a run: when the daemon is
down, the failed batch is requeued and retried with exponential backoff (1 s,
2 s, 4 s … capped at 30 s), so a daemon restart mid-run loses nothing. A
backlog drains in batches one right after another, each at most 1000 events
and 4 MB of JSON (under the daemon’s per-request caps of 1000 events and
5 MB). Only these are dropped and counted in the reporter’s dropped:
events evicted past the buffer cap (maxBuffered, default 5000 events, or
32 MB of serialized events, whichever comes first; oldest first); events
past the newest 16 MB when a crash flushes; events still unsent after the exit flush’s final attempt; a single
event too large for any request (over 5 MB serialized, for example with
content capture on); and a batch the daemon refuses outright (400/413),
which is never retried.
An open run’s root start (queued, its end not yet) is kept through both cuts:
the buffer trim and the crash flush’s 16 MB slice evict other events first, so
a run whose children overflow the buffer still appears. Only when nothing else
is left to evict do open runs’ starts go too, oldest first, so memory stays bounded.
The count rides on every heartbeat as meta.dropped (cumulative for the process),
and each row of the daemon’s /api/v1/agents fleet list exposes the latest one as
dropped: a number, or null for an agent on an older SDK that sent none.
A process that dies of an uncaught exception or an unhandled rejection still
reports its last run: the reporter watches uncaughtExceptionMonitor and
sends what it holds (the newest 16 MB of it, open runs’ root starts first), oldest first, holding up the
crash for up to 1 s per reporter. Node’s error output and exit code stay as they were, and a script
that survives the throw (its own uncaughtException handler, or a capture
callback such as the REPL’s) keeps reporting. An explicit process.exit() or
a SIGKILL still loses what is buffered.
Every run is visible from the moment it starts: the reporter sends a run’s
root span (the top invoke_agent, the one with no parent) as soon as it
opens, without waiting for the 5 s interval, and Studio lists that trace as
running. The run’s end replaces that row with the final status, usage,
cost and output, and counts once. So a process killed mid-run (SIGKILL,
power loss) still leaves its trace behind, and the trace flips to
interrupted when the agent stops heartbeating and the sweep marks it
down. Each reporter process also sends a random instance_id, because a
restart or a second process of the same agent reports under the same agent
id. Processes of one agent running side by side never interrupt each
other’s runs while they keep heartbeating, even when they share a hostname
and pid (--network host replicas are all pid 1). When a process of an
agent goes quiet while another process of it keeps heartbeating (a supervisor restarted it under a new pid, or one of
several workers died), the sweep marks the quiet process’s open runs
interrupted once it has missed about two heartbeats (the stale mark),
judged on that process’s own heartbeat interval and on the daemon’s clock. A
restart on the same host with the same pid (a container’s pid 1) marks them
sooner: at the first heartbeat of the new process (after its first) that
comes more than one heartbeat interval plus 10 seconds after the old
process’s last, if the old process has been silent since the new one
appeared. With no process of the agent left, the
agent goes down and its open runs are marked as above. A process that exits gracefully mid-run
(SIGINT, SIGTERM) marks its own open runs interrupted right away. An agent
with heartbeats turned off (heartbeatIntervalMs: 0) is never swept, so its
killed runs stay running. Child spans (model calls and tool calls)
are still sent only when they finish, so a killed run keeps the root and
any child that finished and was already sent.
Eval scores flow through the same pipe —
lousho eval --observe emits them even when LOUSHO_OBSERVE is unset.
Each case yields one <eval>/<check> row per assertion plus one <eval> pass/fail row (1 when every gate check passed, else 0).
Human feedback is a score too. POST /api/v1/scores with
{ "trace_id", "obs_id"?, "name", "value", "comment"? } (value a number or
null, name up to 200 characters, comment up to 10 000) adds one score with
source: "human" and answers 201 { "score_id" }. Each call adds a row, and
the score is attributed to the trace’s agent. An unknown trace_id (or an
obs_id that is not in that trace) is a 404, an empty name a 400. It
takes the same token as ingest, and GET /api/v1/scores?trace_id=... lists it.
Datasets
A dataset is a named set of examples to run evals over. Each example is aninput and an optional expected value, both any JSON value. Studio’s
Datasets screen lists and creates them, and a
trace’s Add to dataset copies the trace in. The routes take the same token
as ingest:
A dataset row is
{ dataset_id, name, description, created_at, example_count }
and an example row is { example_id, dataset_id, input, expected, source_trace_id, created_at }, with expected and source_trace_id null
when unset. from_trace_id copies the trace’s root observation: its input
becomes input and its output expected. Stored text that is valid JSON is
copied as that value, and any other text as a JSON string. A trace whose root
has no captured input (content capture was off) is a 400. An unknown
dataset, trace or example is a 404 with code LOUSHO_REQUEST_INVALID. The
example body may be up to 1 MiB.
Retention never deletes datasets or examples. An example keeps its
source_trace_id after that trace is pruned.
Online evaluator rules
An evaluator rule says which new traces to score and how. Acode rule holds
the source of a function (trace) => number | { value, comment }, and a
judge rule holds a model and a rubric to grade by. The daemon runs code
rules on new traces (see Online evaluators); judge
rules run through Agent Forge (see Judge rules). Studio’s
Evaluators screen manages them. The routes take the same
token as ingest:
A rule row is
{ rule_id, name, enabled, filter, sample_rate, kind, config, created_at, updated_at }. The fields are:
name: 1-200 characters after trimming, unique.enabled: a boolean,trueby default.filter:{ "agent_id"?, "status"?, "name_contains"? },{}by default.statusisokorerror, and each value is a non-empty string.sample_rate: a number from 0 to 1,1by default.kindandconfig:codetakes{ "source" }(up to 64 KiB of text), andjudgetakes{ "model", "rubric" }(up to 200 and 20 000 characters).
kind or a config that does not match the
kind is a 400 with code LOUSHO_REQUEST_INVALID. An unknown rule is a
404 with the same code. updated_at moves on every PUT. The body may be up
to 128 KiB. Retention never deletes rules.
Online evaluators
When a trace’s root observation ends, the daemon runs every enabledcode
rule whose filter matches the root (agent_id, status, and
name_contains on its name). A rule runs when Math.random() < sample_rate,
so 0 never runs and 1 always does. Each run writes one score with
source: "online", name = the rule’s name, the root’s agent_id, and no
obs_id. A trace that already has an online score with that name is not scored
again.
The rule’s source runs as (<source>)(trace) in the isolate sandbox
(sandbox({ driver: 'isolate', timeoutMs: 2000 })). trace is a plain JSON
object:
observations holds the trace’s rows, oldest first, capped at 200. A row
whose attrs JSON is longer than 8192 characters carries
"attrs": { "truncated": true } instead. error_message is the first error
row’s attrs.error, even when that row is past the first 200. Rules match on
the root row alone, so a trace no rule’s filter keeps loads no other rows, and
the worker handles one trace per event-loop turn. The
function returns a finite number (the score’s value) or
{ "value": <number | null>, "comment"?: "<text>" } (the comment is cut at
1000 characters). Anything else scores value: null with comment
invalid_result. A run that fails scores value: null with the sandbox error
kind as the comment: timeout, oom, or driver. An error your code throws
carries no kind, so it reports as driver.
The isolate driver needs the optional isolated-vm package
(npm install isolated-vm). On Node 20 and later the daemon also needs
NODE_OPTIONS=--no-node-snapshot. Without them, code rules are skipped and
the daemon warns once.
Rules run off the ingest request, one trace at a time, so ingest stays fast
even when an evaluator is slow or fails. Up to 1000 traces wait in memory;
past that, new traces are dropped (and counted). Waiting traces are lost when
the daemon restarts, so those traces never get online scores.
POST /api/v1/evaluators/:id/test { "trace_id": "<trace-id>" } dry-runs a
rule on one trace and answers 200 { "result": { "value", "comment" } }. It
ignores enabled, filter and sample_rate, and it writes no score. An
unknown rule is a 404, and so is a trace that does not exist or has no root
observation. A bad body is a 400. Dry runs go one at a time: a second
request waits until the one in flight finishes. A judge rule is a
409 (judge evaluators run through Agent Forge). Without the isolate it
answers { "result": { "value": null, "comment": "isolate_unavailable" } }.
Judge rules (through Agent Forge)
The daemon holds no provider keys, so it cannot run ajudge rule itself.
When a judge rule matches a new trace (same filter, sample_rate and
“already scored” check as code rules), the daemon queues a job for that rule
and trace. Agent Forge (lousho studio) pulls the jobs, runs each judge with
the provider keys it already has (a key saved in Studio settings wins over the
provider’s env var, as in the playground), and posts the result back. The
daemon writes it as the same source: "online" score. So judges run only
while Studio is running. Until then the jobs wait.
- At most 500 jobs wait at once (pending plus claimed). Past that, new jobs are dropped and counted. Jobs live in memory, so a daemon restart loses them.
- The judge is one model call to the rule’s
model(provider/model, ormock). The system prompt is a fixed grading instruction plus the rule’srubric. The user message holds the trace’s name, status, input, output and error, each cut at about 8 KB. The model must reply with a JSON object{ "score": <0..1>, "comment": "<short reason>" }. Agent Forge reads the first JSON object in the reply. A reply with no valid score scoresvalue: nullwith commentinvalid_judge_output. A provider error (a missing key, for example) scoresvalue: nullwith commentjudge_error: <message>. - The judge call makes no trace of its own, so it never triggers evaluators.
LOUSHO_OBSERVE_TOKEN when it is set):
Agent Forge claims up to 5 jobs every 2 seconds, with a 60-second lease, and
runs them one at a time. Each judge call is cut off after 20 seconds, or
sooner if less of the lease is left (a call cut off scores
value: null with
comment judge_error: timed out after <n> ms). A claimed job whose lease has
run out before its turn is not run: it expires and is claimed again later. A
daemon that is down only pauses the loop, and it logs once per outage. The
dry run below has the same 20-second cap.
To dry-run a judge rule, use Agent Forge: POST /observe/api/v1/evaluators/:id/test
{ "trace_id" } on the Studio server runs the judge on that trace and answers
200 { "result": { "value", "comment" } } without writing a score. For a
code rule it forwards the request to the daemon unchanged. The daemon’s own
/test still answers 409 for a judge rule.
Annotation queues
An annotation queue is a named list of traces for a person to review one by one. Completing an item writes its scores as human scores, the samesource: "human" rows POST /api/v1/scores writes, so they show up in
GET /api/v1/scores and on the trace. Studio’s
Annotation queues screen creates queues
and is the review view. The routes take the same token as ingest:
A queue row is
{ queue_id, name, rubric, created_at, counts: { pending, done, skipped } }. An item row is { item_id, queue_id, trace_id, status, reviewer, created_at, done_at }, where reviewer and done_at are null until the
item is finished. :id is the queue_id.
name: 1-200 characters after trimming, unique.rubric: up to 50 entries of{ "name", "description"?, "min"?, "max"? },[]by default. Names are 1-200 characters and unique, a description is up to 2000 characters, andminmay not exceedmax.
scores, up to 50 entries of { "name", "value", "comment"? } where value is a number or null, an optional reviewer (up
to 200 characters) and status, which is done (the default) or skipped.
Each score is written on the item’s trace, and the status, reviewer and
done_at are set in the same transaction. When the queue has a rubric, every
score name must be one of its names and the value must sit within that
entry’s min and max, or the answer is a 400. A skip with scores is a
400. Completing an item that is already done or skipped is a 409 and
writes nothing. An item that is not in the queue is a 404.
next does not claim or lock the item it returns. Asking twice returns the
same item until someone completes it. The queue is meant for one reviewer
working locally, so two reviewers on one queue can score the same trace.
Retention never touches queues or items. If an item’s trace is pruned, the
item keeps its trace_id and next still returns it. Completing it with
scores is then a 404 naming the trace, and the item stays pending, so skip
it instead. Bad bodies and unknown fields are a 400 with code
LOUSHO_REQUEST_INVALID, and an unknown queue is a 404 with the same code.
Bodies may be up to 256 KiB.
Retention
The daemon keeps 30 days of runs by default. On its first sweep after start, and then once a day, it deletes every trace whose observations have all finished and whose last one ended more than--retention-days ago (default
30; --retention-days 0 keeps everything). A whole trace goes or stays: one
still-open observation (end unset, a run in progress) keeps the entire trace,
however old. With a trace go its scores and its prompt-use links; scores older
than the window whose trace has no observations left go too. A session is
deleted only when it is not active, ended (or, with no end, started) before
the window, and no remaining observation references it, so an old thread
that is still active is never pruned. Idle sessions end on their own (see
Idle sessions), so finished conversations do age out.
Pruning never touches the Insights rollups (charts for pruned days stay),
the prompt registry, hub entries, the agent registry or tokens. The daily
prune does not VACUUM, which would stall ingest; to reclaim disk space, run
lousho observe prune --older-than 30d. It prunes now with the window you
give (whole days only), prints how many observations, scores and sessions it
deleted, then runs VACUUM. It is safe while the daemon is running on the
same db.
Export
lousho observe export [--since 7d] [--agent X] [--out file] dumps traces as
JSONL for offline analysis or another tool. Each line is one trace:
{ "trace": <trace row>, "observations": [...], "scores": [...] }, the same
shapes as the trace list and detail APIs, oldest trace first and not capped.
--since takes <N>d, <N>h or <N>m back from now; --agent matches an
agent_id, or an agent’s name when it names exactly one. Lines go to stdout (the
count goes to stderr) unless --out names a file, which is overwritten. It
reads the db directly and works without a running daemon. Captured content is
exported as stored, so treat the file like the db.
Continuous score
Each finished run of an agent becomes one point in a series, oldest first:- the mean of the run’s finished online evaluator scores, when it has any;
- otherwise 0 when the run is in the failure report (a failed call, no answer, an interrupted process or a low score), and 1 when it is not.
lousho.harness.revision, so a series can be cut to one revision.
The trajectory gate from the Self-Healing Harness paper judges the shape of
the series rather than one number. It works on the rolling mean of 10
points:
- The first full window (10 points) sets a running peak.
- A later window mean at least 0.02 above the peak is a gain, which raises the peak.
The significance check is the paper’s corroboration step. A drop of 0.15 is
only one or two failures in a window of pass/fail runs, so the shape alone
would fire on most healthy agents. The test depends on the values:
- for pass/fail values it compares pass rates (Newcombe’s interval from Wilson bounds);
- otherwise it compares means (Welch’s t).
regression about 5% of the time.
Before the first full window the gate reports ok with ready: false.
The daemon does not judge on every run, because repeating the test would add
up to false alarms. It waits until an agent has 20 finished runs. After
that, each time 10 new runs have arrived, it runs the gate over the last 200
points at a stricter z of 1.96. The long lookback keeps the old baseline in
view when a lasting drop fills recent runs. It raises an
agent.score_regression alert only after three judgements in a row read
regression.
These are simulated figures for pass/fail agents (1,000 seeded streams each).
A stable agent with a pass rate between 0.6 and 0.95 raised a false alert in
under 1% of 1,000-run stretches.
When 200 runs at the old rate were followed by 100 at the new one, the fall
was caught within those 100 runs this often:
The alert fires at most once per agent per revision. The revision is that of
the newest run, and
null when that run has no stamp.
The alert’s details carry:
revision,peak,meananddrop;since_gainand the number ofruns;recent: a summary of the newest 10 runs.
lousho observe score [--agent X] [--revision R] [--limit N] [--json] prints
the same view for each agent from the db:
- runs, mean, and the pass rate with its one-sided 95% Wilson lower bound;
- pass^3, the chance that three independent runs all pass;
- the gate state.
--revision, the line covers every revision in the window and names
the latest one. It reads the db directly and works without a running daemon.
Trace list API
GET /api/v1/traces returns { traces, total, limit, offset }, one row per
trace, newest first. Query parameters (all optional):
An unknown
sort or order, a non-numeric min_duration_ms, or a prompt
with an empty name or a non-integer version answers
400 with code LOUSHO_REQUEST_INVALID.
Each row has trace_id, name, agent_id, session_id, start, end,
status, the counts observations, generations, tools and errors, the
sums input_tokens, output_tokens and cost_usd, and:
agent_name: the agent’s name from its heartbeat, ornull.model: the first model call’s model, ornull.error_message: the first failed observation’serrorattribute (for exampleupstream 529 overloaded), ornull.input_preview/output_preview: the first 200 characters of the run’s stored input and output, ornullwhen none was captured.
Sessions (threads) API
GET /api/v1/sessions returns { sessions, total, limit, offset }, newest
first (started_at descending). Query parameters (all optional): agent_id,
status (exact match), from / to (session start in ms epoch, half-open
from <= started_at < to), and limit / offset (1-500, default 50). Each
session row is { session_id, agent_id, agent_name, status, started_at, ended_at, traces, input_tokens, output_tokens, cost_usd, last_input_preview }:
traces counts distinct trace ids, tokens and cost sum the session’s
observations, and last_input_preview is the first 200 characters of the root
input of its latest trace. status is active, ended (idle, see below) or
interrupted (its agent went down).
GET /api/v1/sessions/:id returns { session, traces } with the traces
(same rows as the trace list) ordered by start ascending, capped at 500; an
unknown id is a 404 with code LOUSHO_REQUEST_INVALID. Studio’s
Threads screen reads these two routes.
Idle sessions
The SDK sends no “session closed” event: a session row exists because its runs carry its id, so the daemon decides when a conversation is over. Each sweep (every 30 s) marks a sessionended when none of its runs is still open
(a run in progress keeps it active however long it takes) and nothing in it
has started or finished for --session-idle-minutes (default 30; 0 turns
this off). A run marked interrupted (its process died while the agent kept
running) no longer counts as in progress; it counts as activity at the moment
it was interrupted. Its ended_at is the time of that last activity, not the time of
the sweep. A session no run references falls back to when the daemon last
updated it. Sessions in a db written by an older daemon close on the first
sweep that finds them idle.
A new run in an ended session makes it active again (ended_at follows
its runs again), and so does new activity in an interrupted one. A
re-sent or reordered event for activity at or before the session’s
ended_at does not reopen it. The idle pass never touches interrupted
sessions. Once ended, a session is pruned by retention like
any other finished session, so Threads shows only live conversations as
active. The idle window is judged against the timestamps your agents
report, so an agent clock far behind the daemon’s can end a session early;
its next run reopens it.
The built server
The serverlousho build --target=node-server or --target=docker generates
exports traces with no code when the standard OTEL_EXPORTER_OTLP_ENDPOINT
(or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT) variable is set where it runs. It
registers an OpenTelemetry tracer provider that sends spans over OTLP/HTTP,
and gives the agent createOtelTraceExporter(), so every /chat turn arrives
as the invoke_agent / chat / execute_tool tree above. An exporter the
agent’s config already has (an agent directory’s or a module’s) keeps
running: both receive every span.
The OpenTelemetry SDK is not bundled. Install these packages next to
dist/server.js, or add them to the build’s package.json dependencies
before you build the docker image:
- The server imports them at start-up only when the variable is set. When one is missing it logs which packages to install and serves without exporting traces.
service.nameisOTEL_SERVICE_NAMEwhen set, otherwise the agent’s name.OTEL_RESOURCE_ATTRIBUTES,OTEL_EXPORTER_OTLP_HEADERS(for an API key) and the batch settings (OTEL_BSP_SCHEDULE_DELAY, …) work as the OTel SDK documents them.- On SIGINT or SIGTERM the server sends the spans it still holds before it exits.
- With neither variable set, nothing is imported and the server runs as before.