Skip to main content
AgentExecutor.execute() and FlowExecutor.execute() emit spans through a TraceExporter. Span names and attributes follow the OpenTelemetry GenAI semantic conventions, so GenAI-aware observability backends can recognise them without custom mapping.
Stability. The GenAI conventions are in Development status (not stable). Attribute names have changed between versions of the spec before (for example gen_ai.system became gen_ai.provider.name) and may change again. This SDK follows the spec as retrieved on 2026-10-01 from the semantic-conventions-genai repository (where the conventions now live). Pin your SDK version if dashboards depend on these names.

Quick start

Pass an exporter to createAgent(); every run of the agent (send(), stream(), session turns, resume() and runs continued by agent.approvals.resolve()) is traced, sub-agents included. To keep traces on disk and browse them with npx lousho traces, use fileTraceExporter() (see Local traces). The vendor/ model prefix chooses the provider; with OpenRouter use openrouter/<vendor>/<model> (e.g. openrouter/openai/gpt-4o-mini).
Any object with onSpanStart and onSpanEnd is an exporter. The run does not wait for them: a hook that throws, or an async one whose promise rejects, is reported once per exporter with console.warn and otherwise ignored.

Advanced: the executor API

With the executor, pass the exporter to AgentExecutor.execute():
For real OpenTelemetry spans, import createOtelTraceExporter() from lousho/otel (needs the optional peer dependency @opentelemetry/api and a registered TracerProvider). It carries span kind and error status over to OpenTelemetry. See examples/tracing (npm run example:tracing:console, npm run example:tracing:otel, which also prints metrics).

Span tree

handoff and approval.wait are marker spans: startTime === endTime, so a consumer can mark the handoff target as the run’s active node (lousho.handoff.from / lousho.handoff.to) and a paused call’s tool node as waiting (lousho.approval.id / lousho.approval.kind / lousho.approval.expires_at) without holding a span open across the wait. Each send() / stream() call is a trace of its own unless you parent it. To put several runs in one trace, pass a span id of yours as parentSpanId; the run’s invoke_agent span becomes its child, and its cost rolls up into it:
A run continued after an approval (agent.approvals.resolve()) is a trace of its own: its invoke_agent span holds the approved call’s execute_tool span (first, as the call runs before the model is called again), then the chat spans of the rest of the run. A rejected call has no execute_tool span. A flow run (FlowExecutor.execute) produces:
To nest a flow under an agent span, pass that span’s id as parentSpanId (withSpan hands the span to its callback); the flow span then becomes its child.

Attributes

Constants for every name live in src/execution/semconv.ts (exported as GenAiAttr, GenAiOperation, GenAiMetric, ErrorAttr, FlowAttr, SdkAttr, LegacyAttr).

Agent run: invoke_agent {agent name}

Summing token usage over spans

The totals on invoke_agent spans count tokens that the run’s chat spans already carry, and a lead run’s total also counts its sub-agents’ runs. A backend that adds up gen_ai.usage.* over every span of a trace (some OpenTelemetry LLM cost dashboards do) therefore counts each token at least twice, and once more per level of sub-agent nesting: a lead with one sub-agent whose model calls used 33 + 7 input tokens shows 40 on the chat spans, 7 on the sub-agent’s invoke_agent span and 40 on the lead’s, 87 in all. To get the true total, sum chat spans only (gen_ai.operation.name = chat), read the top-level invoke_agent span alone, or use the gen_ai.client.token.usage metric. If the backend cannot filter, pass createOtelTraceExporter({ agentUsageTotals: false }): the usage attributes then stay on chat spans only (lousho.cost_usd stays on every span).

Model call: chat {model} (CLIENT)

Tool call: execute_tool {tool}

Guardrail check: guardrail {name}

One span per guardrail check, under the run’s invoke_agent span (input and output checks) or the checked call’s execute_tool span (tools checks). The span carries ids and outcome only — never the checked text or the trip reason, which may hold user data (a trip is already reported by the guardrail.tripped event).

Flows

The GenAI spec defines no convention for flow nodes, so node spans use the lousho.flow.* namespace. The run span uses the spec’s workflow convention.

Errors

A failed span gets error.type and an error span status (OpenTelemetry ERROR, message = the error message). error.type is the thrown error’s name (_OTHER when a non-Error is thrown). A tool call that returns an error to the model (isError) is marked error.type = tool_error.

Metrics

createOtelTraceExporter() also records the OpenTelemetry GenAI client metrics through @opentelemetry/api’s metrics API, using the global MeterProvider (metrics.getMeter(...)), or the meter option: Both use the spec’s recommended bucket boundaries. They follow the gen-ai-metrics page of semantic-conventions v1.40 (Development stability): the newest conventions split the token histogram into per-type counters, which this SDK does not emit yet. Agent and flow spans are not counted in the duration metric, only the model and tool calls inside them.
Without a registered MeterProvider, OpenTelemetry’s no-op meter discards the records. The metrics API lives in @opentelemetry/api, so nothing is recorded (and nothing is loaded) when that optional peer is not installed. The cost is an estimate from the price table (see registerModel), not a bill.

Message and argument content is opt-in

Prompts, model output and tool arguments/results are sensitive and large, so the gen_ai.* content attributes are never recorded by default. Opt in per agent with createAgent({ exporter, captureContent: true }). The exception is a local collector: when LOUSHO_OBSERVE points at loopback (1/true, or a URL whose host is 127.0.0.1, localhost or ::1; a localhost port that an SSH tunnel forwards to a remote daemon counts as loopback too, so set captureContent: false if that daemon must not get content), createAgent defaults captureContent to true, so the daemon’s trace view shows the prompt, the model output and the tool arguments and results that explain a failure. Only the daemon gets that content, and it stays in the daemon’s db on your machine: the agent’s own exporter (fileTraceExporter() files, an OTLP or vendor exporter, the built server’s OTel export) still sees every span without it, unless the agent sets captureContent: true or the OTel content variable below is true. Turn it off with captureContent: false; a collector beyond loopback keeps the opt-in default. The default applies to agents built with createAgent.

Secrets in captured content

Captured content goes through redactContent before it is recorded. Unset or true (the default) runs a built-in scrubber over message text, system instructions, tool-call arguments and tool results, and replaces API-key shapes with [redacted]: The same scrubbing (or your function) runs on the error text recorded on a span, error and the error status message, so a token in a thrown error’s message is redacted too; error.type (the class name) is kept. redactContent: false records error text as is. Each shape needs a long token after its prefix (16+ characters for sk-, lsv2_, gh*_, github_pat_, Stripe and Bearer; 10+ for Slack; 35 for Google; JWT parts of 8+), so words like sk-learn or task- are left alone. Bearer is case-insensitive, so a long word after “bearer” in prose can be redacted too. The scrubber catches common key formats, not every secret: a password typed into a prompt is still recorded, so set captureContent: false if content must never reach the local db. Pass a function to use your own redaction instead of the scrubber. It gets each piece of plain text (before JSON encoding) and returns what to record; if it throws, that text is recorded as [redacted] and the run carries on:
redactContent: false records content as is (and also records the deprecated input/prompt/args/result attributes, see below).

Advanced: the executor API

With the executor, opt in per run:
Flows take the same flag on their context (captureContent). When captureContent is not set, the OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true environment variable (the name the spec uses as its example) turns it on. An explicit captureContent: false always wins. With it on, content is recorded as JSON strings (the spec’s fallback for spans) in the spec’s message schema:

Deprecated attribute names

Before the GenAI conventions, spans used ad-hoc names and the span names agent.run, llm.generate and tool.call. The old attribute names are still emitted next to the new ones so existing dashboards keep working, but they are deprecated and will be removed in a future major version. Span names changed (to what the spec requires); update queries that matched on them. With createOtelTraceExporter({ legacyAttributes: false }) the OpenTelemetry spans carry only the GenAI names, not the deprecated ones next to them (fewer attributes per span); the default true keeps both. The deprecated content attributes (input, prompt, args, result) are off by default: redactContent defaults to true (a function counts as true here). Pass redactContent: false to record them as before; captureContent governs only the gen_ai.* content attributes.

Local traces

fileTraceExporter() from lousho/traces writes each run to a file, and lousho traces shows the files in the terminal. No OpenTelemetry backend, no extra package.
Then, from the project directory:
Options: --dir D reads another directory (default .lousho/traces), --limit N lists N runs (default 20), --json prints the summaries or the spans as JSON. An id prefix that matches several runs lists them and exits 1. Colour is off when NO_COLOR is set or the output is not a terminal. The token counts and the cost are the sums of the run’s chat spans, sub-agents’ included (background ones too); the cost is blank (-) when a model’s price is unknown. Files. One file per run at <dir>/<YYYY-MM-DD>/<traceId>.jsonl (the run’s local start date; the trace id is the id of its invoke_agent span), so old traces can be deleted by day. Each line is one finished span, written as it ends:
startTime and endTime are milliseconds since the epoch; attributes are the span attributes above, unshortened. A span that never ended (the process died) is not in the file. If the directory cannot be written, the exporter warns once and the run goes on. listTraces({ dir, limit }) and readTrace(idOrPrefix, { dir }) read the files from code. The files hold what the spans hold. By default that is no message or tool content: the deprecated input, prompt, args and result attributes are recorded only with redactContent: false (see Deprecated attribute names), and captureContent: true adds the gen_ai.* content. Keep .lousho/ out of version control; projects made by lousho init ignore it. lousho dev and lousho chat write these files when you pass --traces (into .lousho/traces) or --traces=<dir>, for a spec file, an agent directory or a module, so a spec-file agent needs no code to be traced. It is opt-in on purpose: the files persist span timings, token counts and costs, and prompt and tool content only if the agent sets redactContent: false or captureContent: true, so nothing is written to disk unless you ask. The spec file format has no traces setting.

The observe daemon

lousho observe start runs a local collector on http://127.0.0.1:7777 (loopback only, no auth needed). When LOUSHO_OBSERVE is set, every agent built on this SDK reports spans to it automatically — 1, true, on or yes emits to the default URL, an http:///https:// URL emits there, and 0, false, off, no, none, disabled or empty (any case) turn reporting off. Any other value is treated as a typo: one warning naming LOUSHO_OBSERVE is printed per process and reporting stays off. LOUSHO_OBSERVE_PROJECT labels the rows; LOUSHO_OBSERVE_TOKEN sends a bearer token for daemons bound beyond loopback (--require-token plus lousho observe token create). LOUSHO_OBSERVE_PROJECT also decides where an agent lands in Lousho Studio: set it to a workspace id (shown in Studio’s Settings → Workspaces) and the agent’s observed card appears under that workspace; unset, the project name is the package name and the agent appears under Elsewhere. Studio reads the same collector - it attaches to a running daemon or starts one itself on loopback (see The collector) - so a deployed agent that reports to Studio’s collector shows up with no Studio-specific code. Security. Because the loopback daemon takes no token, it refuses requests a web page in your browser could forge. On every /api/* route:
  • a daemon bound to loopback answers only loopback Host names (localhost, *.localhost, 127.x.y.z, [::1]), any other Host gets 403, which defeats DNS rebinding;
  • a write (anything but GET/HEAD/OPTIONS) whose Origin is not a loopback http(s) origin (including Origin: null), or that carries Sec-Fetch-Site: cross-site, gets 403;
  • a write that has a body must send Content-Type: application/json, anything else (text/plain, form posts) gets 415.
The SDK reporter, the CLI and the Studio proxy send no Origin and always send JSON, so they are unaffected; a custom client that writes to the daemon must send application/json too. On a daemon that requires a token, the 401 check comes first. One SQLite file (default ~/.lousho/observe.db, WAL mode) holds the fleet’s observations, sessions, eval scores, prompt versions and shared context. npx lousho-studio renders it (see Lousho Studio): Traces and Trace (trace trees with latency waterfalls), Threads, Monitoring (a live fleet monitor with heartbeat-derived status — a crashed agent’s open sessions flip to interrupted — over per-agent charts of runs, p95 latency, error rate, cost and tokens), Scores, Datasets, Evaluators, Annotation queues, Prompts and Context Hub. The Studio’s nav shows whether it can reach the daemon. Alerts on down transitions, error-rate thresholds and score regressions (see Continuous score) POST to --webhook URLs or an alerts.json next to the db. lousho improve --watch posts to the same alerts.json webhooks: improve.pending, improve.applied, improve.rolled_back, improve.guard_failed, improve.paused and improve.canary. details holds { agent, dir, from, to, reason, cycle }, plus card, the local path of a pending proposal. A regression the monitor or a scheduled guard saw but did not roll back (automatic rollback off, or refused) is improve.guard_failed, posted once per revision. The reason is plain text: links, images and markup are stripped. Each event is also one stderr line, and a webhook that fails never fails the cycle. The sweep also records each agent.score_regression in the db’s score_regressions table, with or without webhooks. A running lousho improve --watch polls it every tick and starts a cycle with trigger: 'score' for a regression of the agent’s current revision, instead of waiting for new failed runs; every budget, the cooldown and the kill switch still apply. The daemon never writes to an agent directory. A watcher lists itself in watchers.json beside the db while it runs (informational: the sweep records every regression, once per agent and revision, across daemon restarts). A score cycle learns from the revision’s failed and low-scored runs; a regression with neither ends idle. The score_regressions table is a schema migration, so once this daemon has opened a db, a process on an older lousho refuses it with LOUSHO_STORAGE_FAILED; upgrade every process that shares the db. See Running it continuously. Insights averages runs/day over the days an agent was active in the window (from the later of the window start and the agent’s first sighting), so an agent first seen today with 12 runs reports 12/day, not 12/7. The per-bucket series (GET /api/v1/insights series, hour or day buckets per agent) carries runs, errors, cost_usd, input_tokens, output_tokens, latency_ms, plus p95_latency_ms (over the bucket’s finished runs; no row when none finished) and error_rate (errors / runs). Those two are stored per bucket, not summed. A db written by an older daemon lacks them for past buckets: run lousho observe rebuild-rollups once to backfill. The daemon also serves two stores agents can use at runtime:
  • Prompt registry. createAgent({ prompt: 'name@version' }) (or 'name@label') or LOUSHO_PROMPT fetches instructions from it at build (@latest and labels resolve on the daemon; an unreachable daemon falls back to instructions; with no instructions, the run fails with LOUSHO_PROMPT_UNRESOLVED). A failed fetch is not cached for good: runs in the next 30 seconds resolve at once (using instructions, or failing), then the next run retries it, so a daemon that was briefly down at build recovers without a restart. That one run waits for the retry fetch, up to 2.5 seconds. Every run logs the exact version used. lousho prompt push publishes a new version; a push whose text and variables equal the latest version’s mints nothing (200 { unchanged: true, version }). A label (prod, staging) points at exactly one version: lousho prompt label mira prod 3 (or PUT /api/v1/prompts/:name/labels/:label with { version }) moves it, push --labels moves it onto the pushed version, and GET /api/v1/prompts/:name/:ref takes an integer, latest or a label. A label ref takes the same path as a version: unresolvable means the instructions fallback or LOUSHO_PROMPT_UNRESOLVED, with the same 30-second retry window.
  • Context hub. hub() / hubTools() let agents publish and query namespaced key-values scoped private (the writing agent only) or fleet (every agent on the daemon).
Studio’s Prompts screen shows the registry’s versions, labels and diffs and moves labels, and its Context Hub screen browses the hub. Runs from the Studio playground (POST /playground/run, see Agent Forge) report as agent playground; a run started from a registry prompt carries the prompt’s ref (name@label, name@3) in its root span’s lousho.prompt.ref attribute. Reporting is fire-and-forget and never blocks a run: when the daemon is down, the failed batch is requeued and retried with exponential backoff (1 s, 2 s, 4 s … capped at 30 s), so a daemon restart mid-run loses nothing. A backlog drains in batches one right after another, each at most 1000 events and 4 MB of JSON (under the daemon’s per-request caps of 1000 events and 5 MB). Only these are dropped and counted in the reporter’s dropped: events evicted past the buffer cap (maxBuffered, default 5000 events, or 32 MB of serialized events, whichever comes first; oldest first); events past the newest 16 MB when a crash flushes; events still unsent after the exit flush’s final attempt; a single event too large for any request (over 5 MB serialized, for example with content capture on); and a batch the daemon refuses outright (400/413), which is never retried. An open run’s root start (queued, its end not yet) is kept through both cuts: the buffer trim and the crash flush’s 16 MB slice evict other events first, so a run whose children overflow the buffer still appears. Only when nothing else is left to evict do open runs’ starts go too, oldest first, so memory stays bounded. The count rides on every heartbeat as meta.dropped (cumulative for the process), and each row of the daemon’s /api/v1/agents fleet list exposes the latest one as dropped: a number, or null for an agent on an older SDK that sent none. A process that dies of an uncaught exception or an unhandled rejection still reports its last run: the reporter watches uncaughtExceptionMonitor and sends what it holds (the newest 16 MB of it, open runs’ root starts first), oldest first, holding up the crash for up to 1 s per reporter. Node’s error output and exit code stay as they were, and a script that survives the throw (its own uncaughtException handler, or a capture callback such as the REPL’s) keeps reporting. An explicit process.exit() or a SIGKILL still loses what is buffered. Every run is visible from the moment it starts: the reporter sends a run’s root span (the top invoke_agent, the one with no parent) as soon as it opens, without waiting for the 5 s interval, and Studio lists that trace as running. The run’s end replaces that row with the final status, usage, cost and output, and counts once. So a process killed mid-run (SIGKILL, power loss) still leaves its trace behind, and the trace flips to interrupted when the agent stops heartbeating and the sweep marks it down. Each reporter process also sends a random instance_id, because a restart or a second process of the same agent reports under the same agent id. Processes of one agent running side by side never interrupt each other’s runs while they keep heartbeating, even when they share a hostname and pid (--network host replicas are all pid 1). When a process of an agent goes quiet while another process of it keeps heartbeating (a supervisor restarted it under a new pid, or one of several workers died), the sweep marks the quiet process’s open runs interrupted once it has missed about two heartbeats (the stale mark), judged on that process’s own heartbeat interval and on the daemon’s clock. A restart on the same host with the same pid (a container’s pid 1) marks them sooner: at the first heartbeat of the new process (after its first) that comes more than one heartbeat interval plus 10 seconds after the old process’s last, if the old process has been silent since the new one appeared. With no process of the agent left, the agent goes down and its open runs are marked as above. A process that exits gracefully mid-run (SIGINT, SIGTERM) marks its own open runs interrupted right away. An agent with heartbeats turned off (heartbeatIntervalMs: 0) is never swept, so its killed runs stay running. Child spans (model calls and tool calls) are still sent only when they finish, so a killed run keeps the root and any child that finished and was already sent. Eval scores flow through the same pipe — lousho eval --observe emits them even when LOUSHO_OBSERVE is unset. Each case yields one <eval>/<check> row per assertion plus one <eval> pass/fail row (1 when every gate check passed, else 0). Human feedback is a score too. POST /api/v1/scores with { "trace_id", "obs_id"?, "name", "value", "comment"? } (value a number or null, name up to 200 characters, comment up to 10 000) adds one score with source: "human" and answers 201 { "score_id" }. Each call adds a row, and the score is attributed to the trace’s agent. An unknown trace_id (or an obs_id that is not in that trace) is a 404, an empty name a 400. It takes the same token as ingest, and GET /api/v1/scores?trace_id=... lists it.

Datasets

A dataset is a named set of examples to run evals over. Each example is an input and an optional expected value, both any JSON value. Studio’s Datasets screen lists and creates them, and a trace’s Add to dataset copies the trace in. The routes take the same token as ingest: A dataset row is { dataset_id, name, description, created_at, example_count } and an example row is { example_id, dataset_id, input, expected, source_trace_id, created_at }, with expected and source_trace_id null when unset. from_trace_id copies the trace’s root observation: its input becomes input and its output expected. Stored text that is valid JSON is copied as that value, and any other text as a JSON string. A trace whose root has no captured input (content capture was off) is a 400. An unknown dataset, trace or example is a 404 with code LOUSHO_REQUEST_INVALID. The example body may be up to 1 MiB. Retention never deletes datasets or examples. An example keeps its source_trace_id after that trace is pruned.

Online evaluator rules

An evaluator rule says which new traces to score and how. A code rule holds the source of a function (trace) => number | { value, comment }, and a judge rule holds a model and a rubric to grade by. The daemon runs code rules on new traces (see Online evaluators); judge rules run through Agent Forge (see Judge rules). Studio’s Evaluators screen manages them. The routes take the same token as ingest: A rule row is { rule_id, name, enabled, filter, sample_rate, kind, config, created_at, updated_at }. The fields are:
  • name: 1-200 characters after trimming, unique.
  • enabled: a boolean, true by default.
  • filter: { "agent_id"?, "status"?, "name_contains"? }, {} by default. status is ok or error, and each value is a non-empty string.
  • sample_rate: a number from 0 to 1, 1 by default.
  • kind and config: code takes { "source" } (up to 64 KiB of text), and judge takes { "model", "rubric" } (up to 200 and 20 000 characters).
An unknown field, an unknown kind or a config that does not match the kind is a 400 with code LOUSHO_REQUEST_INVALID. An unknown rule is a 404 with the same code. updated_at moves on every PUT. The body may be up to 128 KiB. Retention never deletes rules.

Online evaluators

When a trace’s root observation ends, the daemon runs every enabled code rule whose filter matches the root (agent_id, status, and name_contains on its name). A rule runs when Math.random() < sample_rate, so 0 never runs and 1 always does. Each run writes one score with source: "online", name = the rule’s name, the root’s agent_id, and no obs_id. A trace that already has an online score with that name is not scored again. The rule’s source runs as (<source>)(trace) in the isolate sandbox (sandbox({ driver: 'isolate', timeoutMs: 2000 })). trace is a plain JSON object:
observations holds the trace’s rows, oldest first, capped at 200. A row whose attrs JSON is longer than 8192 characters carries "attrs": { "truncated": true } instead. error_message is the first error row’s attrs.error, even when that row is past the first 200. Rules match on the root row alone, so a trace no rule’s filter keeps loads no other rows, and the worker handles one trace per event-loop turn. The function returns a finite number (the score’s value) or { "value": <number | null>, "comment"?: "<text>" } (the comment is cut at 1000 characters). Anything else scores value: null with comment invalid_result. A run that fails scores value: null with the sandbox error kind as the comment: timeout, oom, or driver. An error your code throws carries no kind, so it reports as driver. The isolate driver needs the optional isolated-vm package (npm install isolated-vm). On Node 20 and later the daemon also needs NODE_OPTIONS=--no-node-snapshot. Without them, code rules are skipped and the daemon warns once. Rules run off the ingest request, one trace at a time, so ingest stays fast even when an evaluator is slow or fails. Up to 1000 traces wait in memory; past that, new traces are dropped (and counted). Waiting traces are lost when the daemon restarts, so those traces never get online scores. POST /api/v1/evaluators/:id/test { "trace_id": "<trace-id>" } dry-runs a rule on one trace and answers 200 { "result": { "value", "comment" } }. It ignores enabled, filter and sample_rate, and it writes no score. An unknown rule is a 404, and so is a trace that does not exist or has no root observation. A bad body is a 400. Dry runs go one at a time: a second request waits until the one in flight finishes. A judge rule is a 409 (judge evaluators run through Agent Forge). Without the isolate it answers { "result": { "value": null, "comment": "isolate_unavailable" } }.

Judge rules (through Agent Forge)

The daemon holds no provider keys, so it cannot run a judge rule itself. When a judge rule matches a new trace (same filter, sample_rate and “already scored” check as code rules), the daemon queues a job for that rule and trace. Agent Forge (lousho studio) pulls the jobs, runs each judge with the provider keys it already has (a key saved in Studio settings wins over the provider’s env var, as in the playground), and posts the result back. The daemon writes it as the same source: "online" score. So judges run only while Studio is running. Until then the jobs wait.
  • At most 500 jobs wait at once (pending plus claimed). Past that, new jobs are dropped and counted. Jobs live in memory, so a daemon restart loses them.
  • The judge is one model call to the rule’s model (provider/model, or mock). The system prompt is a fixed grading instruction plus the rule’s rubric. The user message holds the trace’s name, status, input, output and error, each cut at about 8 KB. The model must reply with a JSON object { "score": <0..1>, "comment": "<short reason>" }. Agent Forge reads the first JSON object in the reply. A reply with no valid score scores value: null with comment invalid_judge_output. A provider error (a missing key, for example) scores value: null with comment judge_error: <message>.
  • The judge call makes no trace of its own, so it never triggers evaluators.
The two endpoints Agent Forge uses (they take the same token as ingest, which Agent Forge sends from LOUSHO_OBSERVE_TOKEN when it is set): Agent Forge claims up to 5 jobs every 2 seconds, with a 60-second lease, and runs them one at a time. Each judge call is cut off after 20 seconds, or sooner if less of the lease is left (a call cut off scores value: null with comment judge_error: timed out after <n> ms). A claimed job whose lease has run out before its turn is not run: it expires and is claimed again later. A daemon that is down only pauses the loop, and it logs once per outage. The dry run below has the same 20-second cap. To dry-run a judge rule, use Agent Forge: POST /observe/api/v1/evaluators/:id/test { "trace_id" } on the Studio server runs the judge on that trace and answers 200 { "result": { "value", "comment" } } without writing a score. For a code rule it forwards the request to the daemon unchanged. The daemon’s own /test still answers 409 for a judge rule.

Annotation queues

An annotation queue is a named list of traces for a person to review one by one. Completing an item writes its scores as human scores, the same source: "human" rows POST /api/v1/scores writes, so they show up in GET /api/v1/scores and on the trace. Studio’s Annotation queues screen creates queues and is the review view. The routes take the same token as ingest: A queue row is { queue_id, name, rubric, created_at, counts: { pending, done, skipped } }. An item row is { item_id, queue_id, trace_id, status, reviewer, created_at, done_at }, where reviewer and done_at are null until the item is finished. :id is the queue_id.
  • name: 1-200 characters after trimming, unique.
  • rubric: up to 50 entries of { "name", "description"?, "min"?, "max"? }, [] by default. Names are 1-200 characters and unique, a description is up to 2000 characters, and min may not exceed max.
The complete body takes scores, up to 50 entries of { "name", "value", "comment"? } where value is a number or null, an optional reviewer (up to 200 characters) and status, which is done (the default) or skipped. Each score is written on the item’s trace, and the status, reviewer and done_at are set in the same transaction. When the queue has a rubric, every score name must be one of its names and the value must sit within that entry’s min and max, or the answer is a 400. A skip with scores is a 400. Completing an item that is already done or skipped is a 409 and writes nothing. An item that is not in the queue is a 404. next does not claim or lock the item it returns. Asking twice returns the same item until someone completes it. The queue is meant for one reviewer working locally, so two reviewers on one queue can score the same trace. Retention never touches queues or items. If an item’s trace is pruned, the item keeps its trace_id and next still returns it. Completing it with scores is then a 404 naming the trace, and the item stays pending, so skip it instead. Bad bodies and unknown fields are a 400 with code LOUSHO_REQUEST_INVALID, and an unknown queue is a 404 with the same code. Bodies may be up to 256 KiB.

Retention

The daemon keeps 30 days of runs by default. On its first sweep after start, and then once a day, it deletes every trace whose observations have all finished and whose last one ended more than --retention-days ago (default 30; --retention-days 0 keeps everything). A whole trace goes or stays: one still-open observation (end unset, a run in progress) keeps the entire trace, however old. With a trace go its scores and its prompt-use links; scores older than the window whose trace has no observations left go too. A session is deleted only when it is not active, ended (or, with no end, started) before the window, and no remaining observation references it, so an old thread that is still active is never pruned. Idle sessions end on their own (see Idle sessions), so finished conversations do age out. Pruning never touches the Insights rollups (charts for pruned days stay), the prompt registry, hub entries, the agent registry or tokens. The daily prune does not VACUUM, which would stall ingest; to reclaim disk space, run lousho observe prune --older-than 30d. It prunes now with the window you give (whole days only), prints how many observations, scores and sessions it deleted, then runs VACUUM. It is safe while the daemon is running on the same db.

Export

lousho observe export [--since 7d] [--agent X] [--out file] dumps traces as JSONL for offline analysis or another tool. Each line is one trace: { "trace": <trace row>, "observations": [...], "scores": [...] }, the same shapes as the trace list and detail APIs, oldest trace first and not capped. --since takes <N>d, <N>h or <N>m back from now; --agent matches an agent_id, or an agent’s name when it names exactly one. Lines go to stdout (the count goes to stderr) unless --out names a file, which is overwritten. It reads the db directly and works without a running daemon. Captured content is exported as stored, so treat the file like the db.

Continuous score

Each finished run of an agent becomes one point in a series, oldest first:
  • the mean of the run’s finished online evaluator scores, when it has any;
  • otherwise 0 when the run is in the failure report (a failed call, no answer, an interrupted process or a low score), and 1 when it is not.
A point of 0.5 or more counts as a pass. Scores still without a value, and runs still in flight, are left out. Each point carries the run’s lousho.harness.revision, so a series can be cut to one revision. The trajectory gate from the Self-Healing Harness paper judges the shape of the series rather than one number. It works on the rolling mean of 10 points:
  • The first full window (10 points) sets a running peak.
  • A later window mean at least 0.02 above the peak is a gain, which raises the peak.
The gate judges the newest window: The significance check is the paper’s corroboration step. A drop of 0.15 is only one or two failures in a window of pass/fail runs, so the shape alone would fire on most healthy agents. The test depends on the values:
  • for pass/fail values it compares pass rates (Newcombe’s interval from Wilson bounds);
  • otherwise it compares means (Welch’s t).
One judgement of a stable series reads regression about 5% of the time. Before the first full window the gate reports ok with ready: false. The daemon does not judge on every run, because repeating the test would add up to false alarms. It waits until an agent has 20 finished runs. After that, each time 10 new runs have arrived, it runs the gate over the last 200 points at a stricter z of 1.96. The long lookback keeps the old baseline in view when a lasting drop fills recent runs. It raises an agent.score_regression alert only after three judgements in a row read regression. These are simulated figures for pass/fail agents (1,000 seeded streams each). A stable agent with a pass rate between 0.6 and 0.95 raised a false alert in under 1% of 1,000-run stretches. When 200 runs at the old rate were followed by 100 at the new one, the fall was caught within those 100 runs this often: The alert fires at most once per agent per revision. The revision is that of the newest run, and null when that run has no stamp. The alert’s details carry:
  • revision, peak, mean and drop;
  • since_gain and the number of runs;
  • recent: a summary of the newest 10 runs.
The daemon keeps this memory only while it delivers alerts, and starts it afresh on restart. A failure in this pass is logged and never costs the sweep its other alerts. lousho observe score [--agent X] [--revision R] [--limit N] [--json] prints the same view for each agent from the db:
  • runs, mean, and the pass rate with its one-sided 95% Wilson lower bound;
  • pass^3, the chance that three independent runs all pass;
  • the gate state.
Without --revision, the line covers every revision in the window and names the latest one. It reads the db directly and works without a running daemon.

Trace list API

GET /api/v1/traces returns { traces, total, limit, offset }, one row per trace, newest first. Query parameters (all optional): An unknown sort or order, a non-numeric min_duration_ms, or a prompt with an empty name or a non-integer version answers 400 with code LOUSHO_REQUEST_INVALID. Each row has trace_id, name, agent_id, session_id, start, end, status, the counts observations, generations, tools and errors, the sums input_tokens, output_tokens and cost_usd, and:
  • agent_name: the agent’s name from its heartbeat, or null.
  • model: the first model call’s model, or null.
  • error_message: the first failed observation’s error attribute (for example upstream 529 overloaded), or null.
  • input_preview / output_preview: the first 200 characters of the run’s stored input and output, or null when none was captured.

Sessions (threads) API

GET /api/v1/sessions returns { sessions, total, limit, offset }, newest first (started_at descending). Query parameters (all optional): agent_id, status (exact match), from / to (session start in ms epoch, half-open from <= started_at < to), and limit / offset (1-500, default 50). Each session row is { session_id, agent_id, agent_name, status, started_at, ended_at, traces, input_tokens, output_tokens, cost_usd, last_input_preview }: traces counts distinct trace ids, tokens and cost sum the session’s observations, and last_input_preview is the first 200 characters of the root input of its latest trace. status is active, ended (idle, see below) or interrupted (its agent went down). GET /api/v1/sessions/:id returns { session, traces } with the traces (same rows as the trace list) ordered by start ascending, capped at 500; an unknown id is a 404 with code LOUSHO_REQUEST_INVALID. Studio’s Threads screen reads these two routes.

Idle sessions

The SDK sends no “session closed” event: a session row exists because its runs carry its id, so the daemon decides when a conversation is over. Each sweep (every 30 s) marks a session ended when none of its runs is still open (a run in progress keeps it active however long it takes) and nothing in it has started or finished for --session-idle-minutes (default 30; 0 turns this off). A run marked interrupted (its process died while the agent kept running) no longer counts as in progress; it counts as activity at the moment it was interrupted. Its ended_at is the time of that last activity, not the time of the sweep. A session no run references falls back to when the daemon last updated it. Sessions in a db written by an older daemon close on the first sweep that finds them idle. A new run in an ended session makes it active again (ended_at follows its runs again), and so does new activity in an interrupted one. A re-sent or reordered event for activity at or before the session’s ended_at does not reopen it. The idle pass never touches interrupted sessions. Once ended, a session is pruned by retention like any other finished session, so Threads shows only live conversations as active. The idle window is judged against the timestamps your agents report, so an agent clock far behind the daemon’s can end a session early; its next run reopens it.

The built server

The server lousho build --target=node-server or --target=docker generates exports traces with no code when the standard OTEL_EXPORTER_OTLP_ENDPOINT (or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT) variable is set where it runs. It registers an OpenTelemetry tracer provider that sends spans over OTLP/HTTP, and gives the agent createOtelTraceExporter(), so every /chat turn arrives as the invoke_agent / chat / execute_tool tree above. An exporter the agent’s config already has (an agent directory’s or a module’s) keeps running: both receive every span. The OpenTelemetry SDK is not bundled. Install these packages next to dist/server.js, or add them to the build’s package.json dependencies before you build the docker image:
  • The server imports them at start-up only when the variable is set. When one is missing it logs which packages to install and serves without exporting traces.
  • service.name is OTEL_SERVICE_NAME when set, otherwise the agent’s name. OTEL_RESOURCE_ATTRIBUTES, OTEL_EXPORTER_OTLP_HEADERS (for an API key) and the batch settings (OTEL_BSP_SCHEDULE_DELAY, …) work as the OTel SDK documents them.
  • On SIGINT or SIGTERM the server sends the spans it still holds before it exits.
  • With neither variable set, nothing is imported and the server runs as before.
Message content follows the agent’s own settings (Message and argument content is opt-in).

Backends

Any backend that ingests OpenTelemetry traces (OTLP) receives these spans. Backends and tools that understand the GenAI semantic conventions can render them as LLM, agent and tool calls with token usage; others show them as ordinary spans with attributes. Which conventions a given backend supports, and which version of them, varies; check its documentation.