Verify what your agent actually changed.
Your application still calls the provider. Conseqa records the intended change before dispatch, then independently reads the system of record. The public packages and installation bundle require no access to the private source repository.
Quickstart: three protected calls in two minutes
With Node.js 22.13 or newer, and nothing else installed:
npx --yes @conseqa/cli@0.1.12 quickstart
cd conseqa-quickstart
npm install
npm startThis writes a small project and runs it: a stand-in payments service, a contract, a runner, and three refunds protected by the SDK. One is paid. One is paid although its reply is lost, and is found by its reference instead of being sent again. One is accepted and then fails, and is reported as Failed with the check that failed. All of it runs on your machine and nothing leaves it. No account and no Docker are needed.
To see the same three refunds in your dashboard, create a workspace, make an environment key in Settings, paste it after CONSEQA_ENVIRONMENT_KEY= in the project's .env, and run npm start again. It prints a link to each refund. Use a development environment: every run records three sample refunds and one issue.
The project is also the shortest way into your own integration. Its README names the few places to change so that the runner reads your system instead of the stand-in. The rest of this guide is the reference for each part.
Start with one consequential action
Use Node.js 22.13 or newer, a TypeScript/JavaScript application, and Docker Compose for the published Linux amd64/arm64 runner. Choose an external state you can read independently: a refund, resource, shipment, inventory record, or provisioned account.
To see the idea first, open the public PostgreSQL demo, or sign in and select Run live verification to save the result in a dedicated Live demo environment. It writes synthetic store credit to a real demo database; it does not call an LLM, move money, or contact a customer system.
1. Install without a private checkout
Download and review install.mjs. It fetches only the curated public bundle, creates a new folder, and generates separate local keys. It never starts a service or overwrites an existing installation.
node install.mjs --directory ./conseqa-local
cd conseqa-local
npm install --save-dev @conseqa/cli@0.1.12
node node_modules/@conseqa/cli/dist/index.js init
node node_modules/@conseqa/cli/dist/index.js contracts build
node node_modules/@conseqa/cli/dist/index.js contracts testThe bundled README, Compose file, and blank configuration template are public. Keep the generated .env private. On Windows, review the new folder’s access permissions before adding real credentials.
2. Define the outcome in code
The CLI creates conseqa.contracts.json, client and runner TypeScript files, JSON schemas, and an offline fixture. The starter contract is acme.resource.ready. Its verifier deliberately returns Unknown until you implement a source-of-record probe; passing the example regression does not mean a real resource was verified.
The client derives a correlation key before execution, binds a returned resource ID, and classifies exceptions. The runner discovers an exact resource when supported, reads its state, and produces checks. Keep callbacks and read-only credentials local. The hosted registry accepts only the manifest and schemas, not executable bundles.
Customize the schemas, schedule, evidence allowlist and callbacks. Build after changes. A contract name/version cannot be republished with different bytes; bump its version. Each action pins the SHA-256 build hash, and required protection rejects an application/runner mismatch.
Optionally, declare the verdict as data with expectations: predicates that must all pass for Satisfied, and predicates that mean Violated. Expected values can refer to the registered intent or the bound provider identity. When a contract declares them, the runner decides every observed verdict with them, they are published in the manifest, and saved evidence can be re-checked when you change them. Keep exact identity checks that need secrets or pseudonymized fields in your verifier. Requires CLI and runner 0.1.5 or later; earlier runners reject a package that declares expectations. The checks shown for an action are the rules under satisfied. A rule under violated decides the verdict and is not listed as a check.
"expectations": {
"satisfied": [
{ "id": "processed", "pointer": "/status", "operator": "equals", "expected": "processed" },
{ "id": "amount", "pointer": "/amountMinor", "operator": "equals",
"expected": { "$intent": "/amountMinor" } }
],
"violated": [
{ "id": "failed", "pointer": "/status", "operator": "equals", "expected": "failed" }
]
}Beyond refunds: any action you can read back
A contract fits any action that changes another system you can read afterwards: payouts, orders, bookings, CRM records, tickets, inventory, subscriptions, an email with a delivery status, or a database row (the PostgreSQL verifier, as in the database demo). It needs three things: a record named before the call or a key that travels with it, a read-only way to look the result up, and a rule for what done means. It does not fit output with nothing outside to check, such as an answer or a summary.
The two recipes below use the general HTTP verifier, so they work with any JSON API. Each is a whole contract, type-checked and tested against a stand-in API. Change the paths and fields to your API's, then use it the way section 2 describes: export the contract from your client entry and its runner from your runner entry, give the intent, binding and observation a schema, and allow the observation's two fields as evidence.
A CRM record the agent updates
The contact is named in the intent, so after a lost reply the runner reads it directly. Verified when the contact is at the stage the agent set; Failed when it was archived; Pending, then Overdue, while it is anywhere else.
crm-contact-stage.ts
// Recipe: the agent moves a contact to a new stage in your CRM.
//
// The contact is named in the intent, so after a lost reply the runner simply reads it:
// no search is needed. The verdict is about the record, not the call: Verified when the
// contact is at the stage the agent set, Failed when it was archived, and Pending (then
// Overdue at the deadline) while it is anywhere else.
import type { CheckResult, JsonValue, OutcomeContract, ProbeResult } from "@conseqa/contracts";
import { verifyHttpJson, type JsonPredicate, type VerifierResult } from "@conseqa/verifiers";
type Intent = { contactId: string; stage: string };
type Binding = { contactId: string };
type Observation = { contactId: string; stage: string | null };
export interface CrmOptions {
/** Your CRM's API, for example https://crm.example.com/api/v2. */
readonly baseUrl: () => string;
/** A read-only token, read by the runner from its own environment. */
readonly token: () => string;
/** Only for a test server on this machine. */
readonly allowPrivateNetwork?: boolean;
}
function text(value: unknown, name: string): string {
if (typeof value !== "string" || value.length === 0 || value.length > 200)
throw new TypeError(`${name} must be a non-empty string`);
return value;
}
function object(value: unknown): Record<string, unknown> {
if (!value || typeof value !== "object" || Array.isArray(value))
throw new TypeError("Expected a JSON object");
return value as Record<string, unknown>;
}
function toProbe(
result: VerifierResult,
predicates: readonly JsonPredicate[],
observation: Observation,
): ProbeResult<Observation> {
if (result.verdict === "inconclusive")
return {
kind: "inconclusive",
reason: result.message ?? "The CRM could not be read",
retryable: result.retryable ?? true,
...(result.retryAfterMs === undefined ? {} : { retryAfterMs: result.retryAfterMs }),
};
// The checks that decide Verified; the archived check only decides Failed.
const checks: CheckResult[] = (result.evidence?.checks ?? [])
.slice(0, predicates.length)
.map((check) => ({
id: check.path,
passed: check.passed,
...(check.expected === undefined ? {} : { expected: check.expected as JsonValue }),
...(check.observed === undefined ? {} : { observed: check.observed as JsonValue }),
}));
return {
kind: "observed",
observation,
verdict: result.verdict,
checks,
// Only these fields leave the runner, never the whole contact.
evidence: {
source: result.evidence?.source ?? "crm",
observedAt: result.evidence?.observedAt ?? new Date().toISOString(),
fields: observation,
},
};
}
export function crmContactStageContract(
options: CrmOptions,
): OutcomeContract<Intent, Binding, Observation> {
return {
name: "crm.contact.stage_set",
version: 1,
operation: "crm.contact.update",
// The record is named before the call, so it is always found exactly.
capabilities: { lostResponseDiscovery: true },
intentSchema: {
parse: (value) => {
const intent = object(value);
return {
contactId: text(intent.contactId, "contactId"),
stage: text(intent.stage, "stage"),
};
},
},
bindingSchema: {
parse: (value) => ({ contactId: text(object(value).contactId, "contactId") }),
},
observationSchema: {
parse: (value) => {
const observed = object(value);
return {
contactId: text(observed.contactId, "contactId"),
stage: typeof observed.stage === "string" ? observed.stage : null,
};
},
},
client: {
deriveProviderCorrelationKey: ({ intentId }) => `cq_${intentId.replaceAll("-", "")}`,
bindResult: (result) => {
const id = (result as { id?: unknown } | null)?.id;
return typeof id === "string" ? { contactId: id } : null;
},
classifyException: () => ({ certainty: "ambiguous", retryable: false }),
},
runner: {
discover: async ({ intent }) => ({ kind: "found", binding: { contactId: intent.contactId } }),
verify: async ({ intent, binding }) => {
const api = new URL(options.baseUrl());
const root = api.pathname.replace(/\/+$/u, "");
const predicates: JsonPredicate[] = [
{ path: "/stage", operator: "equals", value: intent.stage },
];
const result = await verifyHttpJson({
baseUrl: api.origin,
path: `${root}/contacts/${encodeURIComponent(binding.contactId)}`,
allowedHosts: [api.hostname],
allowPrivateNetwork: options.allowPrivateNetwork ?? false,
headers: { authorization: `Bearer ${options.token()}` },
predicates,
terminalFailurePredicates: [{ path: "/archived", operator: "equals", value: true }],
source: api.origin,
});
const stage = (result.observation as { stage?: unknown } | undefined)?.stage;
return toProbe(result, predicates, {
contactId: binding.contactId,
stage: typeof stage === "string" ? stage : null,
});
},
},
};
}
// The runner reads both settings from its own environment, only when it checks.
export const crmContactStage = crmContactStageContract({
baseUrl: () => process.env.CRM_API_URL ?? "",
token: () => process.env.CRM_READ_TOKEN ?? "",
});
An email that must be delivered, not just accepted
A provider's yes means it accepted the message; it can still bounce minutes later. The verdict waits for the delivery status, and a bounce after acceptance is a failure. Send the correlation key as the message's metadata, so that after a lost reply the runner finds the message and nobody sends it twice.
email-delivered.ts
// Recipe: the agent sends an email through your provider's API.
//
// "Accepted" is not "delivered": a message can bounce minutes after the provider said
// yes. The verdict waits for the delivery status: delivered is Verified; bounced, rejected
// or failed is Failed; anything else stays Pending until the deadline, when it is Overdue.
//
// Send the correlation key with the message (most providers accept custom metadata). After
// a lost reply the runner finds the message by it, so nobody sends the email twice.
import type { CheckResult, JsonValue, OutcomeContract, ProbeResult } from "@conseqa/contracts";
import { verifyHttpJson, type JsonPredicate, type VerifierResult } from "@conseqa/verifiers";
// A reference to the recipient, not their address: the intent is stored as evidence.
type Intent = { template: string; recipientRef: string };
type Binding = { messageId: string };
type Observation = { messageId: string; status: string | null };
export interface EmailOptions {
/** Your provider's API, for example https://mail.example.com/v1. */
readonly baseUrl: () => string;
/** A read-only token, read by the runner from its own environment. */
readonly token: () => string;
/** Only for a test server on this machine. */
readonly allowPrivateNetwork?: boolean;
}
function text(value: unknown, name: string): string {
if (typeof value !== "string" || value.length === 0 || value.length > 200)
throw new TypeError(`${name} must be a non-empty string`);
return value;
}
function object(value: unknown): Record<string, unknown> {
if (!value || typeof value !== "object" || Array.isArray(value))
throw new TypeError("Expected a JSON object");
return value as Record<string, unknown>;
}
function toProbe(
result: VerifierResult,
predicates: readonly JsonPredicate[],
observation: Observation,
): ProbeResult<Observation> {
if (result.verdict === "inconclusive")
return {
kind: "inconclusive",
reason: result.message ?? "The provider could not be read",
retryable: result.retryable ?? true,
...(result.retryAfterMs === undefined ? {} : { retryAfterMs: result.retryAfterMs }),
};
const checks: CheckResult[] = (result.evidence?.checks ?? [])
.slice(0, predicates.length)
.map((check) => ({
id: check.path,
passed: check.passed,
...(check.expected === undefined ? {} : { expected: check.expected as JsonValue }),
...(check.observed === undefined ? {} : { observed: check.observed as JsonValue }),
}));
return {
kind: "observed",
observation,
verdict: result.verdict,
checks,
// Only the id and the status leave the runner, never the message.
evidence: {
source: result.evidence?.source ?? "email",
observedAt: result.evidence?.observedAt ?? new Date().toISOString(),
fields: observation,
},
};
}
export function emailDeliveredContract(
options: EmailOptions,
): OutcomeContract<Intent, Binding, Observation> {
const read = (path: string, predicates: JsonPredicate[], failure: JsonPredicate[] = []) => {
const api = new URL(options.baseUrl());
const root = api.pathname.replace(/\/+$/u, "");
return verifyHttpJson({
baseUrl: api.origin,
path: `${root}${path}`,
allowedHosts: [api.hostname],
allowPrivateNetwork: options.allowPrivateNetwork ?? false,
headers: { authorization: `Bearer ${options.token()}` },
predicates,
terminalFailurePredicates: failure,
source: api.origin,
});
};
return {
name: "email.message.delivered",
version: 1,
operation: "email.message.send",
capabilities: { lostResponseDiscovery: true },
intentSchema: {
parse: (value) => {
const intent = object(value);
return {
template: text(intent.template, "template"),
recipientRef: text(intent.recipientRef, "recipientRef"),
};
},
},
bindingSchema: {
parse: (value) => ({ messageId: text(object(value).messageId, "messageId") }),
},
observationSchema: {
parse: (value) => {
const observed = object(value);
return {
messageId: text(observed.messageId, "messageId"),
status: typeof observed.status === "string" ? observed.status : null,
};
},
},
client: {
// Pass this as the message's metadata when you send it.
deriveProviderCorrelationKey: ({ intentId }) => `cq_${intentId.replaceAll("-", "")}`,
bindResult: (result) => {
const id = (result as { id?: unknown } | null)?.id;
return typeof id === "string" ? { messageId: id } : null;
},
classifyException: () => ({ certainty: "ambiguous", retryable: false }),
},
runner: {
discover: async ({ providerCorrelationKey }) => {
const found = await read(
`/messages?metadata.conseqa=${encodeURIComponent(providerCorrelationKey)}`,
[{ path: "/messages", operator: "exists" }],
);
if (found.verdict === "inconclusive")
return {
kind: "error",
reason: found.message ?? "The provider could not be searched",
retryable: found.retryable ?? true,
...(found.retryAfterMs === undefined ? {} : { retryAfterMs: found.retryAfterMs }),
};
const messages = (found.observation as { messages?: unknown } | undefined)?.messages;
const list = Array.isArray(messages) ? (messages as { id?: unknown }[]) : [];
if (list.length === 0) return { kind: "not_found", retryAfterMs: 5_000 };
if (list.length > 1)
return {
kind: "ambiguous",
candidateCount: list.length,
reason: "More than one message carries this correlation key",
};
const id = list[0]?.id;
return typeof id === "string"
? { kind: "found", binding: { messageId: id } }
: {
kind: "error",
reason: "The provider listed a message without an id",
retryable: false,
};
},
verify: async ({ binding }) => {
const predicates: JsonPredicate[] = [
{ path: "/status", operator: "equals", value: "delivered" },
];
const result = await read(
`/messages/${encodeURIComponent(binding.messageId)}`,
predicates,
[{ path: "/status", operator: "in", value: ["bounced", "rejected", "failed"] }],
);
const status = (result.observation as { status?: unknown } | undefined)?.status;
return toProbe(result, predicates, {
messageId: binding.messageId,
status: typeof status === "string" ? status : null,
});
},
},
};
}
// The runner reads both settings from its own environment, only when it checks.
export const emailDelivered = emailDeliveredContract({
baseUrl: () => process.env.EMAIL_API_URL ?? "",
token: () => process.env.EMAIL_READ_TOKEN ?? "",
});
Verify the published runner before running it
The public Compose bundle pins the immutable 0.1.12 image, not a mutable latest tag:
ghcr.io/devrajsinh-jhala/conseqa-runner@sha256:75b37d4253c49cbf605df7d3cd3c122a25c7a7ea18ac620f69aaabcb268ecd4bDownload container-release.mjs and runner-signing-key.pub. Independently pin the public key’s SPKI DER SHA-256 fingerprint:
68e5412daf00ef00c43c0e93b1e342add20e2e4239c6833bdc3d8a72fb79441dA fingerprint copied from the same untrusted download is not independent publisher authentication. The signed release descriptor binds the version, source commit, exact image digest and platforms. Use ORAS 1.3.4 or newer to retrieve the descriptor, and verify it with the pinned key before starting the image.
oras pull ghcr.io/devrajsinh-jhala/conseqa-runner:release-0.1.12 --output ./release-proof --keep-old-files
node container-release.mjs verify-public --envelope ./release-proof/signed-release.json --public-key ./runner-signing-key.pub --image ghcr.io/devrajsinh-jhala/conseqa-runner --version 0.1.12 --digest sha256:75b37d4253c49cbf605df7d3cd3c122a25c7a7ea18ac620f69aaabcb268ecd4bStop if verification fails or either required platform is missing. This is a publisher-key signature, not a claim of a GitHub/SLSA attestation.
3. Start the private persistent runner
docker compose up -d conseqa
node --env-file=.env node_modules/@conseqa/cli/dist/index.js contracts install --runner http://127.0.0.1:4319
node --env-file=.env node_modules/@conseqa/cli/dist/index.js doctorThe host port is loopback-only. The admin token installs contracts; the different sidecar token authorizes application lifecycle traffic. Never put either token in browser code or a public URL. Persist the SQLite, contract and spool volumes, and run one runner per environment in v1.
Without Docker, the same runner is the @conseqa/runner package, which is how the quickstart runs it. From version 0.1.8 it reads the installation's .env as it stands, listens on 127.0.0.1:4319 and keeps its records in .conseqa in the folder it is started from:
npm install @conseqa/runner@0.1.12
node --env-file=.env node_modules/@conseqa/runner/dist/main.jsThat suits a laptop or a single server you manage yourself. Use the signed container image where you want a pinned, verified build.
Inject the credential names your trusted verifier needs into the runner’s private environment. PostgreSQL verifiers use parameterized SELECT queries and should have a least-privilege read-only role. HTTP verifiers allow GET only with explicit host/network restrictions. The runner never repeats the provider action.
4. Protect the application’s existing call
npm install @conseqa/sdk@0.1.12 @conseqa/contracts@0.1.12In your application, load the exact built contract package. This example assumes you have implemented createResource and the matching read-only verifier; the unmodified starter is intentionally incomplete.
import { Conseqa, loadClientContractPackage } from "@conseqa/sdk";
const contractRegistry = await loadClientContractPackage({ directory: "dist/conseqa" });
const conseqa = new Conseqa({
contractRegistry,
runnerUrl: "http://127.0.0.1:4319",
sidecarToken: process.env.CONSEQA_SIDECAR_TOKEN,
});
const contract = contractRegistry.get("acme.resource.ready", 1);
try {
const result = await conseqa.protect(contract, {
actionKey: "stable-business-operation-id",
protectionMode: "required",
intent: { name: "example" },
execute: ({ providerCorrelationKey }) =>
createResource({ name: "example", correlationKey: providerCorrelationKey }),
});
// Use the original provider result. Verification happens independently.
} finally {
await conseqa.close();
}Required actions do not execute without durable intent registration and the validated contract hash. The original provider result or exception remains the caller’s result. Without a configured shared SDK spool, runner outages fail closed. Best-effort protection may proceed when persistence fails, but must appear as an unprotected coverage gap.
A durable SDK spool requires the application and runner to share the same encrypted volume, key/key ID and service UID/GID. It is allowed only after that runner environment has validated the hash. Do not substitute a separate host directory for the shared volume.
From Python
A Python program protects its calls with the conseqa package from PyPI. It needs Python 3.9 or newer and has no dependencies. The contract package is still built with conseqa contracts build and installed in the runner as above. Its application half is written in Python, as arguments to package.contract. This example protects the refund that npx @conseqa/cli quickstart sets up.
Python conseqa 0.1.1 works with the 0.1.12 runner and CLI. It has one difference from the TypeScript SDK: no durable spool, so it cannot record an action while the runner is down.
pip install conseqa==0.1.1import os
from conseqa import Conseqa, ambiguous, load_contract_package, rejected
def what_an_error_proves(error):
# A refusal with a 4xx status proves the refund was not made.
# Anything else, a dropped connection above all, proves nothing either way.
status = getattr(error, "status", None)
if isinstance(status, int) and 400 <= status < 500:
return rejected(f"http_{status}")
return ambiguous("no_reply")
package = load_contract_package("dist/conseqa")
refund_processed = package.contract(
"quickstart.refund.processed",
1,
# What identifies the thing that was made, taken from the service's reply.
bind_result=lambda reply: {"refundId": reply["id"]} if reply.get("id") else None,
classify_exception=what_an_error_proves,
)
conseqa = Conseqa(
package,
runner_url="http://127.0.0.1:4319",
sidecar_token=os.environ.get("CONSEQA_SIDECAR_TOKEN"),
)
intent = {"paymentId": "pay_1001", "amountMinor": 2500, "currency": "USD"}
reply = conseqa.protect(
refund_processed,
action_key="ticket-8042-refund",
intent=intent,
# The call you already make, given the reference to send with it.
execute=lambda call: create_refund({**intent, "reference": call.provider_correlation_key}),
)protect returns what your call returned and raises what it raised; protect_async does the same for a program that runs on asyncio. The Python package has no spool: when the runner cannot be reached, a required action stops with ProtectionUnavailableError and a best-effort one runs unprotected. The reference lists every argument.
5. Keep the trace and consequence connected
Traces do two things. They let the dashboard show an action inside the agent run that made it, and they are how coverage is counted: of all the times an operation ran, how many were protected. Verdicts do not need them.
Conseqa does not install tracing in your application. It uses the OpenTelemetry your application already has: conseqa.protect puts its call in a span of the trace that is active and marks it as a protected run of the operation. To count the runs that were not protected, wrap them in conseqa.observe, or set the attribute conseqa.eligible to true on a span of your own.
If your application has no tracing yet, install two packages and write one file:
npm install @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-httpimport { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
const sdk = new NodeSDK({
serviceName: "my-agent",
traceExporter: new OTLPTraceExporter({
url: process.env.CONSEQA_INGEST_ENDPOINT + "/v1/traces",
headers: { authorization: "Bearer " + process.env.CONSEQA_ENVIRONMENT_KEY },
}),
});
sdk.start();
// Send what is still waiting before the program ends.
const stop = () => void sdk.shutdown();
process.once("beforeExit", stop);
process.once("SIGTERM", stop);Save it as instrumentation.mjs and start your application with it loaded first, so that it is in place before your own code runs:
node --env-file=.env --import ./instrumentation.mjs your-app.jsThat sends traces straight from your application to Conseqa. It is the quickest way to see coverage and needs nothing else running. Conseqa keeps a fixed list of span attributes and drops every other one on arrival, and capture of prompts, responses and tool bodies is off by default. Whatever else your spans carry still travels to Conseqa before it is dropped.
For production, send traces through the Collector in the installation bundle instead, so that content is removed inside your network and never leaves it. Point the exporter at http://127.0.0.1:4318/v1/traces with no header, or at http://otel-collector:4318/v1/traces inside the same Compose network, and start the Collector with the cloud profile shown in the next section. It keeps a queue on disk, so traces survive a restart and an outage. Verification spans link to the span of the action they check; they do not reopen a trace that has ended. Add your own attributes deliberately: do not assume every one is safe to send.
6. Connect your hosted dashboard
Create a workspace, then create an environment key in Settings as an owner or administrator. The key is shown once; store it privately. Development is for your real integration; Sample is labelled representative data, and Live demo is reserved for the scripted workspace demonstration.
Set these two values in the private installation .env, leaving the local keys unchanged:
CONSEQA_INGEST_ENDPOINT=https://www.conseqa.dev
CONSEQA_ENVIRONMENT_KEY=<your-environment-key>docker compose --profile cloud up -d
node --env-file=.env node_modules/@conseqa/cli/dist/index.js contracts publish --cloud https://www.conseqa.devInspect Runners for heartbeats and loaded hashes; Contracts for runtime deployment drift; Actions for the intent, dispatch, binding/discovery and evidence; Coverage for observed eligible-action protection; Issues for grouped exceptions and triage.
Compare models, agents and actions
A benchmark scores an agent by what it says it did. Compare, in the dashboard, groups the verdicts by the model or the agent that took each action, or by the action itself, so you can see which one gets the work done. The verified rate is verified out of settled: verified, failed, overdue and unknown. Pending actions are listed but not counted, and fewer than 20 settled actions say little.
The model and the agent come from the action's own trace (section 5): the latest span before the action with gen_ai.request.model or gen_ai.agent.name. Most agent frameworks set both when their OpenTelemetry instrumentation is on.
Your own router can read the same numbers with the environment's key, and send work to the model or provider that succeeds most. Conseqa still never sits in the request path. by is model, agent or operation; days is 1, 7, 30 or 90.
curl -H "Authorization: Bearer $CONSEQA_ENVIRONMENT_KEY" \
"https://www.conseqa.dev/api/v1/outcomes/compare?by=model&days=7"{
"by": "model", "days": 7, "considered": 1240, "truncated": false, "otherGroups": 0,
"since": "2026-10-22T09:00:00.000Z", "until": "2026-10-29T09:00:00.000Z",
"groups": [
{ "name": "model-a", "actions": 812, "verified": 771, "failed": 12, "overdue": 3,
"unknown": 6, "pending": 20, "settled": 792, "verifiedRate": 0.9735 },
{ "name": null, "actions": 18, "verified": 17, "failed": 1, "overdue": 0,
"unknown": 0, "pending": 0, "settled": 18, "verifiedRate": 0.9444 }
]
}name is null for actions whose trace reported none. The latest 20,000 actions in the window are compared; truncated says when there were more. Read it every few minutes and keep the answer: it is counts, not a live feed, and a key may read it about once a second.
// Pick the model whose actions most often verify, once it has enough of them.
const response = await fetch(
"https://www.conseqa.dev/api/v1/outcomes/compare?by=model&days=7",
{ headers: { authorization: `Bearer ${process.env.CONSEQA_ENVIRONMENT_KEY}` } },
);
const { groups } = await response.json();
const best = groups
.filter((group) => group.name !== null && group.settled >= 50)
.sort((a, b) => b.verifiedRate - a.verifiedRate)[0];
const model = best?.name ?? "your-default-model";A higher rate is a signal, not proof that one model is better: the two may have been given different work. Compare them on the same action, over the same days.
7. Hear about a failure when it happens
When an outcome fails, misses its deadline or cannot be verified, Conseqa records an issue and sends one alert for that occurrence to every destination the project has. Failures of the same kind, operation and contract are grouped into one issue. Open it from Issues or from the bell to see each occurrence, the check that failed and whether its alert went out.
A runner that stops reporting raises an issue too, once it has been silent for five minutes while its environment is still sending traces or another of its runners is still reporting. That issue resolves itself when the runner is heard from again. A runner that stops together with everything around it raises nothing, because that cannot be told apart from an agent that is simply not running.
A project owner or administrator adds destinations in Settings, under Alert destinations. Send test sends a message marked as a test, so a destination can be checked before anything fails. A destination shown as enabled has not yet shown that it receives anything: send a test and look for it.
- Slack. Add an incoming webhook to a channel in Slack and paste its address. Anyone holding that address can post to the channel, so revoke it in Slack if it leaks.
- Signed webhook. An HTTPS address of yours that resolves to a public host, and a signing secret of at least 32 characters that you choose and keep.
- Email. One or more addresses, where email alerts are enabled for the service.
Verify a signed webhook
Each delivery is a POST with a JSON body and these three headers. Check the signature against the raw body before parsing it, refuse a timestamp more than five minutes old, and ignore an identifier you have already handled, because a delivery can arrive twice.
Conseqa-Webhook-Id: <identifier of this delivery>
Conseqa-Webhook-Timestamp: <seconds since 1970>
Conseqa-Webhook-Signature: v1=<hex HMAC-SHA256 of "<timestamp>.<raw body>", keyed with your secret>import { createHmac, timingSafeEqual } from "node:crypto";
export function isFromConseqa(rawBody, headers, secret) {
const timestamp = Number(headers["conseqa-webhook-timestamp"]);
const signature = String(headers["conseqa-webhook-signature"] ?? "");
if (!Number.isFinite(timestamp) || Math.abs(Date.now() / 1000 - timestamp) > 300) return false;
if (!signature.startsWith("v1=")) return false;
const expected = createHmac("sha256", secret).update(timestamp + "." + rawBody).digest();
const received = Buffer.from(signature.slice(3), "hex");
return received.length === expected.length && timingSafeEqual(received, expected);
}The body carries id, issueId, kind, title, operation, occurredAt, actionUrl, the identifiers of the organization, project and environment, and contract when the action had one. projectName and environmentName may be absent. A test message carries "test": true; route or ignore it by that field.
Answer with a 2xx status within five seconds. Any other answer is tried again with growing pauses, up to ten attempts in all, except a 3xx or 4xx other than 408, 409 or 429, which is not tried again.
8. Test failures offline
Contract Regression does not execute the agent, model, production action, or installed verifier callbacks. The CLI evaluates data-only predicate assertions and recorded lifecycle transitions. Manually author exact predicates when you own their original meaning; do not guess operators from dashboard display text.
node node_modules/@conseqa/cli/dist/index.js contracts test
node node_modules/@conseqa/cli/dist/index.js contracts test acme.resource.ready --json regression-results.json --junit regression-results.xmlA supported saved action offers a recorded lifecycle export. This uses its real retained redacted registration, complete immutable event sequence, pinned hash and recorded verdict. It does not reconstruct removed fields or re-evaluate the original provider predicates. Missing, incomplete, oversized or unsafe retained data disables export with a reason.
Use your original matching built contract package. After building, save the downloaded fixture as dist/conseqa/regressions.json and run the CLI test. Keep the build hash unchanged; rebuilding can overwrite fixtures. The current CLI accepts one fixture file at that path—review merging cases before replacing an existing one.
When the saved action's contract declares expectations, the export also includes a re-check case: the newest version of that contract in your build re-judges the saved observation, and the case fails if your changed rules would change the saved verdict. Values that redaction removed, replaced or pseudonymized are never guessed; such a case is reported as not run, with the reason. The Contracts page offers the same re-check across the latest 200 finished actions.
Only if you choose to publish result metadata, run:
node --env-file=.env node_modules/@conseqa/cli/dist/index.js contracts test --upload https://www.conseqa.devThe upload contains test names, hashes, verdicts and bounded diagnostics, not fixture bodies or executable code. Review those operator-authored names/diagnostics for identifiers before uploading.
When something does not work
What each message means and what to do about it. In every case below, nothing was sent to your provider unless the entry says so. Every setting, field, command and state is listed in the reference.
ProtectionUnavailableError: Required Conseqa intent could not be durably registered- The SDK could not record the intent with the runner, so in
requiredmode it did not make the call. The end of the message says why. If the runner did not answer, start it, check therunnerUrland the sidecar token your application passes, and rundoctor. If it saysthe runner rejected it, the code that follows names the problem, such ascontract_intent_schema_mismatchfor an intent that does not match the contract's schema. ActionKeyConflictError: Action key … is already registered with a different intent- The key is already used for an action with different details, so the call was not made, in either mode. Retrying will not help: two different actions share one key. Give each operation its own key. The error carries the identifier of the earlier action.
ConseqaValidationError, such asactionKey must be text of 1 to 512 characters- An argument the SDK cannot use. Nothing was registered or sent; the error's
fieldnames the argument to fix. DuplicateActionError: Action key … is already registered- That action key has been used before, and Conseqa will not run the same business operation twice. Use one stable key for each operation, such as the ticket or order it belongs to. The error carries the identifier of the action that already exists.
… is already installed with a different build hashorConseqa refused this contract (409)- That contract version or package version already exists with different contents, and a published version never changes. A new build needs new numbers: in
conseqa.contracts.jsonraisepackageVersionand theversionof each contract, and the version in your contract code too. With a ready-made connector, pass it to the connector, for examplecreateStripeRefundSucceededContract({ version: 2 }). Then build, install and publish again. The refused build is not kept. A runner older than 0.1.10 kept it in its contracts folder; on its next start the runner moves it to.conseqa/contracts/.set-asideand keeps running the installed build. The build in dist/conseqa is older than your sources- You changed the contract, its schemas or its saved cases after the last build, so
contracts testwould test the old one. Runcontracts build, then test again. conseqa.contracts.json has problems- Each line names a setting and what is wrong with it. A setting Conseqa does not know, often a misspelling, is an error rather than being ignored, and the message suggests the name it expected.
Cloud rejected contract (401)- The environment key is wrong, incomplete or revoked. Make a new one in Settings and paste the whole key into the installation's
.env. The runner refused the sidecar token- The token your application or
doctorsent is not the runner'sCONSEQA_SIDECAR_TOKEN. Both must read the same value. Nothing is listening at http://127.0.0.1:4319- No runner is running at that address. Start it with
npm run runner(or its container), or pass the address it uses with--runner. Conseqa runner could not start: …- A setting is missing or unusable. The message names the variable, what it must be, and the value it was given, such as
CONSEQA_PORT must be an integer between 1 and 65535. Build … is not installed in this runneroris installed but not loaded- The runner does not run the contract build your application loaded. Run
contracts build, thencontracts install, and restart the application so both use the same build. When the build is installed but not loaded, the message gives the reason, such as a schema the runner cannot compile. The runner reports itself degraded: cloud_delivery_failing- The runner has failed to send results to the dashboard several times in a row. Its own records are safe and it keeps trying, waiting as long as the service asks. Check
CONSEQA_ENVIRONMENT_KEYand that the runner can reach the service; an event the service refuses as invalid is set aside after five attempts. - An action stays Unknown
- The runner could not read anything conclusive: the read failed, or a lost reply could not be matched to one record. Check the runner's read-only credential and that it can reach your system. The action's page shows each attempt. Unknown is a truthful answer and is never turned into a guess.
- An action ends Overdue
- The record never reached the state the contract expects before its deadline. If your system is simply slower than that, raise
deadlineMs. Otherwise this is a real finding. - Nothing appears in the dashboard
- The Runners page shows when each runner last reported. If yours is missing, its
.envlacksCONSEQA_INGEST_ENDPOINTorCONSEQA_ENVIRONMENT_KEY, or it was started before they were set. Also check that the dashboard is showing the environment the key belongs to. - Coverage shows a dash
- Coverage needs your traces to mark the operation, as described under Traces. Verdicts do not: an action is verified with or without a trace.
Before protecting customer actions
Validate required-mode fail-closed behaviour, exact discovery, multiple-candidate Unknown, provider auth/schema errors, restart recovery, secret redaction and your source-of-record permissions. A verified outcome is an evaluation of observed state at recorded times, not bank-settlement proof. Where a provider can undo a change later, the runner keeps watching it: a succeeded Stripe refund is read again for 72 hours (set CONSEQA_STRIPE_REFUND_WATCH_SECONDS on the runner; 0 turns it off), and if it fails the action turns Failed, opens an issue and sends an alert.
Keep port 4319 private, persist the runner and Collector volumes, and back up the full stopped SQLite/spool/contracts state coherently with matching keys stored separately. Never run docker compose down -v against production volumes. High availability, SSO, automatic retries/remediation, and model replay are not provided by v1.
The hosted service is in early access. Signing up needs an email address you can confirm, usage has monthly limits that inform and never block, support is by email, and there is no uptime commitment yet. Your runners do not depend on it: they keep verifying while the hosted service cannot be reached, and send what they recorded when it returns.
Need help with your first contract? Contact Devraj Jhala. Send a description, never a password, token, connection string or raw customer record.