Typed verdicts for every AI decision

Authentication#

Send your key as a bearer token: Authorization: Bearer evl_live_…. Keys are stored hashed, shown once, and can be revoked from the dashboard. A revoked or unknown key returns 401. Keep keys on your server; never ship them to a browser.

MCP server#

Evally is also an MCP server, so Claude Code, Cursor, Codex and other MCP clients can find and run judges as tools. It speaks Streamable HTTP at https://tryevally.com/api/mcp; send your API key as a bearer token.

1claude mcp add --transport http evally https://tryevally.com/api/mcp \2  --header "Authorization: Bearer $EVALLY_API_KEY"

1{2  "mcpServers": {3    "evally": {4      "url": "https://tryevally.com/api/mcp",5      "headers": { "Authorization": "Bearer evl_live_…" }6    }7  }8}

Every tool calls the same endpoint as the REST API, so runs are metered and limited exactly the same way.

How judges work#

A judge is plain JSON in four parts, and every run is one call to Jev, however many questions it asks.

An optional weighted score combines answers into one 0-1 number that rules can threshold, like a lead score or a slop score.

1{2  "v": 1,3  "fields": [{ "key": "reply", "label": "Reply", "long": true }],4  "questions": [5    { "type": "yes_no", "key": "refund_promise", "label": "Promises a refund",6      "ask": "Does the `reply` promise a refund? (Explaining the refund policy doesn't count.)",7      "evidence": "reply" },8    { "type": "score", "key": "tone", "label": "Tone",9      "ask": "How is the tone of the `reply`?", "levels": ["Rude", "Cold", "Polite", "Warm"] }10  ],11  "outcomes": [12    { "key": "send", "label": "Send", "tone": "pass" },13    { "key": "hold", "label": "Hold for review", "tone": "flag", "review": true }14  ],15  "rules": [16    { "when": [{ "q": "refund_promise", "op": "gte", "value": 0.6 }], "then": "hold" },17    { "when": [{ "q": "tone", "op": "lt", "value": 1.5 }], "then": "hold" }18  ],19  "fallback": "send"20}

Run a judge#

Body: { "input": { … }, "version"?: 3 }. One call is one judgment, however many questions the judge asks.

Input#

Send input as an object keyed by the judge’s fields, or as a plain string when the judge has one field. Values are trimmed; each field is capped at 20,000 characters and 48,000 in total. Required fields must be non-empty, and unknown keys are ignored.

Response#

1{2  "id": 48213,3  "verdict": "hallucinated",4  "label": "Hallucinated",5  "tone": "fail",6  "needs_review": false,7  "score": null,8  "rule": 1,9  "answers": {10    "unsupported":   { "type": "yes_no", "value": 0.99,11                       "evidence": { "text": "Pro includes 50,000 judgments a month and phone support.", "p": 0.97 } },12    "numbers":       { "type": "yes_no", "value": 0.06 },13    "entities":      { "type": "yes_no", "value": 0.11 },14    "contradiction": { "type": "yes_no", "value": 0.92 }15  },16  "judge": { "slug": "hallucination-detector", "name": "Hallucination detector", "version": null, "template": true },17  "ms": 394,18  "model": "jev-1.13.0",19  "usage": { "input_tokens": 473, "output_tokens": 20 }20}

1// yes/no: the probability that the answer is yes, and optionally the sentence that shows it2{ "type": "yes_no", "value": 0.93, "evidence": { "text": "…", "p": 0.81 } }3 4// pick one: the chosen option, how sure, and every option's probability5{ "type": "choice", "choice": "billing", "confidence": 0.88,6  "probabilities": { "billing": 0.88, "bug": 0.1, "how_to": 0.02 } }7 8// score: the expected value on the rubric (0 to max), the most likely level, and its distribution9{ "type": "score", "value": 2.4, "max": 3, "confidence": 0.61,10  "level": { "index": 2, "text": "Good fit" }, "probabilities": [0.02, 0.1, 0.34, 0.54] }

Versions#

Saving a changed judge creates a new version; calls use the latest. To keep production on a version you’ve tested, pass "version": 3. The version that ran is in judge.version and on every logged run. Templates aren’t versioned: fork one to pin it.

Line by line#

POST /api/v1/judges/{slug}/lines runs a judge on every line of a text: a speech, a script, a draft, a transcript. Send text (one line per line, or a paragraph to split into sentences) or lines; field names the input each line goes into (the judge’s first field by default), before an input that receives the three lines before it, and context fills the judge’s other inputs for every line. Up to 60 lines; each is one judgment. The Audience test is this endpoint running the audience-reaction judge.

1// Every sentence of a draft through the draft-sentence-check judge.2const res = await fetch("https://tryevally.com/api/v1/judges/draft-sentence-check/lines", {3  method: "POST",4  headers: { Authorization: "Bearer " + process.env.EVALLY_API_KEY, "Content-Type": "application/json" },5  body: JSON.stringify({6    text: draft,                       // one line per line, or a paragraph to split into sentences7    field: "sentence",                 // the input each line goes into8    context: { goal: "Get developers to try the free plan", sources: releaseNotes },9  }),10});11const { lines } = await res.json();12const toFix = lines.filter((l) => l.verdict !== "keep"); // rewrite only these

Test a spec#

POST /api/v1/judges/test with { "spec": { … }, "input": { … } } runs a spec without saving it. It’s what the builder’s test panel uses, and handy for checking a spec from code before you save it. Invalid specs return 400 with an issues array of paths and messages.

Spec check#

A badly designed question still gets a confident answer: a pick-one with no “none of these” option sends inputs that fit nothing to some option at 0.94-0.99 confidence, past any confidence gate. POST /api/v1/judges/check with { "spec": { … } } returns the structural checks for free: missing escape options, ratings asked as yes/no, two questions in one, references to fields that don’t exist, levels that are bare numbers. Add "semantic": true and Jev also reviews each question (one judgment per question). The builder shows the same checks as you edit.

Templates#

GET /api/v1/templates lists every template with its fields and verdicts; GET /api/v1/templates/{slug} returns its full spec and sample inputs. No key needed.

Flows#

A flow chains judges into the decisions an agent team makes about every item: worth reading, which agent takes it, does the claim hold, have we seen it. Each stage is a judge whose verdicts route the item to the next stage or to a lane: an agent’s queue, a deliver step, store, escalate to a person or a bigger model, or drop. Entries let work come back in, so a desk’s note can re-enter at the fact-check. Build and watch flows from the app; see the control room.

Body: { "input": { … }, "entry"?: "review" }. Every stage an item passes is one judgment, so an item that is skipped at the first gate costs one. Ready-made flows also run on the public demo allowance, one demo run per item.

1// A new item comes in: screen it, route it, hand it to the lane's owner.2const res = await fetch("https://tryevally.com/api/v1/flows/research-desk/run", {3  method: "POST",4  headers: { Authorization: "Bearer " + process.env.EVALLY_API_KEY, "Content-Type": "application/json" },5  body: JSON.stringify({ entry: "intake", input: { title, text, source: "arxiv" } }),6});7const { lane, path } = await res.json();8// lane: { key: "papers", label: "Papers desk", kind: "agent" }9// path: every stage it passed, with its verdict, signal and time10 11if (lane.kind === "agent") await queues[lane.key].push(item);12 13// Later, the desk's note comes back in at the fact-check:14await fetch("https://tryevally.com/api/v1/flows/research-desk/run", {15  method: "POST",16  headers: { Authorization: "Bearer " + process.env.EVALLY_API_KEY, "Content-Type": "application/json" },17  body: JSON.stringify({ entry: "review", input: { title, text, note } }),18});

Is it a judge?#

POST /api/v1/swap with { "prompt": "…", "usage"?: "…" } reads an LLM call from your code and says whether it’s really a decision: swap (a judge can take it over), split (move the decision into a judge, keep the LLM for the writing) or keep. For swap and split it suggests up to three ready-made judges. One judgment for the check plus one per 20 judges ranked. Try it on a prompt.

Rank#

Rerank search results, recommendations or candidates against a query. Jev scores each item’s relevance on a five-level rubric and code sorts them. Up to 200 items per call; every 20 items is one judgment. Pass question to redefine relevance, referring to `item` and `query`, e.g. “How likely is `item` to hire the candidate in `query`?”

1// POST /api/v1/rank2{3  "query": "I can't log in after changing my phone",4  "items": [5    "How to change your billing address",6    { "id": "kb-42", "text": "Recovering your account when you lose your authenticator" },7    "Resetting a forgotten password"8  ],9  "top_k": 210}11 12// 200 OK: best first; score is relevance from 0 to 113{14  "results": [15    { "index": 1, "id": "kb-42", "score": 0.96, "level": 4, "label": "Exactly what the query is looking for" },16    { "index": 2, "id": null, "score": 0.71, "level": 3, "label": "Relevant" }17  ],18  "judgments": 1, "ms": 412, "model": "jev-1.13.0", "usage": { "input_tokens": 388, "output_tokens": 45 }19}

Decide#

For agents, games and anything that needs Jev’s primitives directly: send any state and up to 64 questions in TypeSafe’s format (noul, choice with up to 255 options, score with 2-10 levels). You get Jev’s answers back unchanged, with your key, metering and run log in front. Requires an API key or a session.

1// POST /api/v1/decide: TypeSafe's primitives, metered and logged2{3  "state": { "screen": "Checkout", "cart_total": 129.5, "user_message": "use my saved card" },4  "questions": {5    "action": { "type": "choice", "instructions": "What should the agent do next?",6                "criteria": { "pay_saved_card": null, "ask_for_card": null, "cancel": null } },7    "irreversible": { "type": "noul", "instructions": "Would the next action spend money?" }8  }9}10 11// 200 OK12{ "id": 48219,13  "answers": {14    "action": { "type": "choice", "choice": "pay_saved_card", "confidence": 0.94, "probabilities": { … } },15    "irreversible": { "type": "noul", "noul": 0.97 }16  },17  "model": "jev-1.13.0", "ms": 288, "usage": { … } }

Quick eval#

The original endpoint, POST /api/judge with output, optional context and question, still runs four fixed checks (grounded, answers, safe, quality) and returns overall: pass, flag or fail. New integrations should use the RAG answer grader judge, which asks the same questions with editable rules.

Batches#

Run a judge over a whole dataset from the app: upload a CSV or JSON Lines file (up to 5,000 rows), match its columns to the judge’s fields, and start. Add a column of expected verdicts to see how often the judge agrees with you, then open the disagreements. Results export as CSV with every answer. Each row is one judgment.

Ad research#

An ad study is a batch with a purpose: load a category’s ads with how long each has been running (days, or a start date), and the ad-anatomy judge labels each one’s format, hook, angle, awareness stage, offer, proof, objection and call to action. The study page maps every ad by days live and variants, compares each pattern’s survival rate with the category’s, and, when the ads carry a brand, builds the category playbook: the patterns two or more brands each keep a long-runner with. Labels that were a coin flip between two options (under five points apart) are listed for a person to check. It also lists the survivors’ opening lines and scores briefs with ad-brief-scorer: hook, brand fit, survival odds, and whether every claim is backed by your product facts. Up to 2,000 ads per study; each ad is one judgment, and so is each brief. Both judges are templates, so you can call them directly too. See a finished study.

Keyword research#

A keyword study is a batch of search results: upload page one for each keyword (the title and link of every result, plus the snippet, publish date, domain rating and search volume when your SEO tool has them), and the serp-beatability judge asks five questions of each result: does it answer the exact search, is it a forum thread, a thin page, out of date, on a strong site. A result that fails any of them is beatable, and a keyword with three or more beatable results on page one is takeable (you can set the bar from two to five). The study page shows every keyword’s page one, why its results are beatable, the keyword to go after first, and the results Jev was split on. Only the top ten per keyword are read, up to 2,000 results per study; each result is one judgment. See a finished study.

Review queue#

Any outcome can be marked for review. Runs that land on it, from the API, the app or a batch, wait in your review queue, where a person confirms or corrects the verdict. Each judge’s page tracks how often reviewers agree, so you know when it can be trusted with more. The API also returns needs_review: true so your code can hold the result.

Limits and metering#

Errors#

Errors return JSON as { "error": "…" }.

Guardrail pattern#

Screen what goes into your LLM and what comes out, with two ready-made judges. Each check costs one judgment and a few hundred milliseconds.

1const guard = (slug, input) =>2  fetch(`https://tryevally.com/api/v1/judges/${slug}/run`, {3    method: "POST",4    headers: { Authorization: "Bearer " + process.env.EVALLY_API_KEY, "Content-Type": "application/json" },5    body: JSON.stringify({ input }),6  }).then((r) => r.json());7 8async function safeChat(userMessage) {9  const inbound = await guard("prompt-injection-guard", { message: userMessage });10  if (inbound.verdict === "block") return "I can't help with that.";11 12  const reply = await llm(userMessage);13  const outbound = await guard("llm-reply-guard", { prompt: userMessage, reply });14  if (outbound.verdict === "support") return CRISIS_RESOURCES;15  if (outbound.verdict === "block") return "Let me connect you with someone who can help.";16  if (outbound.needs_review) queueForHuman(reply); // shows up in your review queue too17  return reply;18}

Agent firewall#

Judge every tool call against the task before it runs. The agent never grades itself, and the decision is in code you control.

1// Before your agent runs any tool call:2const check = await fetch("https://tryevally.com/api/v1/judges/tool-call-firewall/run", {3  method: "POST",4  headers: { Authorization: "Bearer " + process.env.EVALLY_API_KEY, "Content-Type": "application/json" },5  body: JSON.stringify({ input: { task: agent.task, tool_call: JSON.stringify(call) } }),6}).then((r) => r.json());7 8if (check.verdict === "deny") return agent.refuse(call, check.answers);9if (check.verdict === "ask") return agent.askHuman(call);   // needs_review is true10return agent.run(call);                                      // "allow"

Confidence routing#

Every pick-one and score answer carries a confidence. Use it as a second axis: act automatically when it’s high, ask for confirmation in the middle, and send low-confidence cases to a person or a bigger model. In a judge, that’s a rule like { "q": "team", "op": "conf_lt", "value": 0.5 } → an outcome marked for review. Check 50-100 cases you know the answers to before you trust any threshold; a batch with an expected column does exactly that.