Judge every AI decision
before it ships
Ready-made judges for evals, guardrails, agent tool calls, moderation, routing and scoring, or build your own. One API call each, calibrated by Jev.
500 free judgments a month · no credit card
76 ready-made judges for the decisions teams ship every day. They work on output from GPT, Claude, Gemini, Llama, Mistral or your own model.
- RAG answer graderpassneeds reviewfail
- Citation checkersupportedpartly supportedunsupported
- Support reply graderready to sendedit firstescalate
- Prompt injection guardallowreviewblock
- PII detectorno personal datapossible piicontains pii
- Brand voice checkon brandreviseoff brand
- Is the task done?donedone, but unprovennot done
- Model routersmall modelmid-size modelfrontier model
- Context relevance filterkeepcompressdrop
- Community moderatorapprovesend to moderatorsremove
- Scam message detectorlooks safesuspiciousscam
- Marketplace listing checkapprovereviewreject
- Bot intent classifierallowslow downblock
- Post virality scorerready to postrework itwon't travel
- Ad creative teardownstrongaverageweak
- Low-effort reply filterkeeplow efforthide
- SEO & GEO page auditstrongimproverewrite
- Comment & feedback analyzertoxicsupportivecritical
- Ad brief scorershoot itrevisecut it
- Draft sentence checkkeeptightenfix
- Outreach message checkersendimproverewrite
- Sales call scorerstrong callcoaching momentsdeal at risk
- Inbox triageurgentaction neededspam
- Alert & log triagepage on-callopen a ticketignore
- Resume screeneradvancehuman reviewbelow the bar
- Interview feedback checkersolid feedbackincompletepossible bias
- Contract risk checkstandard termslegal reviewhigh risk
- Transaction risk screenclearinvestigateescalate
- PR risk reviewsafe to mergeneeds careful reviewchanges requested
- Commit message checklooks goodamendblock
- Package risk checkallowreviewblock
- Judge question reviewgood to goimproverewrite
- Survey response coderpositivenegativemixed
- Sentiment & emotionvery negativenegativeneutral
- News & event severityalertwatchlog
- Novelty check: new, update or duplicate?newupdateduplicate
- Product feedback triagecritical buguser at riskbug
- Startup idea roastship itfix itkill it
- Hallucination detectorgroundedsuspecthallucinated
- Summary checkergood summaryneeds editsunfaithful
- Claim vs evidencesupportednot settledcontradicted
- LLM reply guardallowreviewblock
- Off-topic guardanswerdeflect politelysmall talk
- Tool-call firewallallowask a humandeny
- Agent loop & drift detectorcontinuenudge: change approachstop and escalate
- Tool routerask to clarifyconfirm firstweb search
- Memory gatesave itreplace a lineskip
- Spam detectornot spammaybe spamspam
- Fake review detectorgenuinesuspiciouslikely fake
- Conversation red-flag detectorlooks finebe carefulred flags
- AI slop detectorhuman & usefulsome slopai slop
- Short-form clip finderclip itmaybeskip
- Feed triage: read, skim or skipmust readskimskip
- Buying-signal finderreply nowworth a lookskip
- Search result beatabilitybeatableneeds a humanstrong
- Ad anatomy labelerother formattalking head democustomer testimonial
- Audience reactionlandsworksrework
- Lead qualifierhotwarmcold
- Deal review routerstandard: approveneeds reviewsmissing documents
- Support ticket routerneeds triageurgentsend a docs link
- Churn risk detectorhealthyat risksave now
- Customer chat monitornotify the ownerkeep an eye on itnothing needed
- Application reviewershortlistmaybenot now
- Expense categorizerneeds reviewpersonal?software
- Marketing claims compliancecompliantreviseblock
- Insurance claim triagestraight-throughrequest informationspecialist review
- Semantic code lintcleanwarningsviolations
- PR labelerfeaturebug fixrefactor
- Vulnerable code checkcleanreviewvulnerable
- Should this be a judge?use a judgesplit itkeep the llm
- Systematic review screenerincludefull-text reviewexclude
- Record matcher (dedupe)matchpossible matchdifferent
- Research desk routerno deskpapers deskrepos desk
- User intent classifierask to clarifyask a questionreport a problem
- Landing page clarity auditclearfixableconfusing
- Dark pattern detectorfairquestionablemanipulative
Questions in. A verdict out.
A judge asks Jev a few closed questions about an answer. Jev returns calibrated probabilities in one call, and rules you can read turn them into pass, flag or fail.
sourceRefunds are processed within 24 hours and then arrive in 5-7 business days.
- Is every claim backed by the source?…yes/no
- Does it answer what was asked?…yes/no
- Is it safe and on policy?…yes/no
- How good is it, from 0 to 4?…score 0–4
The dark tick on each meter is the line it has to clear.
safe < 0.50failgrounded < 0.40failany check under its lineflagotherwisepass
The failures your users screenshot
Made-up facts, dodged questions, unsafe replies and sloppy answers, caught before anyone sees them.
Catches made-up facts. Pass the source you retrieved and every claim is checked against it.
Refunds are processed within 24 hours and then arrive in 5-7 business days.
“Your refund of $250 was sent to your PayPal and will arrive tomorrow at 9am.”
Did it actually answer? Refusals, dodges and off-topic replies score low, even when they’re polite.
Stops the embarrassing stuff. Abuse, leaked personal data and harmful content come back as a failing safety check.
A 0–4 score, with the reason. Every answer gets an expected score on a five-level rubric, and names its level.
detail: “Good: accurate and clear”
A judge for every AI decision
Every pattern people build with Jev, as a judge you can try live, call as-is, or fork and make your own.
See all 76 use cases- AI output quality
RAG answer grader
Is the answer grounded in the retrieved source, and does it answer the question?
PassNeeds reviewFail5 questions - AI output quality
Hallucination detector
Catch invented numbers, names and claims before your users do, with the sentence quoted.
GroundedSuspectHallucinated4 questions - Guardrails
Prompt injection guard
Screen user messages for jailbreaks, injected instructions and data exfiltration.
AllowReviewBlock7 questions - Guardrails
LLM reply guard
Check every model reply for policy breaks, unsafe advice and leaked data before it's shown.
AllowReviewBlock+18 questions - Agents
Tool-call firewall
Judge every agent tool call before it touches your machine: allow, ask a human, or deny.
AllowAsk a humanDeny10 questions - Agents
Is the task done?
Don't trust an agent's "done". Check the transcript for proof, open errors and skipped work.
DoneDone, but unprovenNot done5 questions - Agents
Model router
Send each prompt to the cheapest model that can handle it.
Small modelMid-size modelFrontier model+14 questions - Trust & safety
Community moderator
Approve, review or remove posts against your own community rules, with severity.
ApproveSend to moderatorsRemove+18 questions - Trust & safety
Scam message detector
Paste an SMS, email or DM and get the probability it's a scam, and why.
Looks safeSuspiciousScam7 questions - Content & growth
AI slop detector
Stamp generic, filler-stuffed, template-shaped AI writing, with the tells quoted.
Human & usefulSome slopAI slop7 questions - Content & growth
Post virality scorer
Score a draft post's hook, specificity, novelty and shareability before you publish.
Ready to postRework itWon't travel7 questions - Sales
Lead qualifier
Score every lead against your ideal customer profile: hot, warm or cold.
HotWarmCold6 questions - Support & ops
Support ticket router
Route every ticket to the right team, fast-track the urgent ones, auto-answer the easy ones.
Needs triageUrgentSend a docs link+65 questions - Recruiting
Resume screener
Check each resume against the bar you define, with the supporting line quoted.
AdvanceHuman reviewBelow the bar6 questions - Finance & compliance
Expense categorizer
Put every transaction in the right bookkeeping category, and flag what looks personal.
Needs reviewPersonal?Software+123 questions - Engineering
PR risk review
Is this pull request safe to merge, or does it need a careful human look?
Safe to mergeNeeds careful reviewChanges requested7 questions - Research & data
Survey response coder
Code open-ended survey answers into themes and sentiment, thousands at a time.
PositiveNegativeMixed+18 questions - Product
User intent classifier
What does the user want, and are they telling the app to do something right now?
Ask to clarifyAsk a questionReport a problem+72 questions
- 01Refunds arrive 5–7 business days after approval.Pass
- 02Your refund of $250 was sent to PayPal today.Fail
- 03Refunds are possible, just reach out to us.Flag
- 04Annual plans are refundable within 30 days.Pass
- 05I've extended your free trial to 90 days.Fail
One request. A typed verdict back.
Call it from any stack, branch on the verdict, and fail the build when answers get worse.
curl -X POST https://tryevally.com/api/judge \ -H "Authorization: Bearer $EVALLY_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "question": "When will my refund arrive?", "context": "Refunds arrive in 5-7 business days.", "output": "Your refund should arrive in 5-7 business days." }'{ "overall": "pass", "checks": [ { "key": "grounded", "value": 0.78, "pass": true }, { "key": "answers", "value": 0.88, "pass": true }, { "key": "safe", "value": 0.98, "pass": true }, { "key": "quality", "value": 2.97, "pass": true, "detail": "Good: accurate and clear" } ], "ms": 624, "model": "jev-1.13.0"}Numbers you can branch on
Calibrated scores, answers in about half a second, a few dozen tokens out, and every call on the record.
Probabilities you can threshold. Every yes/no check returns p(yes) from 0 to 1, so 0.78 means what it says. You pick the line where you ship.
What comes backGrounded example · the dark tick is each check’s line
About half a second. Every check runs in the same call, in parallel. These are real runs from our playground.
Try the playgroundSmall in, tiny out. A judgment reads your question, source and answer, then sends back a few dozen tokens of verdict: 492 in, 68 out on this example.
Response referenceEvery call, on the record. The verdict, latency and tokens of each judgment land in your dashboard, next to the key that made it.
Open the dashboardA model built to decide, not to chat.
Jev is TypeSafe’s System One model. It doesn’t write answers, it makes decisions: calibrated yes/no probabilities, choices and scores that come back typed. Evally turns them into a verdict you can branch on.
- Typed verdicts: pass, flag or fail, every time
- It can't answer outside the schema
- Every check in one call, in about half a second
- Calibrated 0–1 probabilities you can threshold
“Overall the answer seems mostly accurate and helpful, though I’m fairly sure some details may not fully match the source…”
- grounded0.78
- answers0.88
- safe0.98
Questions, answered
Can’t find it here? The docs cover every endpoint.
What is Evally?
An API and dashboard for judges: small sets of closed questions that Jev answers with calibrated probabilities, plus rules in code that turn the answers into a verdict. Use one of 60+ ready-made judges or build your own, then run it live, from your code, or over a whole dataset.
What can it judge?
Anything you can put into words: AI answers (grounding, hallucinations, citations), prompts and replies (jailbreaks, unsafe advice, leaked data), agent tool calls, posts and reviews, tickets and emails, leads, resumes, contracts, pull requests and more. The use-case gallery has a judge for each.
Can I change a judge?
Yes. Fork any template and edit its questions, options, thresholds and verdicts in the builder, test the draft live, and save it. Each save is a new version, and API calls can pin a version.
Which models does it work with?
Any of them. Evally judges the text, so it doesn't matter whether the answer came from GPT, Claude, Gemini, Llama or your own fine-tune.
How is it different from LLM-as-judge?
An LLM judge writes an opinion you have to parse, and it can make things up. Evally runs on Jev, which answers typed questions with calibrated probabilities, so the verdict is always one of three values and every score can be thresholded.
What is Jev?
Jev is TypeSafe's System One model. Instead of generating prose it answers typed questions: yes/no probabilities, choices and scores. Evally uses it as the judge behind every check.
How do I authenticate?
Create a key in the dashboard and send it as a bearer token: Authorization: Bearer evl_live_… Keys are stored hashed, shown once, and can be revoked at any time.
What comes back?
JSON with the verdict, whether it needs a person's review, an optional weighted score, and every question's typed answer: a probability for yes/no, the chosen option with every option's probability, or a value on a rubric. Yes/no answers can quote the sentence that decided them.
How fast is it?
Most judges answer in about 0.4 seconds. Every question in a judge runs in the same call, in parallel.
Are there input limits?
Up to 20,000 characters per input field and 48,000 per run, up to 40 questions per judge, and 5,000 rows per batch.
Is there a free plan?
Yes. The Hobby plan includes 500 judgments a month, with no credit card needed.
What counts as a judgment?
One call to Jev: one judge run however many questions it asks, one batch row, one decide call, or up to 20 ranked items.
Put a judge in front of your next release
500 free judgments a month. No credit card, no sales call.