Evally
Start free
Engineering

Should this be a judge?

Paste a prompt from your code. Find out whether it's really a decision a judge would answer faster and cheaper.

Plenty of LLM calls in production are decisions in disguise: route this ticket, flag this spam, score this lead, then parse one word out of the reply. A builder who scanned over 1,000 repos found them everywhere. Paste the prompt, and optionally what your code does with the reply: Jev says what kind of answer it really wants, whether that answer is a closed set, and whether a judge can take it over.

Use this judgeCustomize it

Try it live

Real calls to Jev. Pick an example or paste your own.

Examples
173
57
No verdict yet

Pick an example or paste your own input, then run it.

Under the hood

Every question and rule, in the open. Fork it to change any of them.

What it asks

Reads prompt and usage, then asks 4 questions in one call.

  1. Q1What it asks forPick one

    What does the prompt ask the model to produce?

    Yes or noA labelA ratingA rankingExtracted valuesWritingSeveralOther
  2. Q2Closed answerYes / no

    Is every acceptable answer to the prompt one of a small, known set of values, rather than open-ended text?

  3. Q3Parsed by codeYes / no

    Does the code parse the reply to branch, filter or store a value instead of showing it to a person? Judge from the usage, or from the prompt's output instructions if there's no usage.

  4. Q4Needs real reasoningYes / no

    Does answering the prompt require calculation, counting or several steps of reasoning, rather than a judgment about the text it's given?

How it decides

Rules run in code, top to bottom. The first one that matches gives the verdict.

  1. 1
    If Needs real reasoning ≥ 0.7Keep the LLM
  2. 2
    If What it asks for is WritingKeep the LLM
  3. 3
    If What it asks for is SeveralSplit it
  4. 4
    If Closed answer ≥ 0.6Use a judge
  5. 5
    If Parsed by code ≥ 0.6 and What it asks for is not OtherSplit it
  6. ·
    DefaultKeep the LLM

Call it from your code

Every template runs as-is with an API key, no setup: POST /api/v1/judges/llm-call-auditor/run.

curl -X POST https://tryevally.com/api/v1/judges/llm-call-auditor/run \  -H "Authorization: Bearer $EVALLY_API_KEY" \  -H "Content-Type: application/json" \  -d '{  "input": {    "prompt": "You are a support assistant. Classify the customer message into one of: billing, bug, feature_request, account, other. Reply with only the category name.\n\nMessage: {message}",    "usage": "category = reply.strip().lower()\nqueue = QUEUES[category]"  }}'

Seen in the wild

Community builds on Jev that use this pattern. Not affiliated with Evally.

More in Engineering