Evally
Start free
Engineering

Judge question review

Review a question before Jev answers it: two judgments in one, inputs no option fits, a rating hiding in a yes/no.

A badly designed question still gets a confident answer. Published tests found pick-one questions with no "none of these" option sending out-of-scope inputs to an option at 0.99 confidence. This judge reviews the parts of a question that need judgment: is it really two questions in one, would some realistic inputs fit none of its options, is it a rating asked as a yes/no. Each check was calibrated on questions from Evally's own judges and on deliberately broken ones; Evally's judge builder runs it when you press Check with Jev.

Use this judgeCustomize it

Try it live

Real calls to Jev. Pick an example or paste your own.

Examples
40
10
30
No verdict yet

Pick an example or paste your own input, then run it.

Under the hood

Every question and rule, in the open. Fork it to change any of them.

What it asks

Reads question, options and data, then asks 3 questions in one call.

  1. Q1Two questions in oneYes / no

    Does the question ask for two or more separate judgments at once, so that one answer couldn't say which part was true?

  2. Q2Inputs no option fitsYes / no

    Picture the different inputs this question will be asked about. Would some realistic ones fit none of the options, so that a wrong option would have to be picked? If the options aren't a pick-one list, or one already means "other" or "none of these", answer no.

  3. Q3A rating asked as yes/noYes / no

    Does the question really ask how much, how well or how likely, so it ought to be a rating instead of a yes/no? If the options are a pick-one list or rating levels, answer no.

How it decides

Rules run in code, top to bottom. The first one that matches gives the verdict.

  1. 1
    If Two questions in one ≥ 0.8Rewrite
  2. 2
    If Two questions in one ≥ 0.7Improve
  3. 3
    If Inputs no option fits ≥ 0.75Improve
  4. 4
    If A rating asked as yes/no ≥ 0.6Improve
  5. ·
    DefaultGood to go

Call it from your code

Every template runs as-is with an API key, no setup: POST /api/v1/judges/question-check/run.

curl -X POST https://tryevally.com/api/v1/judges/question-check/run \  -H "Authorization: Bearer $EVALLY_API_KEY" \  -H "Content-Type: application/json" \  -d '{  "input": {    "question": "Is this ticket urgent and about billing?",    "options": "Yes or no.",    "data": "ticket: the customer'\''s message"  }}'

Seen in the wild

Community builds on Jev that use this pattern. Not affiliated with Evally.

More in Engineering