Evally
Start free
AI output quality

Hallucination detector

Catch invented numbers, names and claims before your users do, with the sentence quoted.

Checks a model's output against the source it was given and quotes the sentence that went wrong. Built for support bots, summaries and RAG answers, where one invented price or policy is a real incident.

Use this judgeCustomize it

Try it live

Real calls to Jev. Pick an example or paste your own.

Examples
98
109
No verdict yet

Pick an example or paste your own input, then run it.

Under the hood

Every question and rule, in the open. Fork it to change any of them.

What it asks

Reads source and answer, then asks 4 questions in one call.

  1. Q1States an unsupported factYes / no quotes evidence

    Does the answer state a fact that the source does not support? (Rewording and common knowledge such as 'Paris is in France' don't count. Any specific number, name, date, price or policy that isn't in the source does.)

  2. Q2Invented numbersYes / no quotes evidence

    Does the answer include a number, date, price or quantity that doesn't appear in the source?

  3. Q3Invented namesYes / no quotes evidence

    Does the answer name a person, product, company, place or URL that the source never mentions?

  4. Q4Contradicts the sourceYes / no quotes evidence

    Does the answer say something the source contradicts?

How it decides

Rules run in code, top to bottom. The first one that matches gives the verdict.

  1. 1
    If Contradicts the source ≥ 0.7Hallucinated
  2. 2
    If States an unsupported fact ≥ 0.75Hallucinated
  3. 3
    If Invented numbers ≥ 0.75Hallucinated
  4. 4
    If Invented names ≥ 0.75Hallucinated
  5. 5
    If Contradicts the source ≥ 0.4Suspectreview
  6. 6
    If States an unsupported fact ≥ 0.4Suspectreview
  7. 7
    If Invented numbers ≥ 0.45Suspectreview
  8. 8
    If Invented names ≥ 0.45Suspectreview
  9. ·
    DefaultGrounded

Call it from your code

Every template runs as-is with an API key, no setup: POST /api/v1/judges/hallucination-detector/run.

curl -X POST https://tryevally.com/api/v1/judges/hallucination-detector/run \  -H "Authorization: Bearer $EVALLY_API_KEY" \  -H "Content-Type: application/json" \  -d '{  "input": {    "source": "The Pro plan includes 50,000 judgments a month and email support. Annual billing saves two months.",    "answer": "The Pro plan includes 50,000 judgments a month and phone support, and it costs $19 a month if you pay yearly."  }}'

Seen in the wild

Community builds on Jev that use this pattern. Not affiliated with Evally.

More in AI output quality