Evally
Start free
AI output quality

RAG answer grader

Is the answer grounded in the retrieved source, and does it answer the question?

Grades a retrieval-augmented answer the way a careful reviewer would: every claim checked against the source, contradictions caught, refusals spotted, and coverage scored. Run it on every production answer or on a saved batch of traces.

Use this judgeCustomize it

Try it live

Real calls to Jev. Pick an example or paste your own.

Examples
27
119
125
No verdict yet

Pick an example or paste your own input, then run it.

Under the hood

Every question and rule, in the open. Fork it to change any of them.

What it asks

Reads question, source and answer, then asks 5 questions in one call.

  1. Q1Grounded in the sourceYes / no

    Is every factual claim in the answer supported by the source? (Rewording is fine. A claim the source doesn't state counts as unsupported, even if it happens to be true.)

  2. Q2Answers the questionYes / no

    Does the answer directly answer the question? (A partial answer counts only if it addresses the main thing asked.)

  3. Q3Contradicts the sourceYes / no

    Does the answer contradict anything the source says?

  4. Q4Refusal or non-answerYes / no

    Is the answer a refusal, an apology, or a statement that it can't help?

  5. Q5CoverageScore

    How completely does the answer cover what the source says about the question?

    1. 0Misses the point
    2. 1Covers a little
    3. 2Covers the main point
    4. 3Covers everything relevant

How it decides

Rules run in code, top to bottom. The first one that matches gives the verdict.

  1. 1
    If Contradicts the source ≥ 0.6Fail
  2. 2
    If Grounded in the source < 0.4Fail
  3. 3
    If Refusal or non-answer ≥ 0.7Needs reviewreview
  4. 4
    If Answers the question < 0.5Needs reviewreview
  5. 5
    If Grounded in the source < 0.7Needs reviewreview
  6. 6
    If Coverage < 1.5 of 3Needs reviewreview
  7. ·
    DefaultPass

Call it from your code

Every template runs as-is with an API key, no setup: POST /api/v1/judges/rag-answer-grader/run.

curl -X POST https://tryevally.com/api/v1/judges/rag-answer-grader/run \  -H "Authorization: Bearer $EVALLY_API_KEY" \  -H "Content-Type: application/json" \  -d '{  "input": {    "question": "When will my refund arrive?",    "source": "Refunds are processed within 24 hours of approval. The money then arrives in 5-7 business days, depending on your bank.",    "answer": "Your refund is processed within a day of approval and should reach your account in 5-7 business days, depending on your bank."  }}'

Seen in the wild

Community builds on Jev that use this pattern. Not affiliated with Evally.

More in AI output quality