Evally
Start free
AI output quality

Summary checker

Faithful, complete and free of added opinions: the three ways summaries go wrong.

Checks an AI summary against its document: nothing invented, the key point kept, no critical caveat dropped, and no opinions the author never gave.

Use this judgeCustomize it

Try it live

Real calls to Jev. Pick an example or paste your own.

Examples
187
74
No verdict yet

Pick an example or paste your own input, then run it.

Under the hood

Every question and rule, in the open. Fork it to change any of them.

What it asks

Reads document and summary, then asks 5 questions in one call.

  1. Q1FaithfulYes / no

    Is everything the summary says supported by the document?

  2. Q2Keeps the key pointYes / no

    Does the summary include the single most important point of the document?

  3. Q3Drops something criticalYes / no

    Does the summary leave out something a reader needs to avoid a wrong conclusion (a caveat, a condition, or a number that changes the meaning)?

  4. Q4Adds opinionsYes / no

    Does the summary add opinions, judgments or recommendations that the document doesn't make?

  5. Q5ConcisionScore

    How concise is the summary?

    1. 0Padded: repeats itself or wanders
    2. 1Could be tighter
    3. 2Tight

How it decides

Rules run in code, top to bottom. The first one that matches gives the verdict.

  1. 1
    If Faithful < 0.5Unfaithful
  2. 2
    If Drops something critical ≥ 0.6Needs editsreview
  3. 3
    If Adds opinions ≥ 0.6Needs editsreview
  4. 4
    If Keeps the key point < 0.5Needs editsreview
  5. ·
    DefaultGood summary

Call it from your code

Every template runs as-is with an API key, no setup: POST /api/v1/judges/summary-checker/run.

curl -X POST https://tryevally.com/api/v1/judges/summary-checker/run \  -H "Authorization: Bearer $EVALLY_API_KEY" \  -H "Content-Type: application/json" \  -d '{  "input": {    "document": "The new drug reduced symptoms by 40% in adults aged 18-45. It showed no benefit in patients over 65, and 12% of participants reported headaches. Further trials are needed before approval.",    "summary": "A new drug cuts symptoms by 40% and is ready to help patients of all ages."  }}'

More in AI output quality