Evally
Start free
AI output quality

Support reply grader

Check an AI support reply against the customer's message and your policy before it's sent.

Catches the replies that cost you money or customers: promises your policy doesn't allow, answers to the wrong question, a cold tone, and conversations that should go to a person.

Use this judgeCustomize it

Try it live

Real calls to Jev. Pick an example or paste your own.

Examples
69
106
109
No verdict yet

Pick an example or paste your own input, then run it.

Under the hood

Every question and rule, in the open. Fork it to change any of them.

What it asks

Reads customer_message, reply and policy, then asks 5 questions in one call.

  1. Q1Addresses the requestYes / no

    Does the reply address what the customer actually asked or reported in customer_message?

  2. Q2Unauthorized promiseYes / no quotes evidence

    Does the reply promise a refund, credit, discount, deadline or exception that the policy doesn't allow? (If no policy is given, answer yes only for promises a business couldn't reasonably keep.)

  3. Q3ToneScore

    How is the tone of the reply?

    1. 0Rude or dismissive
    2. 1Cold or robotic
    3. 2Polite
    4. 3Warm and empathetic
  4. Q4Clear next stepYes / no

    Does the reply tell the customer clearly what happens next or what they should do?

  5. Q5Needs a humanYes / no

    Should this conversation go to a human agent? (Yes if the customer is angry, mentions legal action or a safety issue, or the reply can't resolve the problem.)

How it decides

Rules run in code, top to bottom. The first one that matches gives the verdict.

  1. 1
    If Unauthorized promise ≥ 0.6Don't send
  2. 2
    If Tone < 1 of 3Don't send
  3. 3
    If Needs a human ≥ 0.7Escalatereview
  4. 4
    If Addresses the request < 0.5Edit first
  5. 5
    If Clear next step < 0.4Edit first
  6. 6
    If Tone < 1.8 of 3Edit first
  7. ·
    DefaultReady to send

Call it from your code

Every template runs as-is with an API key, no setup: POST /api/v1/judges/support-reply-grader/run.

curl -X POST https://tryevally.com/api/v1/judges/support-reply-grader/run \  -H "Authorization: Bearer $EVALLY_API_KEY" \  -H "Content-Type: application/json" \  -d '{  "input": {    "customer_message": "I was charged twice for my subscription this month. Can you fix this?",    "reply": "So sorry about that! I'\''ve refunded both charges and added three free months to your account as an apology.",    "policy": "Duplicate charges are refunded in full. Agents may not offer free months or credits without manager approval."  }}'

More in AI output quality