Evally
Start free
Agents

Is the task done?

Don't trust an agent's "done". Check the transcript for proof, open errors and skipped work.

One job: decide whether an agent really finished. It checks that every part of the task was addressed, that there's evidence it works (tests, output) rather than just a claim, and that no error or TODO was left behind, and quotes what it found.

Use this judgeCustomize it

Try it live

Real calls to Jev. Pick an example or paste your own.

Examples
64
209
No verdict yet

Pick an example or paste your own input, then run it.

Under the hood

Every question and rule, in the open. Fork it to change any of them.

What it asks

Reads task and transcript, then asks 5 questions in one call.

  1. Q1Claims it's doneYes / no

    Does the transcript claim the task is finished?

  2. Q2Shows proofYes / no quotes evidence

    Does the transcript show concrete evidence that the task works (tests passing, command output, a verified result), not just a claim?

  3. Q3Leaves errors openYes / no quotes evidence

    Does the transcript end with an error, a failing test or a warning that was never resolved?

  4. Q4Covers every partYes / no

    Does the transcript address every part of the task? (If the task lists several things, each one must be handled.)

  5. Q5Skipped or stubbed workYes / no quotes evidence

    Does the transcript mention skipping, stubbing, mocking or leaving a TODO for something the task asked for?

How it decides

Rules run in code, top to bottom. The first one that matches gives the verdict.

  1. 1
    If Leaves errors open ≥ 0.6Not done
  2. 2
    If Skipped or stubbed work ≥ 0.6Not done
  3. 3
    If Covers every part < 0.4Not done
  4. 4
    If Claims it's done < 0.4Not done
  5. 5
    If Shows proof < 0.5Done, but unprovenreview
  6. ·
    DefaultDone

Call it from your code

Every template runs as-is with an API key, no setup: POST /api/v1/judges/task-done-checker/run.

curl -X POST https://tryevally.com/api/v1/judges/task-done-checker/run \  -H "Authorization: Bearer $EVALLY_API_KEY" \  -H "Content-Type: application/json" \  -d '{  "input": {    "task": "Add rate limiting to the /login endpoint and write tests for it.",    "transcript": "I added a rate limiter to the /login route using a sliding window of 5 attempts per minute. The tests should pass. I left a TODO to add the tests later since the test setup was complicated. Everything is done!"  }}'

Seen in the wild

Community builds on Jev that use this pattern. Not affiliated with Evally.

More in Agents