Evally
Start free
Engineering

PR risk review

Is this pull request safe to merge, or does it need a careful human look?

Reads the PR description and diff and decides rather than writes: risky areas touched without tests, breaking changes, leftovers such as debug code and secrets, scope that doesn't match the description, and changes too big to review in one sitting.

Use this judgeCustomize it

Try it live

Real calls to Jev. Pick an example or paste your own.

Examples
34
309
No verdict yet

Pick an example or paste your own input, then run it.

Under the hood

Every question and rule, in the open. Fork it to change any of them.

What it asks

Reads description and diff, then asks 7 questions in one call.

  1. Q1Touches a risky areaYes / no

    Does the diff touch a risky area: authentication, permissions, payments, database migrations, encryption or infrastructure config?

  2. Q2Has testsYes / no

    Does the diff add or update tests that cover the change?

  3. Q3Needs testsYes / no

    Does the diff change behavior that tests should cover? (Typo fixes, copy changes, comments and formatting don't.)

  4. Q4Matches the descriptionYes / no

    Does the diff do what the description says, and nothing unrelated?

  5. Q5Breaking changeYes / no

    Could the diff break existing users or callers (a changed API, a removed field, a renamed config, or a changed default)?

  6. Q6LeftoversYes / no

    Does the diff leave debugging code, commented-out code, stray TODOs or secrets?

  7. Q7ScopePick one

    How big is the change in scope (not in lines)?

    TinySmallMediumLarge

How it decides

Rules run in code, top to bottom. The first one that matches gives the verdict.

  1. 1
    If Leftovers ≥ 0.75Changes requested
  2. 2
    If Touches a risky area ≥ 0.6 and Has tests < 0.5Changes requested
  3. 3
    If Breaking change ≥ 0.7Needs careful reviewreview
  4. 4
    If Touches a risky area ≥ 0.6Needs careful reviewreview
  5. 5
    If Matches the description < 0.5Needs careful reviewreview
  6. 6
    If Scope is LargeNeeds careful reviewreview
  7. 7
    If Needs tests ≥ 0.6 and Has tests < 0.4Needs careful reviewreview
  8. ·
    DefaultSafe to merge

Call it from your code

Every template runs as-is with an API key, no setup: POST /api/v1/judges/pr-risk-review/run.

curl -X POST https://tryevally.com/api/v1/judges/pr-risk-review/run \  -H "Authorization: Bearer $EVALLY_API_KEY" \  -H "Content-Type: application/json" \  -d '{  "input": {    "description": "Small cleanup of the login handler",    "diff": "--- a/src/auth/login.ts\n+++ b/src/auth/login.ts\n@@ -12,7 +12,6 @@ export async function login(req) {\n   const user = await db.users.find(req.body.email);\n-  if (!(await verify(user.hash, req.body.password))) throw new Unauthorized();\n+  console.log('\''login attempt'\'', req.body);\n   return issueSession(user);\n }"  }}'

Seen in the wild

Community builds on Jev that use this pattern. Not affiliated with Evally.

More in Engineering