How to Configure PR Review Feedback

Review feedback routing closes the outer loop of agent-driven development. When a human reviewer leaves inline comments on a Sortie-created PR and submits the review with the “Request changes” verdict in the forge’s review UI, Sortie detects those comments, assembles structured context (file paths, line ranges, reviewer names, comment bodies), and dispatches a continuation turn so the agent can address the feedback and push fixes.

Without it, review comments sit on the PR until someone manually re-assigns the issue. With it, turnaround drops from hours to minutes.

Prerequisites

• Sortie running with the GitHub, Gitea, or GitLab tracker adapter, or a registered SCM provider of one of those kinds
• An agent adapter that creates PRs and writes pr_number, owner, and repo to .sortie/scm.json (see Setup workspace hooks)
• An access token with the scope the SCM adapter needs to read reviews: GitHub needs repo; see the Gitea adapter reference and the GitLab adapter reference for theirs

Activate review feedback

Review feedback is off by default. Add a reactions.review_comments block to your WORKFLOW.md front matter:

reactions:
  review_comments:
    provider: github

There is no enabled flag. Presence of the reactions.review_comments block with a provider activates the feature; absence disables it.

Once activated, review polling kicks in after each normal worker exit where the workspace’s .sortie/scm.json contains pr_number, owner, and repo. Sortie only reacts to reviews submitted with the “Request changes” verdict in the forge’s review UI, whatever each one calls that state internally. Reviews submitted as “Comment” or “Approve” are ignored. All three scm.json fields are required. If any is missing or zero-valued, Sortie skips review polling for that workspace silently. Existing workspaces that predate the feature are unaffected.

Your after_run hook or agent workflow writes these fields. Here’s what .sortie/scm.json looks like:

{
  "branch": "feat/PROJ-123",
  "sha": "abc1234",
  "pushed_at": "2026-04-10T12:00:00Z",
  "pr_number": 42,
  "owner": "myorg",
  "repo": "myproject"
}

The branch and sha fields drive CI feedback (if configured). The pr_number, owner, and repo fields drive review feedback. Both features read from the same file.

The optional pushed_at field is an ISO-8601/RFC3339 UTC timestamp of the last push. Sortie reads it on startup when it reconstructs handoff-stage pending reactions, so review polling for an open PR survives a restart instead of waiting for the next push or active-state transition. If pushed_at is missing, recovery falls back to the agent run’s completed_at time.

Long-lived PRs age out after 30 days of inactivity. Write pushed_at from the same hook that pushes the branch so the age-out reflects the latest push, not the agent run. See Recover work across restarts for the recovery model and bounds.

Configure retry limits and escalation

reactions:
  review_comments:
    provider: github
    escalation: label
    escalation_label: needs-human

The retry budget for this kind is max_continuation_turns, configured below, not max_retries: review_comments accepts max_retries for schema consistency with the other reaction kinds but does not consume it, so setting it here has no effect. max_continuation_turns counts continuation turns triggered specifically by review comments, independent of the agent’s max_sessions budget and CI feedback’s retry counter.

If the agent addresses all comments within this budget, the loop ends. If a new comment arrives after the budget is spent, Sortie escalates and releases its claim unless a worker is running for the issue. Comments an earlier run was given do not escalate on their own.

The escalation then stands: the reaction keeps watching the pull request and dispatches nothing, even across a restart. It ends when no comment is left that no run was given, when the pull request merges or closes, or when you move the issue out of tracker.handoff_state; the next run then starts with a fresh turn budget. A comment that arrives while the escalation stands joins it and applies no second label. See standing escalations.

With label, Sortie adds the configured label to the issue. With none, it adds no label. Every escalation emits the escalation.review_comments event, and a tracker_comment entry that lists it posts a comment noting how many turns were attempted and that remaining comments need human attention; see how to route notifications. The deprecated comment value posts that comment without an entry. Every strategy cancels any pending retry and releases the claim, unless a worker is running for the issue; that worker’s exit then decides the claim.

Create the label in advance if using label escalation:

gh label create needs-human --repo myorg/myrepo --color "D93F0B"

Configure polling and debounce

reactions:
  review_comments:
    provider: github
    poll_interval_ms: 120000
    debounce_ms: 60000
    max_continuation_turns: 3

Debounce prevents premature dispatch while a reviewer is still commenting. A reviewer posts 2 inline comments, Sortie detects them on the next poll, and starts a 60-second timer from the newest comment’s timestamp. If the reviewer posts 2 more within that window, the timer resets. Once 60 seconds pass with no new comments, Sortie dispatches all comments in one batch.

poll_interval_ms throttles how often Sortie hits the GitHub Reviews API per tracked PR. The 2-minute default balances responsiveness and API rate budget. If you’re tracking many PRs, consider raising it. The minimum is 30 seconds.

max_continuation_turns prevents infinite reviewer-agent ping-pong. Once the cap is spent, the next new comment makes Sortie escalate and a human takes over. If the last turn resolved the comments, nothing escalates. See retry budgets for the exact rule.

How the review loop works

1. The agent completes coding and pushes a PR. The after_run hook writes pr_number, owner, and repo to .sortie/scm.json.
2. On normal worker exit, the orchestrator reads this metadata and creates a pending review reaction. Polling begins on the next reconcile tick.
3. A reviewer requests changes and leaves inline comments. Each forge spells that state differently, and each adapter selects against its own spelling. Comment-only and approving reviews do not trigger the loop.
4. Sortie detects the comments on its next poll after the debounce window expires.
5. Outdated comments (on lines the agent already changed) are filtered out, as are comments whose author the forge marks as a bot account. Gitea carries no bot marker, so nothing is excluded there; see the Gitea adapter reference.
6. Sortie builds a fingerprint from the remaining comment IDs. If this fingerprint was already dispatched, it skips (deduplication).
7. Sortie checks whether the set holds a comment that no earlier run of the issue was given. If every comment was given before, for example because a fix made some of them outdated, no turn starts and no budget is spent.
8. Otherwise Sortie dispatches a continuation turn with all remaining review comments as structured prompt context.
9. The agent addresses the comments, commits, and pushes fixes.
10. If the reviewer approves, polling stops on the next state change. If the reviewer requests more changes, the cycle repeats from step 3, up to max_continuation_turns.
11. If the turn cap is spent and a new comment arrives, Sortie escalates and releases the claim unless a worker is running for the issue.

What the agent sees

The agent receives review comments through the review_comments template variable, set on every turn of a review-fix run. Without a block that uses it, Sortie sends a built-in prompt that lists every comment with its reviewer and location and asks the agent to address each one and push to the pull request’s branch. To write your own instructions, add a conditional block to your prompt template. A block replaces the built-in prompt:

{{ if .review_comments }}
## Review Comments to Address

The following review comments were left on your PR. Address each one:

{{ range .review_comments }}
### {{ .reviewer }} on {{ .file }}{{ if .start_line }} (line {{ .start_line }}{{ if .end_line }}-{{ .end_line }}{{ end }}){{ end }}

{{ .body }}

{{ end }}
{{ end }}

The variable is nil on non-review turns, so the block renders only when review comments are present. PR-level comments (not attached to a specific file) have an empty file and zero line numbers.

Each comment exposes:

For template syntax details, see Write a prompt template.

Choose between an added block and an exclusive branch

A review-fix run counts every comment it carries as given, whichever prompt it receives, and records that when the run exits normally. The shape of your template decides what else the run receives. An added block, like the one above, sends the work prompt and the comments, on every turn of the run. A branch in an exclusive {{ if }} / {{ else }} chain sends the comments without the work prompt. The rule is in built-in reaction prompts.

Give a new run its pending review comments

A new run of an issue that already has a pull request, such as after you reopen the issue, receives the comments no earlier run was given on its first turn, through the same review_comments variable. No later poll then starts a turn that repeats them. To set it up:

• Have the hook or agent write branch to .sortie/scm.json next to pr_number, owner, and repo. Review polling works without branch, but a new run gets its comments only when it is present.
• Leave the triage block off the reaction.
• Write the comments as an added section, like the {{ if .review_comments }} block above, with the rest of the prompt unchanged.

A template that swaps its main text when review_comments is set keeps its normal prompt on a new run, and the comments reach the agent through a later review turn instead.

If fetching the comments fails, Sortie starts the run without them and logs fresh run not given review comments at warn level. The same applies to bot_review with bot_review_comments. For the exact rules, see comments already given.

Interaction with CI feedback

Review feedback and CI feedback are independent reaction types. They have separate retry budgets, separate fingerprints, separate poll intervals, and separate escalation policies. Both can be active on the same issue simultaneously, and they do not interfere with each other.

CI feedback detects pipeline failures on pushed branches. Review feedback detects human reviewer comments on PRs. If both fire on the same issue (CI fails and a reviewer requests changes), each dispatches its own continuation turn with its own context. The agent receives {{ .ci_failure }} on CI-triggered turns and {{ .review_comments }} on review-triggered turns.

For CI feedback configuration, see Configure CI feedback.

Interaction with self-review

Self-review and review feedback operate at different lifecycle phases. Self-review runs inside the worker before exit, catching local issues (test failures, lint errors) with verification commands you configure. Review feedback runs after the worker exits, during the orchestrator’s reconcile loop, catching human feedback left on the PR.

They do not conflict. Self-review catches problems before the PR is opened; review feedback handles comments after. Both can be active at the same time.

For self-review configuration, see Configure self-review.

Complete example

A full WORKFLOW.md with review feedback, CI feedback, GitHub Issues, and a prompt template that handles both continuation types:

---
tracker:
  kind: github
  api_key: $SORTIE_GITHUB_TOKEN
  project: myorg/myrepo
  active_states: [backlog, in-progress]
  terminal_states: [done, wontfix]
  handoff_state: review
  in_progress_state: in-progress

notifications:
  - kind: tracker_comment
    events: [session.started, session.completed, session.stopped, session.failed, budget.held]

agent:
  kind: claude-code
  command: claude
  max_turns: 5
  max_sessions: 3
  max_concurrent_agents: 2
  stall_timeout_ms: 300000

reactions:
  ci_failure:
    provider: github
    max_retries: 2
    max_log_lines: 50
    escalation: label
    escalation_label: needs-human
  review_comments:
    provider: github
    poll_interval_ms: 120000
    debounce_ms: 60000
    max_continuation_turns: 3
    escalation: label
    escalation_label: needs-human

hooks:
  after_create: |
    git clone --depth 1 "https://${SORTIE_GITHUB_TOKEN}@github.com/myorg/myrepo.git" .
  before_run: |
    git fetch origin main
    git checkout -B "sortie/${SORTIE_ISSUE_IDENTIFIER}" origin/main
  after_run: |
    git add -A
    git diff --cached --quiet || {
      git commit -m "sortie(${SORTIE_ISSUE_IDENTIFIER}): automated changes"
      git push origin "sortie/${SORTIE_ISSUE_IDENTIFIER}" --force-with-lease

      SHA=$(git rev-parse HEAD)
      PR_URL=$(gh pr create \
        --repo myorg/myrepo \
        --head "sortie/${SORTIE_ISSUE_IDENTIFIER}" \
        --base main \
        --title "sortie(${SORTIE_ISSUE_IDENTIFIER}): automated changes" \
        --body "Automated PR for ${SORTIE_ISSUE_IDENTIFIER}" \
        2>/dev/null || gh pr view \
        --repo myorg/myrepo \
        "sortie/${SORTIE_ISSUE_IDENTIFIER}" \
        --json url -q .url 2>/dev/null)
      PR_NUMBER=$(echo "$PR_URL" | grep -oP '\d+$')

      mkdir -p .sortie
      cat > .sortie/scm.json <<EOF
    {
      "branch": "sortie/${SORTIE_ISSUE_IDENTIFIER}",
      "sha": "${SHA}",
      "pushed_at": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
      "pr_number": ${PR_NUMBER:-0},
      "owner": "myorg",
      "repo": "myrepo"
    }
    EOF
    }
  timeout_ms: 120000

db_path: .sortie.db
---

You are a senior engineer working on {{ .issue.identifier }}.

## Task

**{{ .issue.identifier }}**: {{ .issue.title }}

{{ if .issue.description }}
{{ .issue.description }}
{{ end }}

{{ if .ci_failure }}
## CI Failure

CI is failing on branch {{ .ci_failure.ref }}.
{{ .ci_failure.failing_count }} check(s) failed.

{{ if .ci_failure.log_excerpt }}
Failure log excerpt:
```
{{ .ci_failure.log_excerpt }}
```
{{ end }}

{{ range .ci_failure.check_runs }}{{ if eq .conclusion "failure" }}
- {{ .name }}: FAILED{{ if .details_url }} ({{ .details_url }}){{ end }}
{{ end }}{{ end }}

Diagnose the CI failure and fix the code. Do not modify CI workflow files.
{{ end }}

{{ if .review_comments }}
## Review Comments to Address

A human reviewer left feedback on your PR. Address each comment:

{{ range .review_comments }}
### {{ .reviewer }} on {{ .file }}{{ if .start_line }} (line {{ .start_line }}{{ if .end_line }}-{{ .end_line }}{{ end }}){{ end }}

{{ .body }}

{{ end }}

After addressing all comments, commit and push your changes.
{{ end }}

{{ if .run.is_continuation }}
Resuming turn {{ .run.turn_number }}/{{ .run.max_turns }}.
{{ end }}

The after_run hook creates a PR (or finds the existing one), extracts the PR number, and writes all required fields to .sortie/scm.json. This populates the data that both CI feedback and review feedback need.

Verify review feedback

Four approaches to confirm the feature is working.

Logs

Search for key messages that trace the review feedback lifecycle:

# Review comments detected and dispatch scheduled
grep "review comments detected" sortie.log

# Debounce active: comments within the window
grep "review comments within debounce window" sortie.log

# Fingerprint already dispatched: deduplication working
grep "review comments already dispatched for this fingerprint" sortie.log

# Turn cap spent and a new comment arrived, escalation triggered
grep "review fix continuation turns exhausted" sortie.log

# Changed set holds no comment an earlier run was given, nothing dispatched or escalated (debug level)
grep "comment set already handed off, not dispatching" sortie.log

# Set covered only by the turn that is still running, judged again after it exits (debug level)
grep "comment set held by a running turn, not dispatching" sortie.log

# New run started without its pending comments
grep "fresh run not given review comments" sortie.log

Dashboard and status API

When the HTTP server is running, each issue with an active review watch has a row with kind review in the pending_reactions list of GET /api/v1/state. The row shows why the watch waits (wait, which reads debounce while the debounce window is open), when it checks next (next_poll_at), how many continuation turns it has used (used_attempts of max_attempts), and how many comments the last check found (last_check). See the HTTP API reference for the fields.

Prometheus metrics

Two review-specific metrics are available when the HTTP server is enabled:

A healthy setup shows sortie_review_checks_total{result="dispatched"} incrementing when review comments arrive, with error counts staying flat. Persistent errors on the poll metric indicate a token or permissions problem. For the full metrics catalog, see Prometheus metrics reference.

SQLite fingerprints

The reaction_fingerprints table tracks which comment sets have been dispatched. This is what prevents duplicate dispatches across reconcile ticks and process restarts:

sqlite3 sortie.db "SELECT * FROM reaction_fingerprints WHERE kind='review'"

Each row shows the issue ID, the review kind, the SHA-256 fingerprint of comment IDs, and whether it has been dispatched.

The reaction_handoffs table records which comments each issue’s runs were given. A changed comment set starts a turn only when it holds an ID missing from this table:

sqlite3 sortie.db "SELECT comment_id FROM reaction_handoffs WHERE kind='review' AND issue_id='<issue id>'"

Rows are never deleted, and the table starts empty after an upgrade, so a comment given before the upgrade can be given once more.

Configuration reference

All reactions.review_comments fields in one place:

max_retries is also accepted, for schema consistency with the other reaction kinds, but review_comments does not consume it; the retry budget above is max_continuation_turns.

For the full WORKFLOW.md configuration reference including all sections, see workflow config reference.

Related guides

• Configure CI feedback: CI pipeline failure detection and agent retry loop
• Configure self-review: pre-PR agent review with verification commands
• Configure retry behavior: max_sessions, backoff, stall detection
• Setup workspace hooks: hook scripts, scm.json population, environment variables
• Write a prompt template: template syntax, continuation context keys
• Connect to GitHub: GitHub adapter setup, token scopes
• Prometheus metrics reference: review and escalation metrics
• Workflow config reference: reactions.review_comments field reference