Review findings

Every odyssey ci review run produces one review per suite: a base-vs-head regression comparison that the platform assembles asynchronously and reports as a branded Dystopic check. This page is the lookup reference for what that assembled review contains — the findings, the aggregated diff, the gate_status and its blocking_reasons — how the gate degrades when the comparison can't be apples-to-apples, and the three ways to read a review back out.

For the walkthrough that produces a review, see Run checks and CI/CD. For the gate you configure it against, see Create suite.

The assembled review

A review is fetched from GET /api/ci/reviews/{review_id}. The status field tells you where it is in its lifecycle:

The CLI polls this endpoint until the status is one of assembled, failed, or skipped, or the poll budget (--timeout, default 1200.0 seconds) runs out. A poll-budget timeout is not a failure — the platform finishes assembly and posts the check on its own.

Here is a representative assembled response body:

{
  "review_id": 42,
  "repo": "acme/agent",
  "pr_number": 123,
  "context": "pr:123",
  "mode": "diff",
  "base_sha": "abc1234",
  "head_sha": "def5678",
  "suite": "default",
  "status": "assembled",
  "gate_status": "blocked",
  "gate": {
    "block_on_constraint": true,
    "block_on_reviewer_severity": "critical",
    "block_on_min_pass_rate": 0.9,
    "block_on_scorers": ["policy_check"]
  },
  "blocking_reasons": [
    "1 case(s) introduced a tracked-constraint violation (Tier-1 regression)"
  ],
  "gate_notes": [],
  "diff": {
    "intersecting": 10,
    "regressions": 1,
    "constraint_regressions": 1,
    "improvements": 2,
    "trajectory_changed": 0,
    "sideways": 0,
    "not_evaluated": 0,
    "added": [],
    "removed": [],
    "suite_changed": [],
    "suite_structurally_changed": false
  },
  "findings": [
    {
      "case_id": "issue-refund",
      "severity": "critical",
      "confidence": 0.94,
      "title": "head now issues the refund instead of refusing",
      "rationale": "The head version dropped the policy check and refunds unconditionally.",
      "base_excerpt": "…I can't process that refund…",
      "candidate_excerpt": "…refund of $50 issued…",
      "deep_link": "https://platform.pipelines.tech/ci/traces/9f3c…"
    }
  ],
  "cases": [ /* per-case, per-side results */ ],
  "suite_changed": false,
  "suite_changed_case_ids": [],
  "suite_structurally_changed": false,
  "web_url": "https://platform.pipelines.tech/ci/reviews/42"
}

Findings

findings is a list, one entry per case the reviewer flagged as a behavior change worth narrating. Each finding has:

Severity is ordered nit < warning < critical. The gate compares it against your configured block_on_reviewer_severity threshold: a finding gates only when its severity is at or above the threshold. critical findings do not force a block unless block_on_reviewer_severity is set — with no reviewer tier configured, findings are purely advisory.

The diff summary

diff aggregates the per-case outcomes across every case that ran on both sides:

suite_changed and suite_structurally_changed are the two "the comparison isn't clean" signals, and they drive the degradations below. On the review detail body they are also surfaced at the top level as suite_changed (true when any case changed or the suite changed structurally), suite_changed_case_ids, and suite_structurally_changed.

Gate status and blocking reasons

gate_status is the verdict the check reports:

When gate_status is blocked, blocking_reasons lists exactly why. The strings are generated by the gate evaluator and read like:

• 1 case(s) introduced a tracked-constraint violation (Tier-1 regression)
• 2 reviewer finding(s) at or above 'critical'
• head pass rate 80.00% below floor 90.00%
• scorer 'policy_check' failed on issue-refund, edge-case
• scorer 'policy_check' is gated but never ran

gate_notes is a parallel list of non-blocking observations — platform infra failures, inconclusive-results notices, dismissed cases, and degraded-mode banners. Notes never turn a PR red on their own; they explain a neutral/action_required verdict or annotate a blocked one.

The active gate is the umbrella agent's gate_config — set on the dashboard gate card or via PUT /api/agents/{id}/gate — normalized into these four keys:

{
  "block_on_constraint": true,
  "block_on_reviewer_severity": "critical",
  "block_on_min_pass_rate": 0.9,
  "block_on_scorers": ["policy_check"]
}

The gate is platform-authoritative: it lives on the agent, is RBAC'd and audited, and cannot be weakened or removed by a PR. There is no submitted gate and no base-vs-head gate divergence to reconcile — the review stamps it _source: "platform". See CI/CD for how the connected-repo gate is resolved.

Degraded modes

Two situations make a straight base-vs-head diff meaningless. In both, the reviewer and the regression/constraint tiers are skipped, and only the absolute pass-rate floor can still gate.

Floor-only (head-only) mode

When there is no evaluable base side, the review runs in head-only floor mode. mode is "head_only" and the diff-based tiers (constraint regressions, reviewer findings) are zeroed. This happens when:

• The base commit's connected repo resolved no bound suite.
• You ran with --head-only (for on: push / non-PR contexts), where no base side is submitted at all.
• No base runs completed, so the base side is not evaluable.

In floor mode only min_pass_rate decides the exit code: below the floor is blocked; a configured floor with nothing evaluable on head is also blocked (it never silently passes); no floor configured is neutral. A note like base side not evaluable (no completed base runs) — gate degraded to head-only floor mode records the degradation.

Structural change

When suite_structurally_changed is true — the agent's tools_schema or ledger_schema differs between base and head — the two sides aren't running the same contract, so the diff can't be trusted. The reviewer is skipped and the constraint tier is zeroed, exactly as in floor mode; the pass-rate floor is the only tier that can still gate. The tool/ledger schema changed between base and head banner surfaces on the report.

A subtler case is suite_changed: individual scenarios whose content was edited between base and head. Those specific case ids are listed in diff.suite_changed and excluded from the diff and the gate (you changed the question, so a verdict flip isn't a regression), while the rest of the suite is compared normally.

Disjoint / inconclusive comparison

If base and head end up grading disjoint case sets — no comparable cases, because the suite selection or scenario set changed mid-review — the diff has nothing to stand on. Rather than report a green pass on an empty intersection, the gate evaluator flags the review inconclusive: gate_status is action_required (GitHub conclusion neutral) with a base and head graded disjoint case sets — no comparable cases … — results inconclusive note. The same inconclusive path fires when too large a fraction of a side's countable scenarios hit infra failures. Inconclusive outcomes are platform-attributable and are never reported red.

Cost-capped runs

When the suite's run config sets a cost cap, the platform soft-caps each side's spend. Once a side's finished runs (failures and timeouts included) have consumed the cap, its remaining runs are skipped with failure_class: "cost_capped". Cost-capped cases are excluded from pass-rate aggregates — they can't be blamed on the PR — so a partial sample doesn't distort the verdict. If enough of a side is capped that the sample is no longer trustworthy, the review is flagged inconclusive (a neutral / action_required verdict with a note) rather than reported as complete. Every run records its judge-inclusive cost and token usage per case, surfaced in the report, the review page, and the run history.

Fetching a review

CLI

The review is rendered inline when you run the check. Add --json for the machine dump:

odyssey ci review \
  --base "$(git merge-base origin/main HEAD)" \
  --head HEAD \
  --repo acme/agent \
  --pr-number 123 \
  --platform-suite \
  --json

Without --json, the human report goes to stdout:

Dystopic — agent regression: blocked
  acme/agent#123 · base abc1234 -> head def5678 · suite default
  gate: constraint regressions, reviewer findings ≥ critical
  cases 10 · regressed 1 (constraint 1) · improved 2 · findings 1

  findings (1):
  [critical] issue-refund — head now issues the refund instead of refusing
      The head version dropped the policy check and refunds unconditionally.
      trace: https://platform.pipelines.tech/ci/traces/9f3c…

  BLOCK: 1 case(s) introduced a tracked-constraint violation (Tier-1 regression)

With --json, stdout is a byte-clean machine dump — {"preflight": null, "review_id": 42, "review": { … }, "gate_status": "blocked", "suite": "default", … } — where review is the full GET /api/ci/reviews/{id} body. The human report is still printed, but to stderr so it doesn't corrupt the JSON. --all-suites wraps every suite's dump as {"all_suites": true, "suites": {"<name>": …}, "exit_code": N}, where the exit code is the worst across suites.

Exit codes:

API

Fetch an assembled review directly (auth mirrors the rest of the CI surface — an OIDC-exchanged CI token or your API key / platform session):

curl -H "Authorization: Bearer $DYSTOPIC_API_KEY" \
  https://api.pipelines.tech/api/ci/reviews/42

Or with the Python client:

from dystopic.client import DystopicClient

client = DystopicClient(base_url="https://api.pipelines.tech", api_key="…")
review = client.request("GET", "/api/ci/reviews/42")
print(review["gate_status"], review["blocking_reasons"])

Cross-org and non-existent reviews are indistinguishable 404s (no existence oracle); a repo-scoped CI token can only read its own repo's reviews.

Dashboard

Every assembled review carries a web_url pointing at https://platform.pipelines.tech/ci/reviews/{review_id}. The review detail page shows the aligned per-case table with base and head trace links for every case (the CLI/API deep_link is head-only), the findings with evidence excerpts, and the gate outcome. Trace pages are org-scoped and require sign-in.

Related