Create a suite: scenarios, gate & CI binding

Suites, scenarios, and the gate are platform objects attached to your agent. There are no committed config files: you author them on the web dashboard (canonical), with the CLI/SDK (typed wrappers, SDK ≥ 0.5.0), or through the REST API they all share — and CI resolves the bound suite server-side.

This page is the exhaustive reference for those REST endpoints and their typed wrappers. Every route below is under https://api.pipelines.tech/api and scoped to one agent by its numeric {id}. For the golden-path walkthrough that ties this into a first green PR, see 4 · Create a suite. For how the resolved gate and bound suite drive the merge check, see the CI/CD reference.

Scenarios

A scenario is the platform object one check attempts and compares base-vs-head. It replaces the old per-row seed: one scenario describes one task, its world, and how it is judged. A suite binds an ordered list of scenarios.

POST /api/agents/{id}/scenarios

Create a scenario attached to the agent. At least one content axis is required — a body with none of user_instruction, behavior_instructions, or conversation is rejected.

curl -X POST https://api.pipelines.tech/api/agents/42/scenarios \
  -H "Authorization: Bearer $DYSTOPIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "refund-simple",
    "scenario_group": "refunds",
    "user_instruction": "A customer asks: refund order #1234 from last week.",
    "expected_outcome": "completion",
    "initial_state": { "orders": [ { "id": 1234, "status": "delivered" } ] },
    "failure_rules": ["issued a refund above the order total"]
  }'

Typed: client.create_scenario(42, name=..., user_instruction=..., ...) (keyword-per-field; None fields are dropped), or dystopic scenarios create 42 --name … --user-instruction …. The CLI exposes the scalar fields (--name, --user-instruction, --expected-outcome, --behavior-instructions, --failure-rules (repeatable), --group, --world-id); the JSON-shaped fields (initial_state, scorers, tool_mode_overrides, conversation) are SDK/dashboard-only.

GET /api/agents/{id}/scenarios

List the agent's scenario pool (newest first). Query params: group (filter by scenario_group), limit/offset (window the pool). Typed: client.list_scenarios(42, group=..., limit=..., offset=...), client.iter_scenarios(42) (auto-pages), or dystopic scenarios list 42 [--group G] [--limit N] [--offset N].

Bulk-import from CSV

There is no server-side CSV endpoint — bulk import is a client-side loop that parses the CSV and POSTs one scenario per row. Three equivalent clients do it: the dashboard's CSV importer, client.create_scenarios_from_csv(42, "seeds.csv", suite_id=None, append=False), and dystopic scenarios import 42 seeds.csv [--suite-id N] [--append]. All three speak the same column dialect (name, group, user_instruction, behavior_instructions, failure_rules (split on ;/newline), expected_outcome, scorers (JSON array), initial_state (JSON object), and the multi-turn axes such as max_turns / turn_mode / persona, folded into conversation). Rows with no content field are skipped; a non-numeric max_turns cell is dropped rather than aborting the import.

With suite_id, the imported scenarios are bound to that suite afterwards — by default an atomic full-replace of the suite's bindings in import order (anything already bound is unbound). append=True / --append instead keeps the suite's existing bindings and appends the imported scenarios after them (already-bound ids are not duplicated).

GET /api/agents/{id}/scenarios/{sid}

Read one scenario as it currently stands. Typed: client.get_scenario(42, 101) / dystopic scenarios show 42 101.

PATCH /api/agents/{id}/scenarios/{sid}

Update any of the fields above. Send only the keys you want to change. The edit takes effect on the next run of any binding suite — it does not alter past checks. Typed: client.update_scenario(42, 101, **fields) / dystopic scenarios update 42 101 --user-instruction ….

DELETE /api/agents/{id}/scenarios/{sid}

Delete the scenario. It is removed from any suite that binds it; checks that already ran keep their frozen copy. Typed: client.delete_scenario(42, 101) / dystopic scenarios delete 42 101.

Multi-turn scenarios — the conversation object

A scenario is single-shot by default: one instruction, one agent response, judged. Setting keys inside conversation turns it into a multi-turn session — the agent and a simulated user alternate turns under the session orchestrator. The recognized keys:

The same axes are importable per-row via the CSV columns (turn_mode, max_turns, persona, scripted_user_turns, …) — the importer folds them into the nested conversation object.

Suites

A suite is a named, ordered collection of scenarios plus the run configuration used to execute them. Binding a suite to CI (below) is what makes it the set a Dystopic PR check runs.

POST /api/agents/{id}/suites

Create a suite on the agent.

curl -X POST https://api.pipelines.tech/api/agents/42/suites \
  -H "Authorization: Bearer $DYSTOPIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "name": "default", "description": "Refund + policy regression set" }'

Typed: client.create_suite(42, name="default", description=..., world_id=..., config=...) / dystopic suites create 42 --name default [--description D] [--world-id W] (a duplicate name is a 409).

GET /api/agents/{id}/suites

List the agent's suites (bare, un-paginated list). Typed: client.list_suites(42) / dystopic suites list 42.

GET /api/agents/{id}/suites/{sid}

Read the suite's metadata and config. Typed: client.get_suite(42, 7) / dystopic suites show 42 7.

PATCH /api/agents/{id}/suites/{sid}

Update name, description, world_id, or config. Send only the keys you change. Typed: client.update_suite(42, 7, **fields) / dystopic suites update 42 7 --name … --description ….

DELETE /api/agents/{id}/suites/{sid}

Delete the suite. If it was bound to CI, clear the CI binding first or the platform will fail closed on the next check. Typed: client.delete_suite(42, 7) / dystopic suites delete 42 7.

Suite ↔ scenario binding

The membership of a suite is managed separately from the scenarios themselves, so you can reorder or swap scenarios without recreating anything.

GET /api/agents/{id}/suites/{sid}/scenarios

Return the suite's bound scenarios, in order. Typed: client.list_suite_scenarios(42, 7) / dystopic suites scenarios 42 7.

PUT /api/agents/{id}/suites/{sid}/scenarios

Ordered replace-all. The body's list is the suite's new membership, in exactly that order — scenarios not listed are unbound, and the order you send is the order checks run. An empty list unbinds everything; every id must belong to this agent.

curl -X PUT https://api.pipelines.tech/api/agents/42/suites/7/scenarios \
  -H "Authorization: Bearer $DYSTOPIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "scenario_ids": [101, 102, 103] }'

Typed: client.set_suite_scenarios(42, 7, [101, 102, 103]) / dystopic suites bind 42 7 101 102 103 (positional order = run order).

DELETE /api/agents/{id}/suites/{sid}/scenarios/{scenario_id}

Unbind a single scenario from the suite. The scenario itself is untouched — this only removes it from this suite's membership. Typed: client.unbind_suite_scenario(42, 7, 101) / dystopic suites unbind 42 7 101.

The gate

The gate is the rule that turns a base-vs-head diff into pass or block. It lives on the agent (the umbrella agent, for a connected repo), is RBAC'd and audited, and is set with a single AgentGateConfig.

PUT /api/agents/{id}/gate

Set (or replace) the agent's gate. Body is an AgentGateConfig:

curl -X PUT https://api.pipelines.tech/api/agents/42/gate \
  -H "Authorization: Bearer $DYSTOPIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "block_on_constraint": true,
    "block_on_reviewer_severity": "critical",
    "block_on_min_pass_rate": 0.8,
    "block_on_scorers": ["policy_check"]
  }'

Typed: client.set_agent_gate(42, block_on_constraint=True, block_on_reviewer_severity="critical", block_on_min_pass_rate=0.8, block_on_scorers=["policy_check"]), or:

dystopic agents gate set 42 \
  --block-on-constraint --reviewer-severity critical \
  --min-pass-rate 0.8 --scorer policy_check

For teams migrating off a legacy committed config: dystopic agents gate import 42 dystopic.yaml [--suite NAME] lifts a legacy suite's block_on list out of the file and persists it as the platform gate in one shot.

GET /api/agents/{id}/gate

Read the current gate config. Typed: client.get_agent_gate(42) / dystopic agents gate get 42. The response's configured flag distinguishes a persisted gate (even an all-empty advisory one) from no gate at all.

DELETE /api/agents/{id}/gate

Remove the gate. An agent with no gate (or an all-empty AgentGateConfig) is advisory: every PR gets a base-vs-head diff report, but the check never blocks a merge. Typed: client.clear_agent_gate(42) / dystopic agents gate clear 42 — the DELETE is the only way to un-configure the gate; an empty-body PUT persists an all-empty gate that is still configured.

Gate-tier semantics

• Empty gate = advisory. Report only, never blocks. Set at least one tier to make the check enforcing.
• Platform-authoritative, full stop. The gate lives on the agent and is the single source of truth. A PR cannot weaken or remove it — there is nothing in the repo for a head branch to edit. Dropping config on head does not disarm the check; a platform-resolved base fails closed (red).

Human-approval gates (per-tool)

Distinct from the merge gate above: a human-approval gate intercepts individual tool calls during a run and resolves an approval decision before the call may execute. It has two halves:

1. The tool opts in — set requires_human_approval: true on the tool's tools_schema entry (see the tools reference; settable at registration or on the dashboard's tools table). Ungated tools are unaffected.
2. The run decides how approvals resolve — the run-level human_approval_policies config, a JSON object {"mode": ..., "policy"?: ..., "sequence"?: [...]}:

Four decision shapes are recorded in the trace — approve, deny, timeout, needs_more_info — and only approve lets the gated call execute.

On a suite or CI check, when the umbrella agent wires any approval-gated tool and the run carries no explicit policy, the platform stamps the default {"mode": "odyssey_simulated"} — gated tools face a simulated approver rather than a silently bypassed gate.

CI suite binding

Binding a suite to CI tells the platform which suite's frozen scenarios a Dystopic check should run when a connected repo submits dystopic ci review --platform-suite.

PUT /api/agents/{id}/ci-suite

Set the agent's bound CI suite. The suite must belong to this agent. Send null to clear the binding.

# bind suite 7 as the CI suite
curl -X PUT https://api.pipelines.tech/api/agents/42/ci-suite \
  -H "Authorization: Bearer $DYSTOPIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "suite_id": 7 }'

# clear the binding
curl -X PUT https://api.pipelines.tech/api/agents/42/ci-suite \
  -H "Authorization: Bearer $DYSTOPIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "suite_id": null }'

Typed: client.set_agent_ci_suite(42, 7) (pass None to clear) / dystopic agents ci-suite set 42 7 and dystopic agents ci-suite clear 42. Binding a suite never force-creates a gate — the two PUTs are deliberate siblings.

With a suite bound, dystopic ci review --platform-suite submits no config and no scenarios — the platform sources the bound suite's frozen scenarios server-side (byte-identical to an in-app check). It fails closed (422) if the repo is unconnected, the umbrella agent has no run command, or the suite doesn't resolve. See the CI/CD reference for the full zero-config path.

Author it end to end

The whole flow, whether you click it in the dashboard, script it with the CLI, or hit the REST endpoints:

Next