Build

Testing your strategy

Before a strategy touches any market data, it has to prove itself against a fixed set of required tests. This page explains what each test is for in plain terms, how to run the suite, and why the Preview and Backtest buttons stay locked until it is green.

Why tests come first

A backtest can only tell you about your idea if the code faithfully implements the idea. The required tests are how you (and the platform) know it does: they pin down the strategy's behavior on small, hand-crafted data windows where the right answer is known. They also encode the failure modes that quietly ruin backtests — looking ahead, mishandling session boundaries, resolving an ambiguous bar optimistically — so that a strategy which cheats, even accidentally, fails loudly before it produces a single misleading statistic.

The twelve required test cases

Every strategy workspace ships with a test file that names twelve required cases — the test taxonomy. They start as deliberate failures (“write me” stubs); your job is to make each one real. In plain terms:

Two more cases in that file already pass on a fresh workspace and are yours to keep green: test_runs_at_schema_defaults builds the strategy’s parameters from params_schema defaults — the exact values a run with no overrides uses — and calls detect_setups with them; and test_dispatch_idempotence, an optional category, checks that on every bar a fresh instance asked once answers what an instance walked bar by bar answers (a strategy that keeps a flag on itself and reads only the newest bar fails it). The params fixture in tests/conftest.py is how every test should get its parameters; a suite that never builds them at their defaults gets an advisory in the Tests panel (informational — the gate stays green).

Running the tests

1. Cut a version first. Tests run against a specific version of your code, cut from the snapshot timeline. If you haven't cut one, the Run Tests button says so: “Cut a plugin version first.”
2. Press Run Tests. The suite is queued (you'll see “Test suite queued”) and runs in an isolated environment against your saved code — the same code, byte for byte, that a later run would use.
3. Read the results in the Tests panel. A summary line — e.g. “14 passed · 1 failed · 0 skipped · taxonomy INCOMPLETE” — sits above a per-case table with the case name, its taxonomy tag, the outcome, the duration and the failure message. “Taxonomy complete” means every one of the twelve required cases exists and ran.

Green means: at least one case ran, all cases passed, and the taxonomy is complete. A passing suite with a missing case is not green — a strategy can't skip the lookahead test just by deleting it — and a suite where nothing ran is not green either, because it has proved nothing.

Why Preview and Backtest unlock only when tests are green

The Preview and Backtest buttons are gated on the latest test results for the code version you're running. Until the suite is green, they are disabled, and the tooltip tells you exactly why:

• “Tests gate closed: no suite run recorded — Run Tests first.”
• “Tests gate closed: 2 failing case(s) · 4 passed — open the Tests tab and use ‘Fix failing tests with AI’.”
• “Tests gate closed: the latest suite run recorded no passing tests — write at least one passing case, then Run Tests again.”
• “Tests gate closed: required test categories are missing — open the Tests tab and use ‘Add missing tests with AI’.”

Those four are in precedence order, and it is the same order everywhere the result is shown — the Tests panel, the run buttons, the launch dialog and the server that actually refuses the run. Failing cases are named first (they are the loudest fact a suite carries); a suite where nothing passed says so before it talks about categories, because that is the reason it has none. So a suite that is both red and missing categories reads as red on every screen — you will never see one surface call it a passing suite while another refuses to run it.

The launch dialog repeats the gate with a status line — “Tests gate: green — 14 passed / 0 failed, taxonomy complete” — so you always know what the gate saw.

The reasoning: a preview or backtest of untested code produces numbers that look exactly like evidence but may just be bugs. The gate makes the cheap, fast check (seconds of tests on synthetic fixtures) a hard prerequisite for the expensive, persuasive one (minutes of running on market data). It's the same discipline a good quant team enforces socially — here it's built in.

When a test fails

• Read the Message column first — scaffolded stubs fail with a description of what you're supposed to build; real failures show the assertion that broke.
• Fix the code (or the test, if the test encoded the wrong expectation — you own both), save, cut a version and run again.
• Or hand the failures to the AI: when cases fail, the Tests panel offers Fix failing tests with AI, which pre-fills the AI task lane with the failing cases — every one the button counted, named by its test id, including cases that errored (a broken import, a failure in setup) rather than merely failed. The fix arrives as a reviewable proposal like any other AI edit — nothing is applied without your accept — and the task can be stopped mid-flight. And when the AI concludes that nothing should change — the code already does what the failing test asks, or the test cannot pass without changing what you asked for — it says so instead of inventing an edit: you get its reason in the proposal you open, nothing is staged, and your suite is not re-run for it. The AI sees the full output of every failing case — whole tracebacks, not just the first line — including collection errors, so a broken conftest.py arrives with the reason it broke. The AI may fix the cause anywhere in the workspace, production code included — and if one of your tests genuinely encoded the wrong expectation, it may correct that too, assertions and all. Nothing is refused for it. What it owes you in exchange is a reason: for every test file it changes, it has to say why, in words you can check.
• It runs your tests itself, and keeps going until they pass — before you see anything. The AI writes a change, runs your suite in the sandbox, reads the real failures, fixes them and runs again — up to four runs per request. A repair attempt can end the same honest way as a first answer: "nothing should change", with the reason, rather than a broken reply. Only then does it hand you a proposal, and a verdict it earned is always about the exact files in front of you — never another version's result wearing this version's badge. It reports what actually happened, in those words: green after 2 repair attempts is a count, not a promise — you will not see the word "fixed". If it cannot get there, you still get the diff, marked could not get them green with the run count and the cases still failing, so a half-finished attempt is never dressed up as a fix. If the sandbox could not run the final files at all — a busy lane, a timeout — nothing is claimed for them: the ordinary background check runs them instead and the verdict lands when it finishes, and if an earlier attempt did run red, its facts stay in the warnings (the last run that did happen was RED) instead of being replaced by a shrug — never a quiet pass either way. Press Stop after the AI has rewritten the files but before they were re-run, and the proposal says exactly that — the tests were not re-run on this version — rather than pinning the previous version's failures on files that never ran. And if it changed one of your tests without saying why, that version is not run at all and no run is started for it later: you are told the tests were not re-run and that the last real state of the suite was red, so a rewritten assertion can never come back later wearing a green badge. Two honest limits: test stubs nobody has written yet (pytest.fail("not yet written: …")) are red no code change can fix, so the AI stops instead of burning attempts on them and points you at Write tests; and a green run proves your suite passes, which is not the same as proving the tests were not changed to make it pass — that is what the three groups below are for.
• The promise is not "the AI can't touch your tests" — it is that you are never told the tests pass without being told the tests changed. Every fix arrives sorted into three groups, and all three are shown even when they are empty, because "no test files touched" is the reassuring answer and you should be able to see it at a glance: fixes production code · changes what a test checks · changes test setup. Each group has its own Accept button — your strategy code, what your tests check, and the fixtures that run around them are three separate decisions, and accepting one never applies another. Next to each changed test you get the AI's own reason for changing it; if it changed a test and said nothing, the panel says so out loud rather than letting it pass unnoticed. The test-setup group opens expanded, because that is the one thing no automatic check can judge: a fixture can quietly replace the code under test, so a suite can go green without the bug being fixed. That diff is put in front of you on purpose.
• The platform still finds every way a red suite can be turned green without anything being fixed, and names each one next to the diff in plain language — a test deleted, renamed, skipped or with its check rewritten; taken out of the suite without being edited at all (__test__ = False, collect_ignore, a conftest.py hook or fixture that skips it by name, or moving it to a file the runner never looks at); or the value it compares against moved, when that value lives in a constant, a helper or a fixture beside it — or in a shared helper module your tests import. The checks cover the files the test runner itself loads, and inside those they flag top-level code by default rather than trying to list what is dangerous, so an innocent one-liner at the top of a test file gets a mention too. None of that stops the change any more; it just means you are told. Wording stays yours either way: improving a failure message is never read as a change to what a test proves. And when the platform runs your suite and it passes, it says so with the caveat when there is one — "the tests passed" reads exactly the same whether the code was fixed or the test was weakened, so if the run used tests or fixtures this change touched, you are told that in the same breath. One honest limit, stated rather than implied: a fixture that fakes your code can turn a red suite green with the bug still there, and no automatic check catches it. That is exactly why that diff is the one opened for you. The fix arrives as a reviewable proposal like any other AI edit — nothing is applied without your accept — and the task can be stopped mid-flight.
• You can watch it work. The fix does not hide behind a spinner: each attempt at running your suite appears as its own row and resolves from Running to Passing or Failing, with the counts, the names of the cases that failed, and the actual assertion text you can open and read right there — not a summary of it. Expand or collapse any attempt; the panel keeps your place while new results arrive. You can stop at any point, and stopping keeps what the run has already produced: it finishes the step it is paying for right now — a test run can take a couple of minutes — then stops starting new ones and hands you the change it had written as a normal reviewable proposal, marked as stopped, with the last real test result attached. It never claims a stopped run passed. Every run ends by saying plainly which of four things happened — your suite passes; it could not be got passing (the attempt is still there to read); your tests could not be run at all; or you stopped it — because "finished" and "fixed" are not the same claim. If you open the same workspace in another tab, or switch away from the panel and come back, it picks the story up where it is — and reloading the page no longer ends the fix: the run keeps going on the server and the panel reconnects to it, with the Stop button still there. Starting a second fix in the same workspace while one is running is refused rather than queued, so a double-click cannot bill you twice.
• If your suite is both failing and missing required test categories, every screen tells you about the failures first. Adding a category to a suite that cannot run would not make it pass.
• Remember the suite runs your saved files — the autosave chip should read Saved before you run.