Run & evaluate

Optimization studies

An optimization study searches your strategy's own parameters over many trials to find promising settings — without letting the search quietly overfit. Every trial runs on the same conservative simulator that grades a backtest, trials are scored on a train window only, the test window is shown but never scored, and every rank is read against a multiple-testing correction.

What a study is

A study explores the settings your strategy declares — the whole-number, decimal and choice fields of its settings class (a built-in template's own parameters, for a template). Each trial picks one set of values and runs a full conservative-tier backtest over the train window — the same fills, costs and venue a graded backtest uses; there is no cheaper tier — scoring a single objective (for example Sharpe). A sampler proposes the next trial from what it has learned so far, up to a trial budget you set (an administrator's ceiling bounds the budget, one trial's wall clock and the whole study's machine time, and the launch tells you the projected bill before it is spent).

• A reproducibility receipt. The study records its sampler and its random seed, so the same study can be reproduced exactly.
• Errors are data. Every sampled point gets a row — including pruned trials and error trials. A trial that fails records its error and moves on; it never kills the study.
• Live counters. A study shows complete · pruned · error against its trial budget as it runs.

Launching a study

Studies launch from the Optimization page's New study button. You pick an approved, tests-green plugin version — or, since 2026-10-22, A built-in template: a strategy version of one of the two templates, whose study varies the spec's numeric settings on the spec's own markets and data, needs the spec's brief approved (it spends none of its submissions), and promotes no row (the values to try are shown for a new version of the spec) — and its market — a study searches parameters on one market, or, for a strategy that reads several markets together (since October 2026), on that set: every trial runs all of them, in your order, and the promoted backtests too (to fan one strategy across many markets one at a time, use the Screener) — then choose the objective, the sampler and the trial budget, and tick the parameters to search: the search space comes from the ranges your strategy's own params schema declares. The train and test windows each have a full calendar picker, with a one-click Split covered range 70/30 helper; overlapping train/test windows are refused up front (trials score on train only — an overlap would leak scored sessions into the displayed test metrics), and both windows must be fully covered by data you hold — a window with uncovered session days is refused, in the dialog and again at the server, because a study scored over sessions the platform does not hold would be a fiction. As with every run, an estimated cost is shown before you commit.

Reading the leaderboard

The leaderboard is ranked server-side — the app never re-orders or re-scores it. Each row shows the trial's parameters (copyable), its scored train objective, and its test objective for context.

• Train is scored; test is displayed, never scored. The number that earns a rank comes from the train window only — under walk-forward, from the validation days inside it (the bullet below). The test column sits beside it so you can eyeball how well the setting held up out of sample — but it never feeds the ranking.
• Walk-forward, for engine-native strategies. A study can roll forward instead of sampling: pick a train size, a validation size and a step, and the study fits every combination of your search grid on each fold's train days, judges the fold's winner once on the validation days it never saw (its honest out-of-sample result), and runs each distinct winner once on the test window (shown, never scored). On such a study's leaderboard the fold winners come first, ranked by their mean out-of-sample score — shown in its own Out-of-sample column beside each winner's train score — and the cells after them rank by train score. Every cell on every fold counts as one trial in the multiple-testing record, an administrator's walk-forward budget bounds folds × cells, and the study page shows a fold-by-fold table — the evidence a walk-forward study is for. A run promoted from such a study carries the folds, so its audit's walk-forward check reads them. A fold winner that loses out of sample (its mean out-of-sample score at or below −1) is your call per study: skip its full backtest (the default — the row says why) or promote it anyway and have the row marked degenerate; the launch dialog explains both with an example. A step smaller than the validation size makes the folds share sessions — the study says so on its page (or refuses the launch, where the platform's administrator chose so). The trial pruner — stop at a drawdown, stop a parameter set that never trades — runs inside every trial, and the study records which rules it ran under. The strategy's remaining config fields — the ones you are not searching — are set once on the launch under Strategy settings and ride under every trial and the promoted backtest; a required field with no default must be given.
• One tier, nothing to confirm. Because every trial already ran on the conservative simulator, the leaderboard is the score order — no second pass re-ranks it. Studies run before this change ranked on an idealized tier and re-confirmed their leaders; those studies keep their labels and their confirmation column exactly as recorded, under a heading that says so.
• A multiple-testing verdict. Search enough settings and something will look good by luck. Every rank is read under a deflated-Sharpe / multiple-testing context that accounts for how many trials you ran, with its own verdict per row.
• Top rows get real audits. When a study completes, each top-K row is automatically promoted: the platform runs it as a full pipeline backtest of that exact parameter set (same approved plugin version, the train window, conservative fills) with stored artifacts and its own audit verdict. The Promoted run column shows the audit chip — click it to open that run's Audit tab. Promoted runs are counted in your strategy's multiple-testing (trial-ledger) tally like any backtest — your study's own trial budget is untouched — and the leaderboard's ORDER never changes because of them: chips add evidence, they don't re-rank.
• Open as a backtest draft. Any row can be opened as a normal backtest for its exact plugin version and parameters, so a promising setting becomes a real, graded, provenance-tracked run.

Variation fan-out — one approval, a whole family

Short of a full study, a strategy page can fan out variations: a batch of parameter-varied siblings launched together under a single brief approval — you approve once, with the batch's cost shown up front, and the whole family runs.

• The group page. Every fan-out lands in a variation group (opened from the strategy), listing each member with its result and audit status side by side.
• Plateau detection and auto-cull. When the family's results flatten out — more variants, no more edge — the group says so, and flat variants are retired automatically. Nothing is deleted: the ledger keeps every member and every verdict, culled or not.
• Honest accounting. Every variation counts in your strategy's multiple-testing tally, exactly like study trials — a family of thirty “tries” is thirty tries, and the audits read it that way.

Why train-only scoring and multiple-testing matter

Optimizing on all of your data, then reporting the best result, is the classic way to fool yourself: you have fitted the noise, not found an edge. A study is built to make that hard. It scores on train only, shows you the test window without letting it flatter the ranking, deflates the result for the number of trials you ran, and runs every trial on the realistic simulator in the first place — so the leaders never need a second, more honest look.