Run & evaluate

Optimization studies

An optimization study searches your strategy's own parameters over many trials to find promising settings — without letting the search quietly overfit. Trials are scored on a train window only, the test window is shown but never scored, promising rows are re-confirmed under conservative fills, and every rank is read against a multiple-testing correction.

What a study is

A study explores the parameters your strategy declares in its params_schema. Each trial picks one set of values and runs a fast, idealized backtest over the train window, scoring a single objective (for example Sharpe). A sampler proposes the next trial from what it has learned so far, up to a trial budget you set.

• A reproducibility receipt. The study records its sampler and its random seed, so the same study can be reproduced exactly.
• Errors are data. Every sampled point gets a row — including pruned trials and error trials. A trial that fails records its error and moves on; it never kills the study.
• Live counters. A study shows complete · pruned · error against its trial budget as it runs.

Launching a study

Studies launch from the Optimization page's New study button. You pick an approved, tests-green plugin version and one instrument — a study searches parameters on a single market (to fan one strategy across many instruments, use the Screener) — then choose the objective, the sampler and the trial budget, and tick the parameters to search: the search space comes from the ranges your strategy's own params schema declares. The train and test windows each have a full calendar picker, with a one-click Split covered range 70/30 helper; overlapping train/test windows are refused up front (trials score on train only — an overlap would leak scored sessions into the displayed test metrics), and both windows must be fully covered by data you hold — a window with uncovered session days is refused, in the dialog and again at the server, because a study scored over sessions the platform does not hold would be a fiction. As with every run, an estimated cost is shown before you commit.

Reading the leaderboard

The leaderboard is ranked server-side — the app never re-orders or re-scores it. Each row shows the trial's parameters (copyable), its scored train objective, and its test objective for context.

• Train is scored; test is displayed, never scored. The number that earns a rank comes from the train window only. The test column sits beside it so you can eyeball how well the setting held up out of sample — but it never feeds the ranking.
• Layer-2 confirmation. The top candidates are re-run under the same conservative fills a graded backtest uses. Confirmed rows rank first, in their post-confirmation order; until confirmation completes, every number is idealized Layer-1 and provisional.
• A multiple-testing verdict. Search enough settings and something will look good by luck. Every rank is read under a deflated-Sharpe / multiple-testing context that accounts for how many trials you ran, with its own verdict per row.
• Confirmed rows get real audits. When a study completes, each confirmed row is automatically promoted: the platform runs it as a full pipeline backtest of that exact parameter set (same approved plugin version, the train window, conservative fills) with stored artifacts and its own audit verdict. The Promoted run column shows the audit chip — click it to open that run's Audit tab. Promoted runs are counted in your strategy's multiple-testing (trial-ledger) tally like any backtest — your study's own trial budget is untouched — and the leaderboard's ORDER never changes because of them: chips add evidence, they don't re-rank.
• Open as a backtest draft. Any row can be opened as a normal backtest for its exact plugin version and parameters, so a promising setting becomes a real, graded, provenance-tracked run.

Variation fan-out — one approval, a whole family

Short of a full study, a strategy page can fan out variations: a batch of parameter-varied siblings launched together under a single brief approval — you approve once, with the batch's cost shown up front, and the whole family runs.

• The group page. Every fan-out lands in a variation group (opened from the strategy), listing each member with its result and audit status side by side.
• Plateau detection and auto-cull. When the family's results flatten out — more variants, no more edge — the group says so, and flat variants are retired automatically. Nothing is deleted: the ledger keeps every member and every verdict, culled or not.
• Honest accounting. Every variation counts in your strategy's multiple-testing tally, exactly like study trials — a family of thirty “tries” is thirty tries, and the audits read it that way.

Why train-only scoring and multiple-testing matter

Optimizing on all of your data, then reporting the best result, is the classic way to fool yourself: you have fitted the noise, not found an edge. A study is built to make that hard. It scores on train only, shows you the test window without letting it flatter the ranking, deflates the result for the number of trials you ran, and re-confirms the leaders under realistic fills before you trust them.