Run & evaluate

Optimization studies

An optimization study searches your strategy's own parameters over many trials to find promising settings — without letting the search quietly overfit. Trials are scored on a train window only, the test window is shown but never scored, promising rows are re-confirmed under conservative fills, and every rank is read against a multiple-testing correction.

What a study is

A study explores the parameters your strategy declares in its params_schema. Each trial picks one set of values and runs a fast, idealized backtest over the train window, scoring a single objective (for example Sharpe). A sampler proposes the next trial from what it has learned so far, up to a trial budget you set.

• A reproducibility receipt. The study records its sampler and its random seed, so the same study can be reproduced exactly.
• Errors are data. Every sampled point gets a row — including pruned trials and error trials. A trial that fails records its error and moves on; it never kills the study.
• Live counters. A study shows complete · pruned · error against its trial budget as it runs.

Launching a study

Studies launch from a workspace's run controls, the same place you start a preview or a backtest. You choose the objective, the train and test windows, the sampler and the trial budget; the study then runs its trials and fills in the leaderboard as results arrive. As with every run, an estimated cost is shown before you commit.

Reading the leaderboard

The leaderboard is ranked server-side — the app never re-orders or re-scores it. Each row shows the trial's parameters (copyable), its scored train objective, and its test objective for context.

• Train is scored; test is displayed, never scored. The number that earns a rank comes from the train window only. The test column sits beside it so you can eyeball how well the setting held up out of sample — but it never feeds the ranking.
• Layer-2 confirmation. The top candidates are re-run under the same conservative fills a graded backtest uses. Confirmed rows rank first, in their post-confirmation order; until confirmation completes, every number is idealized Layer-1 and provisional.
• A multiple-testing verdict. Search enough settings and something will look good by luck. Every rank is read under a deflated-Sharpe / multiple-testing context that accounts for how many trials you ran, with its own verdict per row.
• Open as a backtest draft. Any row can be opened as a normal backtest for its exact plugin version and parameters, so a promising setting becomes a real, graded, provenance-tracked run.

Why train-only scoring and multiple-testing matter

Optimizing on all of your data, then reporting the best result, is the classic way to fool yourself: you have fitted the noise, not found an edge. A study is built to make that hard. It scores on train only, shows you the test window without letting it flatter the ranking, deflates the result for the number of trials you ran, and re-confirms the leaders under realistic fills before you trust them.