Five reviewers.
Each one earns its vote.
The panel that marked dips in run tess-pi-men-s1: the Sentinels and the Probes, no people. Sentinels run on Claude (Anthropic) by default; other providers are supported and labelled. Probes are deterministic matched-filter detectors.
Sentinels run on Claude (Anthropic) by default; other providers are supported and labelled. Probes are deterministic matched-filter detectors. Weights come from 10 planted and 10 empty light curves each: too few to separate 0.6 from 0.5.
Measured on real TOIs
The live feed's new signals are consistent with noise. Noise alone should give about 12.3 false new_candidate signals per 100 targets reviewed, against 15.8 per 100 on the live feed. Under that earlier rule a new_candidate on the feed was more likely noise than a signal.
The feed's stricter rule, pre-registered and measured again on this code. The signal must now pass in both reviewed sectors with a FAP below 0.001 over 2,000 trials. On the held-out half, noise alone gives 6.1 false signals per 100 targets (95% CI 4.0–9.4), down from 27.8 (23.1–33.1). That is still far above the 1.6 new signals per 100 targets the real light curves give, so the feed is still dominated by false alarms, and feed-level completeness fell from 39% to 17%. A feed admission is a lead for human or follow-up review, not a candidate of any confirmed status. Re-run on this code, the live feed's 19 targets admit 0 new signals.
Measured on a stated benchmark, not discoveries. 300 confirmed or known planets (CP/KP), 300 false positives and 62 false alarms from the TOI table (fetched 2026-10-19, seed 20261019), one TOI per star; held-out half shown. Panel: the three Probes (deterministic detectors) with their frozen calibrated weights; sector-wide BLS; vetting; limb-darkened fit; the feed-level table adds the feed's FAP (200 trials).
| Headline: planets recovered on searchable stars (CP/KP TOIs, a vetted dip on the published ephemeris) | 60% (90/150; 95% CI 52–67%) |
|---|---|
| Planets recovered over all CP/KP stars (no 2-minute data counts as missed) | 59% (90/152; 95% CI 51–67%) |
| Planets recovered in the first run, on the pre-fix code (the historical record) | 57% (86/150; 95% CI 49–65%) |
| Not searchable (no SPOC 2-minute light curve) | 36 stars (2 CP/KP, 34 FP, 0 FA) skewed toward FP: the FP rows describe FP stars with 2-minute data only |
| False-positive TOIs found by the panel | 77% (96/124; 95% CI 69–84%) |
| … of those, rejected or flagged binary-like | 68% (65/96; 95% CI 58–76%) |
| False alarms on inverted light curves at the feed's level (its full rule, FAP < 0.01) | 58 per 1,000 star-sectors (18 over 309; 198 vetted dips before the feed's rule) |
| False alarms on scrambled light curves at the feed's level (its full rule, FAP < 0.01) | 61 per 1,000 star-sectors (19 over 309; 266 vetted dips before the feed's rule) |
| claude-volunteer: CP/KP dips marked (subsample of 37 stars) | 47% (9/19; 95% CI 27–68%) |
| claude-skeptic: CP/KP dips marked (subsample of 37 stars) | 47% (9/19; 95% CI 27–68%) |
The clipping bug is fixed in the code that measured this. 27 CP/KP, 11 FP and 0 FA held-out stars were exposed (depth over 8× the scatter); 23 of the 27 exposed CP/KP are recovered now, against 21 in the first run.
Full results (the first run, on the pre-fix code: benchmark_real_tois.md). TFOPWG dispositions are the reference, not ground truth for every TOI.
Weights. A reviewer's weight is its calibration skill from injection tests: the share of planted transits it finds minus the share of empty light curves it marks, with a floor. The weight a run used is the one its record stores; when it differs from the calibrated weight, the run did not use the calibrated weights. Files and configuration ›
Calibration plane
Each reviewer at its false-alarm rate (x: share of empty subjects it marked) and recall (y: share of injected transits it found) in calibration. Point size is its calibrated weight. The lines join points of equal weight, weight = TPR − FPR, floored at 0.05; up and to the left is better. Stored values only.
Marks per dip
Which reviewers marked each dip, from the run's stored reviewer list. The verdict ledger link opens the dip's checks in its case file.
The real-TOI leaderboard ›SetuHQ's configurations, and any method submitted, scored blind on the held-out half.
Runs and configuration ›Every search run, its errors as stored, and the configuration behind these pages.