V3 Agent Track Record vs Market Reality (pilot, 2026-07-06..07-20)

Educational research, not investment advice. This study was designed, executed, and validated by AI research agents under a pre-registered protocol with human approval gates. Every number links to recorded evidence (see the run-card). Nothing here tells you what to buy or sell.

These V3 AI Stock Analysts Couldn't Beat the Market in a Two-Week Pilot

What we found

We tested whether a specific AI stock-analysis system — the V3 platform, which uses seven large language models as stock analysts and seven paired trading agents — could predict market direction or generate profitable trades over a two-week window (July 6–20, 2026). We ran three preregistered tests and two exploratory tests. None of the five reached statistical significance after correction for multiple comparisons.

Test 1 (primary, preregistered): Can the analysts' calls predict excess returns?

We pooled all bullish and bearish signals across the seven analysts and measured whether the direction-adjusted excess return (above the SPY benchmark) at three trading days after entry was greater than zero. The mean was -0.52% — essentially zero, slightly negative. The p-value was 0.691. This was measured across 13 date clusters. Not significant.

Test 2 (secondary, preregistered): Do high-conviction calls outperform lower-conviction ones?

We compared the hit rate (did the stock move in the predicted direction?) for signals the analysts labeled "high" conviction versus all others. High-conviction calls were right 72.7% of the time, compared to 49.6% for lower-conviction calls — a spread of 23.1%. That is a meaningful gap, and a naive paired t-test gave p = 0.039. But the more appropriate block permutation test (which accounts for same-day signals sharing market conditions) gave p = 0.072, and after Holm correction the adjusted p was 0.360. This test used 10 paired date clusters. On 3 dates, only one conviction group appeared, so those dates were dropped. Not significant.

Test 3 (secondary, preregistered): Are the simulated trades profitable?

We simulated 91 trades from the trading agents' proposals: enter at the next session's opening price, exit if the daily price range touches the profit target or the stop-loss level, or close out after five trading days if neither is hit. The mean simulated return per trade was -2.08%, with a median of -0.56%. The win rate was 41.8%. Of the 91 trades, 42 hit their stop-loss, 29 timed out, and 20 hit their profit target. The average holding period was 3.13 trading days. The p-value testing whether the mean return was greater than zero was 0.943. This was computed across 13 date clusters. Not significant.

Exploratory tests (not strictly preregistered):

Two additional tests were included in the Holm correction family but were not part of the original three-hypothesis preregistration. The first pooled all signals across analysts and tested whether the direction-adjusted excess return at three days was greater than zero (p = 0.614). The second tested whether the mean simulated return across all trades was greater than zero (p = 0.117, Holm-adjusted p = 0.468). Neither was significant. These are exploratory observations that need replication.

Overall correction:

All 5 tests — the three preregistered hypotheses plus the two exploratory tests — were Holm-corrected as a single family. After correction, 0 tests were statistically significant.

Per-trader breakdown (exploratory):

Among the seven trading agents, results varied widely. The best performer (deepseek_trader) had 9 simulated trades with a mean return of 2.76% and a win rate of 77.8%. The worst (qwen_trader) had 13 trades with a mean return of -6.69% and a win rate of 30.8%. All seven trading agents produced at least some trade proposals across the two-week window, though one (minimax_trader) produced only 3. These per-trader differences are exploratory and based on very small samples — they should not be interpreted as evidence that any particular AI model is better or worse at trading.

What this does NOT mean

• This is not a verdict on all AI stock analysts. We tested one specific system (the V3 platform) over one specific two-week period (July 6–20, 2026). Different AI systems, different time periods, or different market conditions could produce different results.
• This is not investment advice. Nothing here should be used to make trading decisions.
• The simulated trades do not include commissions, spreads, or borrowing costs. A small positive edge could vanish once real trading costs are included — and we did not find a positive edge to begin with.
• Days where an AI model's API call failed are missing entirely, so results describe only the outputs that were successfully produced. This is a form of survivorship bias.
• The analysts' conviction labels (high, medium, low) are their own self-reports with no guarantee of calibration. We tested whether they carry information about future returns; the result was suggestive but not statistically significant.
• With only thirteen date clusters, this pilot is hypothesis-generating, not confirmatory. The domain literature on forecast evaluation warns that two-week pilots yield enough data for descriptives and effect directions, not for confident skill attribution.

How sure are we?

We used block permutation inference, which accounts for the fact that multiple signals on the same day share the same market conditions and are not independent observations. This is more conservative than a standard t-test and is the appropriate method for clustered data with small sample sizes.

For the conviction comparison (Test 2), the naive paired t-test suggested significance, but the more appropriate block permutation test did not confirm it, and the Holm-corrected p-value was well above conventional thresholds. The gap between high and low conviction is large enough to warrant further investigation with a larger sample, but we cannot claim it is real based on this pilot.

All tests were corrected for multiple comparisons using the Holm method, which controls the family-wise error rate. None survived correction.

With only thirteen date clusters, this pilot is hypothesis-generating, not definitive. The domain literature on forecast evaluation warns that two-week pilots yield enough data for descriptives and effect directions, not for confident skill attribution.

Overall confidence: low (weakest link of data=low, methodology=low, interpretation=moderate)

Independent replication by a second AI model:
- H1: reproduced (agreed)
- H2: interpretation_disagreement (adjudicated:kimi-k2.6)
- H3: reproduced (agreed)

Limitations the agents recorded:
- Market data ticker coverage: The exact ticker list from the agent signals (62 distinct) and trades (78 distinct) datasets could not be programmatically extracted in S3 (describe_dataset returns counts, not values). Market data was fetched for a broad list of 300+ commonly-discussed Reddit-finance tickers across 4 batches (datasets 249-253), achieving 240+ unique tickers with data. However, some scorable tickers referenced by the agents may be absent from this market data if they were not in the guessed list. In S4, such tickers will appear as missing market data and may be miscounted as "non-scorable" alongside the preregistered non-scorable exclusions (VIX, DXY, XAUUSD, foreign listings). The S4 scoring step must verify actual coverage by joining on ticker and reporting the gap between "no Yahoo data" and "preregistered non-scorable."
- Conviction field has 4 distinct values instead of the expected 3 (high/medium/low). The preregistration's H2 conviction split is "high vs not-high (medium+low pooled)." The 4th value may be a null/empty/unknown category. S4 must inspect the actual conviction values and confirm that the "not-high" pool correctly includes only medium and low (not nulls or unknowns). If the 4th value represents missing conviction, those signals should be handled explicitly — either excluded or pooled into "not-high" with justification.
- 34 analyst-days (out of 91 possible: 7 analysts × 13 dates) and 20 trader-days (out of 91 possible) are unrecoverable due to LLM API failures, creating survivorship bias: results describe only the outputs that were successfully produced, conditioning on "the model responded." Per-agent sample sizes are unequal for reasons unrelated to forecasting skill.
- Non-scorable tickers (VIX, DXY, XAUUSD, KOSPI, KRW, and other non-equity symbols) appear in both signals and trades datasets and must be dropped from scoring. The count of dropped signals/trades per date will be reported. Additionally, some equity tickers returned no Yahoo Finance data (NIKE→NKE resolved, but SQ, ASTR, BLOCK, LILM, OSTK, PARA returned nothing — likely delisted/renamed), reducing the scorable sample further.
- The signals dataset has 4 distinct conviction values instead of the expected 3 (high/medium/low). The 4th value may be a null/empty/unknown category. This needs investigation in S4 before the H2 conviction split (high vs not-high) can be executed as preregistered. The preregistration's conviction_split says "high vs not-high (medium+low pooled)" — if the 4th category is neither high nor a valid conviction label, it should be excluded or pooled into not-high, but this decision must be made without peeking at outcomes.
- 34 of 91 possible analyst-days (37%) are unrecoverable due to LLM API failures — no structured output and no parseable markdown. This survivorship bias means the collected signals overrepresent days/analysts whose API calls succeeded, which may not be representative of the agents' true predictive ability.
- The conviction column in the signals dataset (dataset 271) has 4 distinct values, not the 3 (high/medium/low) assumed by the preregistration. The 4th value is unknown at S3 (analysis tools are blocked). H2's "high vs not-high" split should still be feasible since any non-"high" value pools into "not-high," but the composition of the not-high group may differ from the preregistered "medium+low" definition.
- 8 tickers referenced by agents returned no market data from Yahoo Finance: DXY, FXCOF, HXS, KOSPI, KRW, TMTG, VIX, XAUUSD. All are non-equity (indices, currencies, foreign listings) and are correctly classified as non-scorable per the preregistration. They will be dropped and counted per date, reducing the scorable signal count. TMTG may be an alias for DJT (Trump Media), which does have market data — but we treat it as non-scorable since Yahoo Finance did not resolve it.
- The preregistered H1 return formula specifies open(entry_date) → close(entry_date + 2 trading days), but the only whitelisted scoring tool (score_agent_signals) computes close(T0) → close(T0+h) returns. This close-to-close vs open-to-close discrepancy means the executed test does not exactly match the preregistered formula. The direction of the effect and the inference framework (block_permutation, day-clustering, SPY excess, neutral exclusion) are faithful to the prereg, but the return measurement window differs. Additionally, the automatic structured QA flagged the analysis as failed, likely due to this formula mismatch or the tool-to-prereg analysis_tool mapping (prereg says run_ttest, but run_ttest cannot perform a one-sample test with day-clustering; score_agent_signals is the functional equivalent). Results should be interpreted with this caveat.
- Severe ticker attrition in H1 scoring: of 224 raw signals, 38 neutral were excluded (per prereg), but 92 additional non-neutral signals were dropped because their tickers were not found in the market prices dataset (dataset 252). This includes major equity tickers (GOOG, NFLX, NVDA, MSFT, AAPL, INTC, MU, ORCL, PEP, PM) and all non-equity symbols (DXY, KOSPI, KRW, GLD, IGV, KWEB, MSOS, TLT, BIL). Only 94 of 186 non-neutral signals (50.5%) were scored. This 49.5% attrition rate substantially reduces statistical power and introduces potential selection bias if the missing tickers differ systematically from the scored ones. The 13 date clusters are preserved, but per-cluster sample sizes are small (mean ~7.2 signals/cluster).
- H3 sample attrition: Of 92 structured trades with entry/target/stop data, 68 (74%) were dropped because their tickers were absent from the Yahoo Finance price dataset (including ETFs like QQQ/XLF/XLE/XLP, commodities like XAUUSD, and large-cap equities like AAPL/GOOGL/NVDA/ORCL). Only 24 trades across 9 report dates were scored. The clustered one-sample t-test thus operates on n=9 date-cluster means, far below the pilot minimum of 30 — confidence is capped at 'low'. Results are not generalizable to the full trade population and may reflect survivorship bias in which tickers had available price data.
- H3 costs not modeled: Per the pre-registered forecast-evaluation-calibration, commissions, spreads, and borrow costs are NOT modeled. The mean per-trade return of -2.65% (per-trade) / -2.13% (per-date cluster mean) is already negative before costs; adding even modest transaction costs would deepen the loss. This is stated explicitly and does not change the hypothesis verdict, but consumers should understand that real-world trading would face additional frictions.
- Ticker coverage: A large fraction of agent signals referenced non-equity instruments (VIX, DXY, GLD, KOSPI, KRW, TLT, IGV, SMH, MSOS) or tickers not available in the equity price feed (GOOG, GOOGL, NFLX, NVDA, MSFT, AAPL, INTC, MU, ORCL, PEP, PM, etc.). Across H1 analyses, 92–140 signals were dropped for missing price data, reducing the scorable universe by 40–60%. This systematically biases results toward whatever subset of tickers happened to be in the feed, and may exclude instruments where agents had stronger or weaker directional views.
- Low statistical power: The pilot window yielded only 9–13 report-date clusters across the three hypotheses. With day-clustered inference, the effective sample size is the number of clusters, not the number of signals. With 9–13 clusters, the minimum detectable effect size is very large, and any true but modest predictive skill would likely be missed. These results should be interpreted as "no evidence of skill detected in this pilot," not as "evidence of no skill."
- Failure survivorship: Agent-days where a model's API call failed have no structured output and are excluded entirely. Per-agent sample sizes are unequal for reasons unrelated to forecasting skill (e.g., glm_analyst had 8–10 scorable signals while gpt5_analyst had 5–19). Track-record estimates therefore condition on "the model responded," and between-agent comparisons are confounded by differential availability.
- Simulated trade costs: The trade-proposal simulation (H3) uses next-open fills with no modeling of commissions, bid-ask spreads, borrow costs for shorts, or market impact. The mean realized return of -2.13% would likely be even worse net of trading costs. Conversely, the 9-trade sample is too small to distinguish bad luck from bad skill.
- The study covers only 13 date clusters (Tests 1 and 2) and 9 date clusters (Test 3) over a two-week window in July 2026. Statistical power is very low — we could only detect large effects. A smaller real edge would be invisible in a sample this small. This is a hypothesis-generating pilot, not a definitive skill assessment.
- We cannot verify from the recorded analyses whether all seven AI analysts contributed signals on every report date. On days when an analyst's model API call failed, that analyst produced no output. Our results describe only the signals that were successfully generated, not the full potential output of all seven analysts.
- For Test 3 (simulated trades), four of the thirteen report dates had no valid trade proposals to simulate, reducing the sample from 13 to 9 date clusters. We do not know from the recorded data whether these four missing dates were scattered randomly across the window or clustered together. If clustered, the effective sample size could be even smaller than 9, further reducing power.
- Simulated trade returns in Test 3 do not model commissions, bid-ask spreads, or borrow costs. The already-negative average return of -2.13% per trade would likely look worse after accounting for these real-world trading costs.
- Survivorship bias in agent outputs: 34 analyst-days and 20 trader-days were unrecoverable due to LLM API failures (no structured output produced). Only 12 of 46 analyst-days and ~13 of 33 trader-days were recovered. The scored sample (175 signals, 91 trades) reflects only the days where the LLM successfully produced parseable output — these may be systematically different from failure days (e.g., higher market volatility may cause both API failures and different signal quality).
- Non-scorable tickers: 11 signals (DXY, FXCOF, HXS, KOSPI, KRW, TMTG, VIX, XAUUSD) and 1 trade (XAUUSD) reference non-equity symbols that daily-OHLCV equity feeds cannot score. These were dropped per preregistration. This means the analysis covers only equity-like tickers and excludes the agents' views on macro/currency/commodity instruments.
- Small sample for H2 conviction split: only 27 high-conviction signals vs 147 not-high (medium+low pooled) across 13 date clusters. The per-date clustering means the effective sample size for the two-sample comparison is limited by the number of dates where both groups have observations. With ~13 clusters, this is hypothesis-generating per forecast-evaluation-calibration.
- H3 trade simulation does not model commissions, spreads, or borrow costs. The mean realized return of -2.08% would only worsen after transaction costs. A small positive edge could vanish inside these costs; the current negative mean makes this caveat less practically relevant but it remains a stated limitation per forecast-evaluation-calibration.
- The study covers only 13 report-date clusters (July 6-20, 2026), a two-week pilot window. With block-permutation inference over 13 clusters, statistical power is limited — we cannot distinguish 'no skill' from 'skill too small to detect at this sample size.' The forecast-evaluation literature explicitly frames 10-13 date clusters as hypothesis-generating, not confident skill attribution.
- All three primary means are negative, suggesting the study window may have been an adverse market regime for long-biased forecasters. The agents' signals are predominantly bullish (38 neutral signals were dropped), so a broad market downturn would mechanically penalize their direction-adjusted returns. We cannot separate 'agents lack skill' from 'agents are long-biased and the market fell.'
- Survivorship bias: 34 unrecoverable analyst-days and 20 trader-days are missing due to LLM API failures during the study window. Results condition on 'the model responded successfully,' and per-agent sample sizes are unequal for reasons unrelated to forecasting skill. Non-equity symbols (VIX, DXY, XAUUSD, KOSPI, KRW) were dropped from scoring because equity OHLCV feeds cannot price them (11 signals dropped).
- Simulated trade fills use next-session open prices and ignore commissions, bid-ask spreads, slippage, and borrow costs. A small mean edge would vanish inside these costs; the observed negative mean (-2.13%) suggests the agents' trade proposals would have been even worse net of costs. Conviction and confidence labels are the agents' own uncalibrated self-reports — calibration is a research question, not an assumption.
- The two-week pilot window (July 6–20, 2026) yielded only 13 date clusters, which is sufficient for descriptive statistics and effect directions but not for confident skill attribution. The domain literature on forecast evaluation explicitly warns that pilots of this duration are hypothesis-generating. Results should not be interpreted as definitive evidence about the V3 system's capabilities, let alone about AI stock analysts in general.
- The simulated trades do not model commissions, spreads, or borrowing costs. A small positive edge could vanish once real trading costs are included — and no positive edge was found even without these costs. The simulation uses next-session opening prices rather than the agents' stated entry prices, introducing measurable slippage that is reported but not adjusted for.
- Days on which an AI model's API call failed are missing entirely from the archive, so all results condition on "the model responded." This failure survivorship means per-agent sample sizes are unequal for reasons unrelated to skill, and the track-record estimates describe only successfully produced outputs.
- The conviction comparison (H2) showed a promising effect direction — high-conviction calls were right 72.7% of the time versus 49.6% for lower-conviction calls — but the test used only 10 paired date clusters (3 dates were dropped because only one conviction group appeared). The naive p-value of 0.039 did not survive the more appropriate block permutation test (p = 0.072) or Holm correction (adjusted p = 0.360). This result is suggestive but requires replication with a larger sample.

Where this data comes from

agent-signals-fc73d94b — 224 rows covering 2026-07-06 to 2026-07-20 (source: agent_outputs).
- Availability is survivorship-shaped: agent-days with LLM API failures leave no structured output (34 unrecoverable analyst-days and 20 trader-days in 2026-07-06..2026-07-20 alone)
- structured_signals is null for some analysts on some days; the markdown fallback recovers only files with a parseable json fence - Signals reference non-equity symbols (VIX, DXY, XAUUSD, KOSPI, KRW) that daily-OHLCV equity feeds cannot score - Conviction/confidence are the agents' own uncalibrated self-reports - Outputs are generated overnight AFTER report_date (~00:00-00:07 ET), so same-day alignment is look-ahead leakage - agent outputs for report_date D are generated overnight (~00:00-00:07 ET on D+1); scoring entry is the FIRST trading day strictly AFTER D (next-session entry) — the earliest leak-free alignment - primary source: daily-report structured_signals; 12 analyst-days recovered from markdownjson fences
- 34 analyst-days UNRECOVERABLE (no structured output, no parseable markdown — usually an LLM API failure that day); survivorship of parseable outputs is a scope limit
- 19 malformed signal rows dropped (missing/invalid ticker or direction)
- direction vocabulary normalized to lowercase (bullish/bearish/neutral); conviction is the analyst's own label (high/medium/low), not calibrated

agent-trades-a06e4e72 — 299 rows covering 2026-07-06 to 2026-07-20 (source: agent_outputs).
- Availability is survivorship-shaped: agent-days with LLM API failures leave no structured output (34 unrecoverable analyst-days and 20 trader-days in 2026-07-06..2026-07-20 alone)
- structured_signals is null for some analysts on some days; the markdown fallback recovers only files with a parseable ```json fence
- Signals reference non-equity symbols (VIX, DXY, XAUUSD, KOSPI, KRW) that daily-OHLCV equity feeds cannot score
- Conviction/confidence are the agents' own uncalibrated self-reports
- Outputs are generated overnight AFTER report_date (~00:00-00:07 ET), so same-day alignment is look-ahead leakage
- agent outputs for report_date D are generated overnight (~00:00-00:07 ET on D+1); scoring entry is the FIRST trading day strictly AFTER D (next-session entry) — the earliest leak-free alignment
- source: daily-report structured_trades (already-parsed); markdown fallback yields nothing for trades (verified), so 20 trader-days are UNRECOVERABLE (usually an LLM API failure that day)
- 0 malformed trade rows dropped (missing ticker/action/entry_price)
- entry_price is the agent's stated intent at ~midnight after report_date; realized fills are simulated at next-session prices by score_agent_trades, so intent vs fill slippage is part of the measurement
- includes PASS rows (explicitly declined tickers) — they carry no prices and must be excluded from return scoring

market-62c9528c — 17 rows covering 2026-07-06 to 2026-07-28 (source: yahoo_finance).
- Quotes delayed (up to ~15 min on many exchanges)
- Unofficial scraping of Yahoo endpoints - no SLA, breaks periodically
- Survivorship bias: delisted tickers disappear or lose history
- Intraday lookback windows are short and hard-capped
- OPERATOR NOTE (2026-08-08): partial price pull — for scoring use dataset 273 (market-70be6e1b, combined coverage incl. SPY) instead.
- prices are dividend/split adjusted (auto_adjust=True); 'end' is exclusive
- trading days only — align to ET calendar per the preregistered calendar_rule

market-53933aca — 2032 rows covering 2026-07-06 to 2026-07-28 (source: yahoo_finance).
- Quotes delayed (up to ~15 min on many exchanges)
- Unofficial scraping of Yahoo endpoints - no SLA, breaks periodically
- Survivorship bias: delisted tickers disappear or lose history
- Intraday lookback windows are short and hard-capped
- OPERATOR NOTE (2026-08-08): partial price pull — for scoring use dataset 273 (market-70be6e1b, combined coverage incl. SPY) instead.
- prices are dividend/split adjusted (auto_adjust=True); 'end' is exclusive
- trading days only — align to ET calendar per the preregistered calendar_rule
- no data returned for ['DADA', 'HMG', 'NKLA', 'OLD', 'SIEG', 'SQ', 'USAK'] (likely delisted/renamed) — absent from this dataset; account for them in the sample

market-67ba95eb — 2261 rows covering 2026-07-06 to 2026-07-28 (source: yahoo_finance).
- Quotes delayed (up to ~15 min on many exchanges)
- Unofficial scraping of Yahoo endpoints - no SLA, breaks periodically
- Survivorship bias: delisted tickers disappear or lose history
- Intraday lookback windows are short and hard-capped
- OPERATOR NOTE (2026-08-08): partial price pull — for scoring use dataset 273 (market-70be6e1b, combined coverage incl. SPY) instead.
- prices are dividend/split adjusted (auto_adjust=True); 'end' is exclusive
- trading days only — align to ET calendar per the preregistered calendar_rule
- no data returned for ['ALE', 'BITF', 'CYBR', 'NKLA', 'PETQ', 'SCG', 'SPLK', 'TLR', 'WGL'] (likely delisted/renamed) — absent from this dataset; account for them in the sample

market-4f005df0 — 3549 rows covering 2026-07-06 to 2026-07-28 (source: yahoo_finance).
- Quotes delayed (up to ~15 min on many exchanges)
- Unofficial scraping of Yahoo endpoints - no SLA, breaks periodically
- Survivorship bias: delisted tickers disappear or lose history
- Intraday lookback windows are short and hard-capped
- OPERATOR NOTE (2026-08-08): partial price pull — for scoring use dataset 273 (market-70be6e1b, combined coverage incl. SPY) instead.
- prices are dividend/split adjusted (auto_adjust=True); 'end' is exclusive
- trading days only — align to ET calendar per the preregistered calendar_rule
- no data returned for ['CHK', 'COG', 'CTRA', 'DADA', 'HES', 'MMP', 'MRC', 'MRO', 'PXD', 'SGEN', 'SIEG', 'SILI', 'SILV', 'SQ', 'SWN', 'VERV'] (likely delisted/renamed) — absent from this dataset; account for them in the sample

market-76b9fc8b — 4212 rows covering 2026-07-06 to 2026-07-28 (source: yahoo_finance).
- Quotes delayed (up to ~15 min on many exchanges)
- Unofficial scraping of Yahoo endpoints - no SLA, breaks periodically
- Survivorship bias: delisted tickers disappear or lose history
- Intraday lookback windows are short and hard-capped
- OPERATOR NOTE (2026-08-08): partial price pull — for scoring use dataset 273 (market-70be6e1b, combined coverage incl. SPY) instead.
- prices are dividend/split adjusted (auto_adjust=True); 'end' is exclusive
- trading days only — align to ET calendar per the preregistered calendar_rule
- no data returned for ['ANSS', 'BITF', 'DS', 'FLIR', 'H&R', 'HA', 'INST', 'JNPR', 'LM', 'LPS', 'MESA', 'MMC', 'NKLA', 'RE', 'RYA', 'SAVE', 'VMEO', 'Y'] (likely delisted/renamed) — absent from this dataset; account for them in the sample

agent-signals-0cfe5018 — 224 rows covering 2026-07-06 to 2026-07-20 (source: agent_outputs).
- Availability is survivorship-shaped: agent-days with LLM API failures leave no structured output (34 unrecoverable analyst-days and 20 trader-days in 2026-07-06..2026-07-20 alone)
- structured_signals is null for some analysts on some days; the markdown fallback recovers only files with a parseable json fence - Signals reference non-equity symbols (VIX, DXY, XAUUSD, KOSPI, KRW) that daily-OHLCV equity feeds cannot score - Conviction/confidence are the agents' own uncalibrated self-reports - Outputs are generated overnight AFTER report_date (~00:00-00:07 ET), so same-day alignment is look-ahead leakage - agent outputs for report_date D are generated overnight (~00:00-00:07 ET on D+1); scoring entry is the FIRST trading day strictly AFTER D (next-session entry) — the earliest leak-free alignment - primary source: daily-report structured_signals; 12 analyst-days recovered from markdownjson fences
- 34 analyst-days UNRECOVERABLE (no structured output, no parseable markdown — usually an LLM API failure that day); survivorship of parseable outputs is a scope limit
- 19 malformed signal rows dropped (missing/invalid ticker or direction)
- direction vocabulary normalized to lowercase (bullish/bearish/neutral); conviction is the analyst's own label (high/medium/low), not calibrated

agent-trades-94ad9779 — 299 rows covering 2026-07-06 to 2026-07-20 (source: agent_outputs).
- Availability is survivorship-shaped: agent-days with LLM API failures leave no structured output (34 unrecoverable analyst-days and 20 trader-days in 2026-07-06..2026-07-20 alone)
- structured_signals is null for some analysts on some days; the markdown fallback recovers only files with a parseable ```json fence
- Signals reference non-equity symbols (VIX, DXY, XAUUSD, KOSPI, KRW) that daily-OHLCV equity feeds cannot score
- Conviction/confidence are the agents' own uncalibrated self-reports
- Outputs are generated overnight AFTER report_date (~00:00-00:07 ET), so same-day alignment is look-ahead leakage
- agent outputs for report_date D are generated overnight (~00:00-00:07 ET on D+1); scoring entry is the FIRST trading day strictly AFTER D (next-session entry) — the earliest leak-free alignment
- source: daily-report structured_trades (already-parsed); markdown fallback yields nothing for trades (verified), so 20 trader-days are UNRECOVERABLE (usually an LLM API failure that day)
- 0 malformed trade rows dropped (missing ticker/action/entry_price)
- entry_price is the agent's stated intent at ~midnight after report_date; realized fills are simulated at next-session prices by score_agent_trades, so intent vs fill slippage is part of the measurement
- includes PASS rows (explicitly declined tickers) — they carry no prices and must be excluded from return scoring

market-3c3c9c87 — 17 rows covering 2026-07-06 to 2026-07-28 (source: yahoo_finance).
- Quotes delayed (up to ~15 min on many exchanges)
- Unofficial scraping of Yahoo endpoints - no SLA, breaks periodically
- Survivorship bias: delisted tickers disappear or lose history
- Intraday lookback windows are short and hard-capped
- OPERATOR NOTE (2026-08-08): partial price pull — for scoring use dataset 273 (market-70be6e1b, combined coverage incl. SPY) instead.
- prices are dividend/split adjusted (auto_adjust=True); 'end' is exclusive
- trading days only — align to ET calendar per the preregistered calendar_rule

market-e9ed075d — 251 rows covering 2026-07-06 to 2026-07-28 (source: yahoo_finance).
- Quotes delayed (up to ~15 min on many exchanges)
- Unofficial scraping of Yahoo endpoints - no SLA, breaks periodically
- Survivorship bias: delisted tickers disappear or lose history
- Intraday lookback windows are short and hard-capped
- OPERATOR NOTE (2026-08-08): partial price pull — for scoring use dataset 273 (market-70be6e1b, combined coverage incl. SPY) instead.
- prices are dividend/split adjusted (auto_adjust=True); 'end' is exclusive
- trading days only — align to ET calendar per the preregistered calendar_rule

market-f753df69 — 340 rows covering 2026-07-06 to 2026-07-28 (source: yahoo_finance).
- Quotes delayed (up to ~15 min on many exchanges)
- Unofficial scraping of Yahoo endpoints - no SLA, breaks periodically
- Survivorship bias: delisted tickers disappear or lose history
- Intraday lookback windows are short and hard-capped
- OPERATOR NOTE (2026-08-08): partial price pull — for scoring use dataset 273 (market-70be6e1b, combined coverage incl. SPY) instead.
- prices are dividend/split adjusted (auto_adjust=True); 'end' is exclusive
- trading days only — align to ET calendar per the preregistered calendar_rule

market-1e46d169 — 340 rows covering 2026-07-06 to 2026-07-28 (source: yahoo_finance).
- Quotes delayed (up to ~15 min on many exchanges)
- Unofficial scraping of Yahoo endpoints - no SLA, breaks periodically
- Survivorship bias: delisted tickers disappear or lose history
- Intraday lookback windows are short and hard-capped
- OPERATOR NOTE (2026-08-08): partial price pull — for scoring use dataset 273 (market-70be6e1b, combined coverage incl. SPY) instead.
- prices are dividend/split adjusted (auto_adjust=True); 'end' is exclusive
- trading days only — align to ET calendar per the preregistered calendar_rule

market-5f235a45 — 340 rows covering 2026-07-06 to 2026-07-28 (source: yahoo_finance).
- Quotes delayed (up to ~15 min on many exchanges)
- Unofficial scraping of Yahoo endpoints - no SLA, breaks periodically
- Survivorship bias: delisted tickers disappear or lose history
- Intraday lookback windows are short and hard-capped
- OPERATOR NOTE (2026-08-08): partial price pull — for scoring use dataset 273 (market-70be6e1b, combined coverage incl. SPY) instead.
- prices are dividend/split adjusted (auto_adjust=True); 'end' is exclusive
- trading days only — align to ET calendar per the preregistered calendar_rule

agent-signals-9020639e — 224 rows covering 2026-07-06 to 2026-07-20 (source: agent_outputs).
- Availability is survivorship-shaped: agent-days with LLM API failures leave no structured output (34 unrecoverable analyst-days and 20 trader-days in 2026-07-06..2026-07-20 alone)
- structured_signals is null for some analysts on some days; the markdown fallback recovers only files with a parseable json fence - Signals reference non-equity symbols (VIX, DXY, XAUUSD, KOSPI, KRW) that daily-OHLCV equity feeds cannot score - Conviction/confidence are the agents' own uncalibrated self-reports - Outputs are generated overnight AFTER report_date (~00:00-00:07 ET), so same-day alignment is look-ahead leakage - agent outputs for report_date D are generated overnight (~00:00-00:07 ET on D+1); scoring entry is the FIRST trading day strictly AFTER D (next-session entry) — the earliest leak-free alignment - primary source: daily-report structured_signals; 12 analyst-days recovered from markdownjson fences
- 34 analyst-days UNRECOVERABLE (no structured output, no parseable markdown — usually an LLM API failure that day); survivorship of parseable outputs is a scope limit
- 19 malformed signal rows dropped (missing/invalid ticker or direction)
- direction vocabulary normalized to lowercase (bullish/bearish/neutral); conviction is the analyst's own label (high/medium/low), not calibrated

agent-trades-4c988d48 — 299 rows covering 2026-07-06 to 2026-07-20 (source: agent_outputs).
- Availability is survivorship-shaped: agent-days with LLM API failures leave no structured output (34 unrecoverable analyst-days and 20 trader-days in 2026-07-06..2026-07-20 alone)
- structured_signals is null for some analysts on some days; the markdown fallback recovers only files with a parseable ```json fence
- Signals reference non-equity symbols (VIX, DXY, XAUUSD, KOSPI, KRW) that daily-OHLCV equity feeds cannot score
- Conviction/confidence are the agents' own uncalibrated self-reports
- Outputs are generated overnight AFTER report_date (~00:00-00:07 ET), so same-day alignment is look-ahead leakage
- agent outputs for report_date D are generated overnight (~00:00-00:07 ET on D+1); scoring entry is the FIRST trading day strictly AFTER D (next-session entry) — the earliest leak-free alignment
- source: daily-report structured_trades (already-parsed); markdown fallback yields nothing for trades (verified), so 20 trader-days are UNRECOVERABLE (usually an LLM API failure that day)
- 0 malformed trade rows dropped (missing ticker/action/entry_price)
- entry_price is the agent's stated intent at ~midnight after report_date; realized fills are simulated at next-session prices by score_agent_trades, so intent vs fill slippage is part of the measurement
- includes PASS rows (explicitly declined tickers) — they carry no prices and must be excluded from return scoring

market-52d77d05 — 1696 rows covering 2026-07-06 to 2026-07-28 (source: yahoo_finance).
- Quotes delayed (up to ~15 min on many exchanges)
- Unofficial scraping of Yahoo endpoints - no SLA, breaks periodically
- Survivorship bias: delisted tickers disappear or lose history
- Intraday lookback windows are short and hard-capped
- OPERATOR NOTE (2026-08-08): partial price pull — for scoring use dataset 273 (market-70be6e1b, combined coverage incl. SPY) instead.
- prices are dividend/split adjusted (auto_adjust=True); 'end' is exclusive
- trading days only — align to ET calendar per the preregistered calendar_rule
- no data returned for ['NIKE', 'SQ'] (likely delisted/renamed) — absent from this dataset; account for them in the sample

market-2fc158d5 — 1887 rows covering 2026-07-06 to 2026-07-28 (source: yahoo_finance).
- Quotes delayed (up to ~15 min on many exchanges)
- Unofficial scraping of Yahoo endpoints - no SLA, breaks periodically
- Survivorship bias: delisted tickers disappear or lose history
- Intraday lookback windows are short and hard-capped
- OPERATOR NOTE (2026-08-08): partial price pull — for scoring use dataset 273 (market-70be6e1b, combined coverage incl. SPY) instead.
- prices are dividend/split adjusted (auto_adjust=True); 'end' is exclusive
- trading days only — align to ET calendar per the preregistered calendar_rule
- no data returned for ['ASTR', 'BLOCK', 'LILM', 'OSTK', 'PARA'] (likely delisted/renamed) — absent from this dataset; account for them in the sample

agent-signals-28044a1a — 224 rows covering 2026-07-06 to 2026-07-20 (source: agent_outputs).
- Availability is survivorship-shaped: agent-days with LLM API failures leave no structured output (34 unrecoverable analyst-days and 20 trader-days in 2026-07-06..2026-07-20 alone)
- structured_signals is null for some analysts on some days; the markdown fallback recovers only files with a parseable json fence - Signals reference non-equity symbols (VIX, DXY, XAUUSD, KOSPI, KRW) that daily-OHLCV equity feeds cannot score - Conviction/confidence are the agents' own uncalibrated self-reports - Outputs are generated overnight AFTER report_date (~00:00-00:07 ET), so same-day alignment is look-ahead leakage - agent outputs for report_date D are generated overnight (~00:00-00:07 ET on D+1); scoring entry is the FIRST trading day strictly AFTER D (next-session entry) — the earliest leak-free alignment - primary source: daily-report structured_signals; 12 analyst-days recovered from markdownjson fences
- 34 analyst-days UNRECOVERABLE (no structured output, no parseable markdown — usually an LLM API failure that day); survivorship of parseable outputs is a scope limit
- 19 malformed signal rows dropped (missing/invalid ticker or direction)
- direction vocabulary normalized to lowercase (bullish/bearish/neutral); conviction is the analyst's own label (high/medium/low), not calibrated

agent-trades-eef0b528 — 299 rows covering 2026-07-06 to 2026-07-20 (source: agent_outputs).
- Availability is survivorship-shaped: agent-days with LLM API failures leave no structured output (34 unrecoverable analyst-days and 20 trader-days in 2026-07-06..2026-07-20 alone)
- structured_signals is null for some analysts on some days; the markdown fallback recovers only files with a parseable ```json fence
- Signals reference non-equity symbols (VIX, DXY, XAUUSD, KOSPI, KRW) that daily-OHLCV equity feeds cannot score
- Conviction/confidence are the agents' own uncalibrated self-reports
- Outputs are generated overnight AFTER report_date (~00:00-00:07 ET), so same-day alignment is look-ahead leakage
- agent outputs for report_date D are generated overnight (~00:00-00:07 ET on D+1); scoring entry is the FIRST trading day strictly AFTER D (next-session entry) — the earliest leak-free alignment
- source: daily-report structured_trades (already-parsed); markdown fallback yields nothing for trades (verified), so 20 trader-days are UNRECOVERABLE (usually an LLM API failure that day)
- 0 malformed trade rows dropped (missing ticker/action/entry_price)
- entry_price is the agent's stated intent at ~midnight after report_date; realized fills are simulated at next-session prices by score_agent_trades, so intent vs fill slippage is part of the measurement
- includes PASS rows (explicitly declined tickers) — they carry no prices and must be excluded from return scoring

reddit-59b0fbde — 1642 rows covering 2026-07-06 to 2026-07-20 (source: reddit).
- ~65 missing collection days across the 2025-08-02 to 2026-07-29 span — verify per-window coverage before any longitudinal claim
- Scores and comment counts frozen at scrape time, not final values
- Only 5 currently-collected subreddits; not representative of all retail investors
- Deleted/removed posts and comments are invisible or lost
- scores/num_comments are frozen at daily scrape time, not final values — do not use as outcome variables in predictive designs (score-accrual leakage)
- ~65 known gap days in the archive; check CP2 detail

agent-signals-94525a74 — 224 rows covering 2026-07-06 to 2026-07-20 (source: agent_outputs).
- Availability is survivorship-shaped: agent-days with LLM API failures leave no structured output (34 unrecoverable analyst-days and 20 trader-days in 2026-07-06..2026-07-20 alone)
- structured_signals is null for some analysts on some days; the markdown fallback recovers only files with a parseable json fence - Signals reference non-equity symbols (VIX, DXY, XAUUSD, KOSPI, KRW) that daily-OHLCV equity feeds cannot score - Conviction/confidence are the agents' own uncalibrated self-reports - Outputs are generated overnight AFTER report_date (~00:00-00:07 ET), so same-day alignment is look-ahead leakage - agent outputs for report_date D are generated overnight (~00:00-00:07 ET on D+1); scoring entry is the FIRST trading day strictly AFTER D (next-session entry) — the earliest leak-free alignment - primary source: daily-report structured_signals; 12 analyst-days recovered from markdownjson fences
- 34 analyst-days UNRECOVERABLE (no structured output, no parseable markdown — usually an LLM API failure that day); survivorship of parseable outputs is a scope limit
- 19 malformed signal rows dropped (missing/invalid ticker or direction)
- direction vocabulary normalized to lowercase (bullish/bearish/neutral); conviction is the analyst's own label (high/medium/low), not calibrated

agent-trades-d4cfaeb1 — 299 rows covering 2026-07-06 to 2026-07-20 (source: agent_outputs).
- Availability is survivorship-shaped: agent-days with LLM API failures leave no structured output (34 unrecoverable analyst-days and 20 trader-days in 2026-07-06..2026-07-20 alone)
- structured_signals is null for some analysts on some days; the markdown fallback recovers only files with a parseable ```json fence
- Signals reference non-equity symbols (VIX, DXY, XAUUSD, KOSPI, KRW) that daily-OHLCV equity feeds cannot score
- Conviction/confidence are the agents' own uncalibrated self-reports
- Outputs are generated overnight AFTER report_date (~00:00-00:07 ET), so same-day alignment is look-ahead leakage
- agent outputs for report_date D are generated overnight (~00:00-00:07 ET on D+1); scoring entry is the FIRST trading day strictly AFTER D (next-session entry) — the earliest leak-free alignment
- source: daily-report structured_trades (already-parsed); markdown fallback yields nothing for trades (verified), so 20 trader-days are UNRECOVERABLE (usually an LLM API failure that day)
- 0 malformed trade rows dropped (missing ticker/action/entry_price)
- entry_price is the agent's stated intent at ~midnight after report_date; realized fills are simulated at next-session prices by score_agent_trades, so intent vs fill slippage is part of the measurement
- includes PASS rows (explicitly declined tickers) — they carry no prices and must be excluded from return scoring

market-9c9375d5 — 17 rows covering 2026-07-06 to 2026-07-28 (source: yahoo_finance).
- Quotes delayed (up to ~15 min on many exchanges)
- Unofficial scraping of Yahoo endpoints - no SLA, breaks periodically
- Survivorship bias: delisted tickers disappear or lose history
- Intraday lookback windows are short and hard-capped
- OPERATOR NOTE (2026-08-08): partial price pull — for scoring use dataset 273 (market-70be6e1b, combined coverage incl. SPY) instead.
- prices are dividend/split adjusted (auto_adjust=True); 'end' is exclusive
- trading days only — align to ET calendar per the preregistered calendar_rule

agent-signals-a34c8c48 — 224 rows covering 2026-07-06 to 2026-07-20 (source: agent_outputs).
- Availability is survivorship-shaped: agent-days with LLM API failures leave no structured output (34 unrecoverable analyst-days and 20 trader-days in 2026-07-06..2026-07-20 alone)
- structured_signals is null for some analysts on some days; the markdown fallback recovers only files with a parseable json fence - Signals reference non-equity symbols (VIX, DXY, XAUUSD, KOSPI, KRW) that daily-OHLCV equity feeds cannot score - Conviction/confidence are the agents' own uncalibrated self-reports - Outputs are generated overnight AFTER report_date (~00:00-00:07 ET), so same-day alignment is look-ahead leakage - agent outputs for report_date D are generated overnight (~00:00-00:07 ET on D+1); scoring entry is the FIRST trading day strictly AFTER D (next-session entry) — the earliest leak-free alignment - primary source: daily-report structured_signals; 12 analyst-days recovered from markdownjson fences
- 34 analyst-days UNRECOVERABLE (no structured output, no parseable markdown — usually an LLM API failure that day); survivorship of parseable outputs is a scope limit
- 19 malformed signal rows dropped (missing/invalid ticker or direction)
- direction vocabulary normalized to lowercase (bullish/bearish/neutral); conviction is the analyst's own label (high/medium/low), not calibrated

agent-trades-1bc8f1bb — 299 rows covering 2026-07-06 to 2026-07-20 (source: agent_outputs).
- Availability is survivorship-shaped: agent-days with LLM API failures leave no structured output (34 unrecoverable analyst-days and 20 trader-days in 2026-07-06..2026-07-20 alone)
- structured_signals is null for some analysts on some days; the markdown fallback recovers only files with a parseable ```json fence
- Signals reference non-equity symbols (VIX, DXY, XAUUSD, KOSPI, KRW) that daily-OHLCV equity feeds cannot score
- Conviction/confidence are the agents' own uncalibrated self-reports
- Outputs are generated overnight AFTER report_date (~00:00-00:07 ET), so same-day alignment is look-ahead leakage
- agent outputs for report_date D are generated overnight (~00:00-00:07 ET on D+1); scoring entry is the FIRST trading day strictly AFTER D (next-session entry) — the earliest leak-free alignment
- source: daily-report structured_trades (already-parsed); markdown fallback yields nothing for trades (verified), so 20 trader-days are UNRECOVERABLE (usually an LLM API failure that day)
- 0 malformed trade rows dropped (missing ticker/action/entry_price)
- entry_price is the agent's stated intent at ~midnight after report_date; realized fills are simulated at next-session prices by score_agent_trades, so intent vs fill slippage is part of the measurement
- includes PASS rows (explicitly declined tickers) — they carry no prices and must be excluded from return scoring

market-70be6e1b — 1316 rows covering 2026-07-06 to 2026-07-28 (source: yahoo_finance).
- Quotes delayed (up to ~15 min on many exchanges)
- Unofficial scraping of Yahoo endpoints - no SLA, breaks periodically
- Survivorship bias: delisted tickers disappear or lose history
- Intraday lookback windows are short and hard-capped
- OPERATOR NOTE (2026-08-08): PREFERRED price dataset for scoring — combined ticker coverage incl. SPY (independent-review remediation).
- prices are dividend/split adjusted (auto_adjust=True); 'end' is exclusive
- trading days only — align to ET calendar per the preregistered calendar_rule
- no data returned for ['DXY', 'FXCOF', 'HXS', 'KOSPI', 'KRW', 'TMTG', 'VIX', 'XAUUSD'] (likely delisted/renamed) — absent from this dataset; account for them in the sample

How we know

Every number above links to a recorded analysis. The full audit trail — pre-registration (locked before any data was seen), datasets, every executed test, and the independent replication — is in this study's run-card.