Reddit sentiment lead-follow dynamics with market prices

Hypotheses: predicted → found

Datasets

Pre-registration amendments

• [2026-08-01 05:33:05] H1 · hypotheses[0].test_plan.params.variable_definitions.reddit_post_volume — Collection revealed r/stocks has zero posts in the analysis window (2025-08-02 to 2026-06-30). The source profile documents r/stocks was added late to the collector with only 391 total rows in the full archive (2025-08-02 to 2026-07-29). The preregistration's calendar_rule already specifies: 'Per-subreddit verification: confirm each of 5 subreddits has non-zero posts in analysis window before aggregation; exclude any with zero observations and document reduction.' This amendment documents the actual exclusion, reducing the subreddit count from 5 to 4. (stage S3)
• [2026-08-01 05:33:08] H2 · hypotheses[1].test_plan.params.variable_definitions.reddit_sentiment — Same r/stocks exclusion as H1 amendment (amendment_id 12). H2's sentiment aggregation also depends on the 5-subreddit pool; reducing to 4 subreddits for consistency. The per-subreddit verification contingency in the calendar_rule applies to all hypotheses using Reddit data. (stage S3)

Activity log (per-stage summaries — no raw model transcripts)

Limitations & lessons

• [limitation] The provided preregistration text was truncated at H3's justification_refs, so H3's full test plan (variable definitions, data requirements, test parameters) is unknown. Data collection for H3 could not be performed. Only H1 and H2 test plans were available in full. If H3 requires additional data sources (e.g., VIX, Fear & Greed index, FRED series), those datasets were not collected.
• [limitation] Reddit dataset (dataset 114) has only 4 distinct subreddits instead of the preregistered 5. r/stocks has zero posts in the analysis window (2025-08-02 to 2026-06-30). The source profile confirms r/stocks was added late to the collector with only 391 total rows in the full archive. Amendments 12 and 13 document the exclusion, reducing the subreddit pool from 5 to 4 for both H1 and H2. This reduces the breadth of the attention/sentiment composite and may bias results toward the 4 surviving subreddits' community norms (r/economy dominates at ~41% of posts).
• [limitation] Reddit content field is null for 12,636 of 31,215 posts (40.5%). These are link posts with empty bodies. The preregistration's title-only VADER fallback (per lab skill preregister-reddit-sentiment-title-fallback) handles this, but VADER performance on short titles differs from body text. The preregistered sensitivity analysis (title-only vs title+body) will be important for assessing robustness.
• [limitation] Reddit archive has ~65 missing collection days across the 2025-08-02 to 2026-07-29 span (270 distinct source_dates out of ~362 calendar days). The preregistration's calendar_rule specifies forward-fill from most recent prior collection day (max 1 calendar day gap) and exclusion of trading days with no Reddit collection within 2 calendar days. This will reduce the effective sample size for Granger causality tests, which require complete consecutive pairs. The gap days may cluster temporally (per lab skill audit-gap-clustering-for-lagged-designs), further reducing effective sample size.
• [limitation] SPY market data (dataset 115) starts 2025-08-04 (first trading day after 2025-08-02, which was a Saturday). The preregistration's primary window starts 2025-08-02, but since markets are closed on weekends, this is expected and not a gap. 228 trading days are available through 2026-06-30. The preregistered calendar_rule aligns to US equity trading days (Mon-Fri, excluding NYSE holidays), so this is consistent.
• [lesson:data] When preregistering a multi-subreddit Reddit aggregation, verify that ALL requested subreddits have non-zero coverage in the target analysis window BEFORE locking the preregistration. r/stocks was listed as one of 5 subreddits but had zero posts in the 2025-08-02 to 2026-06-30 window because it was added late to the collector (only 391 total rows in the full archive). The per-subreddit verification contingency in the calendar_rule saved the study, but the amendment still caps confidence at 'moderate'. Future studies should query per-subreddit coverage in the specific window during S1/S2 design, not just check the source profile's overall counts.
• [qa_note] [dataset:119] reddit_daily_measures_construction=pass: Verified daily Reddit measures: reddit_post_volume = post_id_count per source_date (270 collection days), reddit_sentiment = sum(weighted_sentiment)/sum(engagement_weight) where weight = score + num_comments + 1. Both columns non-null, values in expected ranges (volume 47-148 in sample, sentiment in [-1,1]). Column names match prereg variable definitions.
• [qa_note] [dataset:120] market_measures_construction=pass: Verified SPY market measures: spy_returns = pct_change(close) (NaN for first row as expected), vol_ma20 = 20-day rolling mean of volume, spy_volume_ratio = volume/vol_ma20, spy_realized_vol = 5-day rolling std of spy_returns. All 228 trading days present. Column names match prereg robustness specs.
• [qa_note] [dataset:122] plan_variable_name_conformance=pass: Verified all preregistered variable names present with exact matching: reddit_post_volume, reddit_sentiment, spy_volume, spy_returns, spy_volume_ratio, spy_realized_vol. These match the H1 and H2 test_plan variable_definitions and robustness specs. The panel has 270 rows (Reddit collection days) left-joined to 228 SPY trading days — weekends/holidays show NaN for market columns as expected.
• [limitation] Reddit post volume is capped at ~250 posts/day by the posts_limit=50 per subreddit collection parameter, which truncates high-attention days and may attenuate any true attention-volume relationship. This cap is a structural feature of the data source, not a design choice in this study.
• [limitation] Granger causality tests only linear predictive content in the conditional mean. Nonlinear dependencies (e.g., threshold effects where only extreme attention spikes predict volume) are not detected by this test. The null result does not rule out nonlinear relationships.
• [limitation] Only 5 subreddits (wallstreetbets, stocks, stockmarket, investing, economy) were sampled. Reddit activity outside these communities is not captured. The representativeness of these 5 subreddits for broader retail-investor attention is an untested assumption.
• [limitation] Sample size of ~200 trading days limits statistical power to detect small effects. The literature suggests typical forward sentiment-to-returns effects are 1-10 basis points per day, which may be below the detection threshold of this sample. A larger sample or longer history could reveal effects that this study cannot rule out.
• [limitation] Daily aggregation obscures intraday dynamics. A price move at 10am can generate Reddit posts by 11am, which precede the close — making contemporaneous daily correlations causally uninterpretable. Intraday timestamp analysis would be needed to disentangle the sequencing of sentiment and price moves.
• [limitation] No news-event controls were included. The dominant confound in sentiment-price studies is that news events drive both social-media reactions and price moves simultaneously. Without conditioning on news counts or event indicators, any observed correlation could be co-movement driven by a common cause rather than a predictive relationship.
• [limitation] VADER sentiment was used without a dedicated financial-domain validation. VADER was designed for general social media text, not financial jargon. Sarcasm, financial terminology, and context-dependent phrases (e.g., 'bullish', 'bear trap') may be misclassified. The sentiment audit (if conducted) should be checked for adequacy; if VADER labels were deemed inadequate, the pre-registered pivot to attention/volume measures would apply.
• [lesson:inference] When running Granger causality tests across multiple hypotheses with multiple lags, the Holm correction family grows quickly (3 hypotheses × 5 lags × 2 directions = 20 tests in our case). With n~200 daily observations, the corrected p-values inflate to 1.0 for all but very strong effects. Future studies should consider: (1) pre-specifying a single primary lag rather than testing all 1-5, (2) using a larger sample if small effects are plausible, or (3) acknowledging upfront that the multiple-comparison burden may render the study effectively underpowered.
• [limitation] The study window covers approximately 200 daily observations, which limits statistical power to detect weak Granger causality effects. Daily aggregation may obscure intraday relationships between Reddit activity and market movements. VADER sentiment analysis was not specifically trained on financial discourse and may inadequately capture the nuance of investing-related Reddit discussion. SPY is a broad-market ETF proxy; Reddit-driven effects may concentrate in individual stocks (e.g., meme stocks) rather than the aggregate market. Only linear Granger causality was tested; nonlinear dependencies would not be detected.
• [failure] S5 failed its gate twice: ["validator emitted no 'VERDICT H1: ...' line", "validator emitted no 'VERDICT H2: ...' line", "validator emitted no 'VERDICT H3: ...' line"]