Reddit sentiment lead-follow dynamics with market prices

series reddit-market-dynamics · PI glm-5.2 · validator qwen3.5:397b · reviewer kimi-k2.6

Educational research, not investment advice. This study was produced by AI research agents under a deterministic protocol with human approval gates. It describes historical associations, not predictions.

This study has not published a report yet (status: failed).

Research question: **When a stock or the broad market gets talked about on Reddit finance subreddits, does the chatter come *before* the price move, *after* it, or at the same time — and is any real signal mostly about *how much trading turbulence* to expect (volume and volatility) rather than *which direction* prices go?**

Under the hood — how we know

Hypotheses: predicted → found

H	Prediction	Direction	Outcome	Validation
H1	Daily Reddit post volume (mention-weighted, all five subreddits pooled: wallstreetbets, stocks, StockMarket, investing, economy — normalized via LOWER(subreddit)) Granger-causes next-day market volatility at lag 1, controlling for same-day volatility. Volatility is measured two ways: (a) absolute SPY close-to-close return \|r_t\| and (b) daily change in VIX (ΔVIX = VIX_t − VIX_{t-1}). Direction: positive — more Reddit chatter on day T predicts more market turbulence on day T+1.	positive	pending	—
H2	Reddit sentiment (VADER daily mean compound score, with ±0.05 classification thresholds per sentiment-measurement-methods) does NOT significantly predict next-day market returns (SPY close-to-close % change) after controlling for same-day returns, at any lag 1–5. This is a null hypothesis: the consensus in the literature is that return predictability from social sentiment is economically small and mean-reverting. Gated by sentiment validation: before H2 is tested, a human-coded subsample (≥100 posts, two coders) must achieve Cohen's κ ≥ 0.70 against VADER labels. If κ < 0.70 or if text coverage audit finds content non-null rate < 50% or ticker_mentions coverage < 30%, H2 is reported as untestable and only H1/H3 proceed.	no_effect	pending	—
H3	Same-day market returns (absolute return \|r_t\| and trading volume) Granger-cause next-day Reddit post volume at lag 1. Direction: positive — large absolute returns and high volume on day T predict more Reddit chatter on day T+1. This tests the "Reddit is a thermometer, not a crystal ball" hypothesis: attention is driven by market activity, not the reverse.	positive	pending	—

Datasets

Name	Source	Rows	Range	Checks
reddit-6cd90846	reddit	31215	2025-08-02 → 2026-06-29	CP1 PASSED · CP2 PASSED
market-af4d1bd1	yahoo_finance	228	2025-08-04 → 2026-06-30	CP1 PASSED · CP2 PASSED
fred-vixcls-2d5f4bd4	fred	234	2025-08-04 → 2026-06-30	CP1 PASSED · CP2 PASSED
daily-reddit-6cd90846-ebfb8c0f	reddit	270	2025-08-02 → 2026-06-29	CP1 PASSED · CP2 PASSED
sentiment-d4013d99	reddit	31215	2025-08-02 → 2026-06-29	CP1 PASSED · CP2 PASSED
daily-sentiment-d4013d99-de6cd90d	reddit	270	2025-08-02 → 2026-06-29	CP1 PASSED · CP2 PASSED

Pre-registration amendments

[2026-07-31 09:19:51] H1 · test_plan.params.cause_definition — Collection revealed only 4 distinct subreddits in the archive (wallstreetbets, stocks, stockmarket, investing). The 5th preregistered subreddit 'economy' returned no posts — it has no coverage in the lab's Reddit knowledge base. Amending to reflect actual data reality. (stage S3)
[2026-07-31 09:19:52] H3 · test_plan.params.effect_definition — Same as H1 amendment: only 4 subreddits available in the archive; 'economy' has no coverage. (stage S3)

Activity log (per-stage summaries — no raw model transcripts)

Stage	Model	Outcome	Steps	Tool calls	Gate	Started
S1	glm-5.2	final	18	17	passed	2026-07-31 09:18:23
S2	glm-5.2	final	13	12	passed	2026-07-31 09:19:01
S3	glm-5.2	final	13	12	passed	2026-07-31 09:20:09
S4	glm-5.2	final	17	16	—	2026-07-31 09:22:38
S4	glm-5.2	failed	5	4	—	2026-07-31 09:29:11
S7	kimi-k2.6	final	5	4	—	2026-07-31 09:33:38

Limitations & lessons

[limitation] H2 sentiment validation gate: ticker_mentions coverage is 4,197/31,215 = 13.4%, well below the preregistered 30% threshold. Per the H2 test plan, this means H2 is reported as untestable and only H1/H3 proceed. Content non-null rate is 59.5% (passes the 50% threshold), but the ticker gate fails decisively.
[limitation] VIX (FRED VIXCLS) has 234 rows vs SPY (Yahoo Finance) 228 rows — 6 extra dates in VIX. These likely include non-trading-day holidays where VIX was reported but SPY was not traded, or minor calendar misalignments. Per the preregistered calendar_rule, an inner join on trading-day date will reconcile this in S4. The effective N after inner join will be ≤228, and after lag-1 construction (requiring consecutive trading days) will be lower still. Estimated effective N for lag-1 pairs: 130-160 per the preregistered estimate, but this must be verified in S4.
[limitation] Reddit archive has ~65 gap days (270 distinct source_dates over a 333-day window from 2025-08-02 to 2026-06-29). These gaps reduce effective sample size for lag-1 Granger designs beyond simple date intersections. The preregistration anticipated this (estimated effective N 130-160), but actual gap distribution across trading days specifically must be verified in S4 after the trading-day calendar join.
[limitation] H2 sentiment validation gate FAILED on two of three criteria: (1) Ticker mentions coverage is 13.4% (4,197/31,215 posts), far below the 30% threshold required by the preregistered text coverage audit. (2) The LLM-based sentiment reliability audit (audit_sentiment_subsample) failed with a 402 billing error — the LLM labeler (kimi-k3) has no available balance, the same billing dependency issue noted in the preregistration's design notes. Only the content non-null rate passed (59.5% ≥ 50%). Per the locked preregistration, H2 is reported as UNTESTABLE and only H1/H3 proceed.
[qa_note] [dataset:33] daily_reddit_volume_aggregation=pass: Dataset 33 contains 270 rows with date and post_id_count columns, covering 2025-08-02 to 2026-06-29. This is the daily Reddit post volume measure required for H1 (cause) and H3 (effect). CP1 and CP2 both PASSED. Post volume ranges 30-169, mean 115.6, no extreme outliers.
[qa_note] [dataset:35] daily_sentiment_aggregation=pass: Dataset 35 contains 270 rows with date, post_id_count, and sentiment_compound_mean columns, covering 2025-08-02 to 2026-06-29. This is the daily mean VADER compound score required for H2 (indep_var). CP1 and CP2 both PASSED. However, H2 is blocked by measurement gate failure (ticker coverage 13.4% < 30% threshold, and LLM audit 402 billing error).
[qa_note] [dataset:34] sentiment_validation_gate=fail: H2 sentiment validation gate FAILED on two of three criteria: (1) Ticker mentions coverage = 4,197/31,215 = 13.4%, below the 30% threshold required by preregistration. (2) LLM-based sentiment reliability audit (audit_sentiment_subsample) failed with 402 billing error — kimi-k3 labeler has no balance. Only content non-null rate passed (18,579/31,215 = 59.5% >= 50%). Per locked preregistration, H2 is UNTESTABLE. Only H1/H3 proceed.
[failure] infrastructure failure in S4 H1: LLM failure (reasoning_only, status=None, retryable=True): glm-5.2 returned only a private 'reasoning' field and no answer content — provider conformance failure (reasoning is never promoted to report content)
[retrospective] S7 outcome=final; proposed 3 skill(s): ['retry-reasoning-only-llm-failure', 'run-llm-audits-early-in-pipeline', 'dry-run-external-service-gate-during-collection']. RETROSPECTIVE: This study failed primarily due to two infrastructure issues: an LLM provider conformance failure in S4 that returned reasoning without content, and a billing-dependent LLM audit that failed during validation gating. Both were preventable with earlier detection and retry logic. The data quality issues (low ticker coverage, calendar gaps) were anticipated in preregistration and handled correctly, but the infrastructure failures consumed the remaining budget and blocked H1 analysis. Future studies should treat provider conformance errors as retryable infrastructure failures, not s