Reddit sentiment lead-follow dynamics with market prices

Hypotheses: predicted → found

Datasets

Activity log (per-stage summaries — no raw model transcripts)

Limitations & lessons

• [limitation] Reddit content field is only 59.8% non-null (20,416 of 34,153 posts). 40% of posts are link-only with empty body text, so VADER and LLM sentiment will rely primarily on post titles for those posts. Titles are 100% non-null but are typically shorter and less informative than body text, which may reduce sentiment classification quality and increase noise in the bullish-bearish ratio and sentiment dispersion measures. This is a structural property of the Reddit archive (link posts have no body), not a data collection failure.
• [limitation] Reddit ticker_mentions field is only 13.4% non-null (4,578 of 34,153 posts). While tickers are not directly used in the preregistered sentiment measures (which use VADER compound scores and LLM labels on post text), the sparse ticker coverage limits any potential supplementary analysis of ticker-specific sentiment. The preregistered analysis aggregates at the daily level across all posts regardless of ticker mentions, so this does not block the planned tests.
• [limitation] Reddit archive has 297 distinct collection days over a 362-calendar-day span (2025-08-02 to 2026-07-29), meaning ~65 gap days. The preregistration explicitly accounts for this (65 known gaps, effective N estimated 140-200 trading-day pairs after excluding non-consecutive pairs). The calendar_rule specifies excluding non-consecutive pairs from lagged tests. However, the gap days are irregularly distributed and may cluster, which could further reduce effective N below the 140-200 estimate if gaps fall on consecutive trading days. If effective N falls below 120, H1 and H2 are to be downgraded to exploratory per the preregistration.
• [limitation] Market data starts 2025-08-04 (first trading day after the 2025-08-02 weekend) while Reddit data starts 2025-08-02 (Saturday). The inner join on U.S. equity trading days per the calendar_rule will naturally exclude the non-trading Reddit days at the start. Market data has 249 distinct trading dates; Reddit has 297 distinct source_dates. After inner-joining on trading days and excluding non-consecutive pairs for lagged tests, the effective N will be determined in S4. The preregistration's effective N floor of 120 must be verified before proceeding to Granger tests.
• [lesson:data] Reddit link-only posts (no body content) can account for ~40% of posts in finance subreddits. When preregistering sentiment analysis on Reddit data, explicitly account for title-only fallback in the sentiment computation plan, as VADER/LLM performance on short titles may differ from body text. Consider preregistering a sensitivity analysis comparing sentiment computed on title+content vs title-only.
• [limitation] LLM-based sentiment audit (audit_sentiment_subsample) could not execute: labeler 'kimi-k3' returned HTTP 402 (extra usage balance empty, non-retryable). The preregistration requires dual sentiment methods (VADER + LLM labeler) with a reliability audit gate. VADER sentiment is computed but UNVALIDATED — the audit gate could not run. All sentiment-based hypotheses (H1, H2, H3) are affected: they can only be tested with VADER-only sentiment, and the LLM cross-validation required by the preregistration is missing. This is a measurement gate failure, not a finding of inadequacy. Per preregistration, if the audit returns 'inadequate' the affected hypothesis is blocked; here the audit could not run at all, which is at least as severe. VADER-only results should be treated as exploratory/fragile for sentiment-dependent claims.
• [defer] agent deferred in S4 H1: DECISION: DEFER **What was accomplished:** - Described all source and derived datasets (42–48): Reddit per-post sentiment (ds44, 34,153 posts, 297 collection days), market OHLCV (ds43, 745 rows, 3 tickers SPY/QQQ/^VIX, 249 trading days), and their daily aggregations. - Recorded the mandatory `post_data_collection` checkpoint (reflection_id=9) citing real dataset IDs. - Aggregated daily Reddit sen
• [retrospective] S7 outcome=final; proposed 3 skill(s): ['preregister-reddit-sentiment-title-fallback', 'verify-external-validator-before-measurement', 'audit-gap-clustering-for-lagged-designs']. RETROSPECTIVE: This pilot revealed that Reddit's structural link-post architecture and external service fragility are the dominant failure modes for social-media sentiment studies. The 40% body-null rate and the LLM validator's HTTP 402 outage were both discoverable before measurement construction, yet both were encountered late, producing unvalidated sentiment and forcing deferral. Future studies should front-load data-structure audits and validator health checks before any derived measures are computed. SKILLS PROPOSED: preregister-reddit-sentiment-title-fallback, verify-external-validator-