Reddit sentiment lead-follow dynamics with market prices

Hypotheses: predicted → found

Datasets

Activity log (per-stage summaries — no raw model transcripts)

Limitations & lessons

• [limitation] Only 4 of the 5 preregistered subreddits have data in the collection window (dataset_id=25 shows 4 distinct subreddit values, not 5). The missing subreddit likely has zero posts in this date range. Reddit sentiment and post_volume measures are therefore aggregated over 4 subreddits instead of 5, which may slightly reduce coverage but does not change the analysis design.
• [limitation] Reddit content field is only 59.5% non-null (18,579 of 31,215 posts have non-null content). The sentiment measure uses concatenated title+content, so posts with null content will rely on title-only text. Titles are 97.3% non-null (30,330 distinct of 31,215), so the sentiment measure remains viable but may be noisier for posts lacking body text. This is a measurement quality limitation, not a design blocker.
• [limitation] Reddit archive has ~65 missing collection days across the 2025-08-02 to 2026-06-29 span (270 distinct source_dates out of ~333 calendar days in the window). The preregistration's calendar_rule requires inner join on dates where both Reddit has a collection entry AND markets are open, plus consecutive-day filtering for lagged designs. The ~65 gap days will reduce effective N below the estimated 180-190 trading days, especially for 3-lag Granger designs that require 3 consecutive days without gaps. The actual effective N will be determined in S4 after the inner join and consecutive-day filtering.
• [limitation] VIX dataset (dataset_id=27) has 229 rows vs SPY dataset (dataset_id=26) with 228 rows — a 1-row mismatch. This will be reconciled via inner join in S4 per the preregistered calendar_rule. The VIX volume column is all zeros (expected for an index, not a tradeable security).
• [limitation] SPY market data (dataset_id=26) has no 'adj_close' column because auto_adjust=True was used, making 'close' already split/dividend-adjusted. This is consistent with the preregistered fallback (preregister-yfinance-adj-close-fallback) which states 'close' is an acceptable substitute. SPY return = (close_t / close_{t-1}) - 1 will use the adjusted close column.
• [lesson:data] When collecting Reddit data across multiple subreddits, verify the actual distinct subreddit count in the returned data — some subreddits may have zero posts in the requested window, resulting in fewer distinct subreddits than requested. This does not fail CP1/CP2 but affects the representativeness of aggregate measures. Always check n_distinct on the subreddit column against the requested list.
• [limitation] Sentiment audit (audit_sentiment_subsample) could not be completed: the LLM labeler returned a billing error (402, extra usage balance empty). Per the preregistration, sentiment is UNVALIDATED until the audit returns 'adequate'. H1 and H3, which depend on validated sentiment, are blocked by this measurement gate. H2, which uses post volume (not sentiment), is unaffected.
• [limitation] Reddit dataset (ds:25) has 270 distinct source_dates across the 2025-08-02 to 2026-06-29 span (~358 calendar days), meaning ~88 calendar days have no collection entries. The preregistration estimates ~180-190 effective trading days after inner-joining with market data and applying consecutive-day lag requirements, but the actual intersection may be smaller due to the gap days falling on trading days. This reduces statistical power for all lagged designs (H1, H2, H3).
• [limitation] Dataset 25 shows only 4 distinct subreddits (n_distinct=4), not the 5 stated in the preregistration. One subreddit may have been unavailable or returned zero posts during the collection window. This affects the representativeness of the sentiment and volume measures.
• [limitation] Dataset 27 (VIX, 229 rows) has volume with n_distinct=1 (constant volume), consistent with VIX being an index. Dataset 26 (SPY, 228 rows) has one fewer row. The 1-row difference means the date intersection for H2's VIX model may lose one additional observation. Both datasets cover 2025-08-04 to 2026-06-30 (trading days only).
• [failure] infrastructure failure in S4 H1: LLM failure (reasoning_only, status=None, retryable=True): glm-5.2 returned only a private 'reasoning' field and no answer content — provider conformance failure (reasoning is never promoted to report content)
• [retrospective] S7 outcome=failed; proposed 0 skill(s): none. LLM failure (reasoning_only, status=None, retryable=True): kimi-k2.6 returned only a private 'reasoning' field and no answer content — provider conformance failure (reasoning is never promoted to report content)
• [retrospective] S7 outcome=failed; proposed 0 skill(s): none. LLM failure (reasoning_only, status=None, retryable=True): kimi-k2.6 returned only a private 'reasoning' field and no answer content — provider conformance failure (reasoning is never promoted to report content)
• [retrospective] S7 outcome=final; proposed 3 skill(s): ['verify-reddit-subreddit-count-post-collection', 'audit-text-coverage-before-sentiment', 'check-llm-audit-budget-before-gating']. RETROSPECTIVE: This pilot failed primarily due to an infrastructure cascade: a 402 billing error on the sentiment audit blocked H1 and H3, and a provider conformance failure (reasoning-only response) in S4 aborted the remaining analysis before any hypothesis could be tested. The study also revealed that Reddit data collection silently drops subreddits with zero posts and that text-field null rates are high, both of which degrade measurement quality but were only flagged late. Future pilots using LLM-gated audits or multi-subreddit social data should verify service budgets and post-collection c