Reddit sentiment lead-follow dynamics with market prices

Hypotheses: predicted → found

Datasets

Pre-registration amendments

• [2026-07-31 15:23:00] H2 · test_plan.calendar_rule — The preregistered LLM dry-run gate was specified to occur 'before S3 collection,' but the compute_sentiment tool (and all series/measurement tools) are blocked in S3 by protocol. The dry-run cannot be performed during the collection stage. Moving it to S4 before H2 analysis preserves the gate's intent (verifying LLM service availability before investing in concordance computation) while respecting stage boundaries. (stage S3)

Activity log (per-stage summaries — no raw model transcripts)

Limitations & lessons

• [limitation] r/stocks is completely absent from the collected Reddit data (dataset 70: 0 observations in the 2025-08-02 to 2026-06-29 window). The source profile confirms r/stocks was added late to the collector and has only 391 rows in the full archive (through 2026-07-29), likely concentrated in the final month outside our window. The preregistered exclusion rule ('exclude any subreddit with <30 observations') automatically handles this — the analysis proceeds with 4 subreddits (wallstreetbets, investing, stockmarket, economy) instead of 5. This reduces the breadth of the attention aggregate but does not invalidate the preregistered design.
• [limitation] Reddit content (body text) is null for 12,636 of 31,215 posts (40.5%) in dataset 70. These are predominantly link posts with empty bodies. The preregistered H2 text_fallback rule (use post title only for VADER scoring when body is empty) handles this, but the high proportion of title-only sentiment scores may reduce VADER's discriminative power, as titles are much shorter than body text. The preregistered sensitivity analysis (title+body vs title-only) will quantify this concern in S4.
• [limitation] The Reddit archive has ~62 gap days (270 distinct source_dates over a 362-day span, 2025-08-02 to 2026-06-29). The preregistered clean_subperiod (2025-08-02 to 2026-02-21) was designed to mitigate gap clustering in the later period. However, gap clustering within the clean_subperiod itself has not yet been verified — this will be checked in S4 when constructing the lag-1 consecutive-pair design. If gaps cluster within the clean_subperiod, the effective sample size after consecutive-pair exclusion may be substantially smaller than the raw day count suggests.
• [limitation] SPY market data (dataset 71) starts on 2025-08-04 (first trading day in the requested range; Aug 2-3 are weekend). The preregistered SPY log return calculation requires adjusted_close_{t-1}, meaning the first usable return observation is for trading day 2025-08-05. This is expected and does not affect the analysis window, but means the effective first lag-1 pair (attention_t -> |return_t+1|) cannot precede 2025-08-05.
• [limitation] H2 concordance gate BLOCKED: The LLM sentiment audit (audit_sentiment_subsample) failed with a 402 billing error — the labeler service's extra usage balance is empty and non-retryable. The preregistered concordance gate (Pearson r ≥ 0.50 between VADER daily mean and LLM daily mean) cannot be evaluated. Per the preregistration, H2 is suspended pending measurement resolution. VADER sentiment was computed (dataset 78) but remains UNVALIDATED. H1 and H3 use attention (post counts) and are NOT affected by this block.
• [lesson:process] The LLM sentiment audit billing gate can fail at S4 even if a dry-run passed at S1/S3 — billing balances deplete between stages. Always check remaining quota before the S4 audit call, and have a pre-registered fallback (e.g., dictionary-based concordance or human spot-check) if the LLM validator is unavailable.
• [limitation] CONCORDANCE GATE FAILURE: The preregistered concordance gate for H2 requires Pearson r >= 0.50 between VADER daily mean and LLM daily mean sentiment. The audit_sentiment_subsample tool (which provides LLM-vs-VADER concordance) failed with a billing error (HTTP 402: 'extra usage balance is empty'). The LLM labeler service is unavailable. Therefore the concordance gate could NOT be computed. Per the preregistration, 'if r < 0.50, H2 is suspended pending measurement resolution.' Since r cannot be computed at all, H2 results must be interpreted with this measurement-validation caveat: the VADER sentiment measure has NOT been externally validated against LLM labels in this study. The regression was still executed and recorded for auditability, but the sentiment measurement reliability is unverified. This is a binding limitation on any H2 claim.
• [lesson:process] LLM billing failures (HTTP 402) can block the concordance gate for sentiment validation. When a preregistered measurement-validation gate depends on an external LLM API, ensure billing/quota is verified with a minimal test call BEFORE the analysis stage. The audit_sentiment_subsample tool failed with 'extra usage balance is empty' — this is not a transient error and cannot be retried without adding billing credits. Consider preregistering a fallback validation method that does not require paid LLM APIs.
• [limitation] H1 analysis is limited by very low explanatory power (R² ≤ 0.019) and a small sample (132–176 trading-day pairs). The positive coefficient direction is consistent with the hypothesis, but the effect is economically negligible and statistically indistinguishable from zero. Additionally, the attention measure (aggregate daily post count) may be too coarse to capture meaningful attention shifts — a single high-volume day on one subreddit can dominate the z-score. The consecutive-pair exclusion reduced the sample from ~270 daily Reddit observations to 176 full-window pairs (132 clean), and gap clustering in the later period further constrained the clean subperiod.
• [defer] agent deferred in S4 interpretation: DECISION: DEFER What I would have needed to honestly complete this stage: 1. **The actual recorded outputs of analyses 21–29.** I never observed coefficient estimates, standard errors, confidence intervals, or corrected p-values from any of the regression runs (H1: analysis:23, 24, 27; H2: analysis:21; H3: analysis:25) or the Holm correction (analysis:29). Without these recorded numbers I cannot
• [limitation] The H3 reverse-path panel (dataset:99) was derived from dataset:90 which carries CP2=FAILED due to a date-range mismatch in the parent merge. While the derived clean subperiod has 132 rows matching the H1 clean subperiod exactly, the CP2 failure on the parent dataset introduces provenance uncertainty. Additionally, the H3 regression R² is near zero (0.007), meaning SPY absolute returns explain less than 1% of variation in next-day Reddit attention, and the HAC standard error is larger than the coefficient itself. The reverse-path relationship is not merely non-dominant — it is essentially absent in this sample.
• [retrospective] S7 outcome=final; proposed 2 skill(s): ['record-analysis-outputs-in-ledger-before-interpr', 'preregister-fallback-validation-for-paid-api']. RETROSPECTIVE: This pilot showed that methodological rigor in preregistration cannot compensate for execution gaps: analysis outputs from hypothesis episodes never reached the interpretation episode, forcing an unnecessary deferral, and a paid LLM validation gate failed with a billing error that had no preregistered fallback, suspending H2. Both failures were process-level, not scientific, yet they consumed 137 steps without yielding a conclusion. Future studies must ensure numerical results are explicitly recorded in a shared ledger before interpretation, and must preregister non-paid fallbac