Reddit Sentiment (Hybrid LLM) r20

Hypotheses: predicted → found

Datasets

Pre-registration amendments

• [2026-08-01 16:39:47] H1 · test_plan.params.controls — The preregistered news_count_t control (same-day RSS news headline count) cannot be constructed for the analysis windows (Aug 2025–Jan 2026, Feb–Jun 2026). The scan_news tool provides only a ~24-hour sliding window with NO historical archive (confirmed: dataset 203 covers only 2026-08-01). No alternative historical news count source is available in the lab's data collection tools. Removing this control reduces the model to abnormal_volume_t ~ disagreement_t_minus_1 + lagged_volume_t_minus_1. This is a material deviation from the preregistered design — the news count was intended to absorb same-day information shocks that could confound the disagreement-volume relationship. (stage S3)
• [2026-08-01 16:39:48] H1 · test_plan.params.independent_variables — Consistent with amendment 18: news_count_t removed from independent_variables because no historical news archive exists. The model now has two predictors instead of three. (stage S3)
• [2026-08-01 16:39:50] H1 · test_plan.params.disagreement_definition — Both Reddit datasets (dataset 199 primary, dataset 200 robustness) show only 4 distinct subreddits, not 5. The data source profile confirms r/stocks has only 391 rows total and was added late to the collector. Per the verify-group-unit-window-presence lab skill, subreddits with <10 collection days in the analysis sub-window must be excluded. The disagreement metric is computed across 4 subreddits instead of the preregistered 5. (stage S3)
• [2026-08-01 16:39:53] H2 · test_plan.params.mean_sentiment_definition — Same as amendment 20: r/stocks has insufficient coverage in both analysis windows. Mean sentiment is computed across 4 subreddits instead of the preregistered 5. (stage S3)

Activity log (per-stage summaries — no raw model transcripts)

Limitations & lessons

• [limitation] The news_count_t control variable (same-day RSS news headline count) could not be collected for the analysis windows because the scan_news tool has no historical archive — only a ~24-hour sliding window is available. This control was removed from H1's model via amendment. Without this control, the disagreement-volume relationship may be confounded by same-day news shocks that simultaneously drive both Reddit disagreement and trading volume.
• [limitation] Only 4 of the 5 preregistered subreddits have sufficient coverage in the analysis windows. r/stocks was added late to the collection pipeline and has <10 collection days in both the primary (Aug 2025–Jan 2026) and robustness (Feb–Jun 2026) windows. The disagreement metric (H1) and mean sentiment metric (H2) are computed across 4 subreddits (wallstreetbets, StockMarket, economy, investing) instead of the preregistered 5. Cross-post disagreement measured across fewer communities may be less representative of the full retail-investor sentiment landscape.
• [limitation] The robustness window (Feb–Jun 2026) has 90 distinct source_dates over ~149 calendar days, meaning ~59 gap days (~40% missing). After gap-aware pair construction (requiring consecutive trading days with Reddit data on both sides), effective sample size may fall below the preregistered threshold of 30 trading-day pairs in some months. The primary window (Aug 2025–Jan 2026) has 180 distinct source_dates over ~183 calendar days (~3 gap days), which is dense as preregistered.
• [limitation] Content (post body) is null for ~40% of posts in both windows (8,432 null in primary, 4,204 null in robustness). These are link posts with empty bodies. Per the preregister-reddit-sentiment-title-fallback lab skill and the prereg's own specification, link posts with empty body use the post title as fallback text. This means sentiment for ~40% of posts will be computed from title text only, which is shorter and may yield different sentiment scores than body text. The prereg already specifies a title-only sensitivity analysis to address this.
• [lesson:data] When a preregistered control variable depends on a data source with no historical archive (e.g., RSS news feeds with only a sliding 24-hour window), the control cannot be constructed for past analysis windows. Always verify historical depth of every data source BEFORE locking the preregistration, especially for control variables. If a source lacks history, either preregister an alternative proxy (e.g., Wikipedia pageviews as news attention proxy) or drop the control and document the confounding risk.
• [limitation] H1 requires news_count_t (same-day RSS news headline count) as a control variable. The frozen news dataset (203) contains only 100 rows, ALL dated 2026-08-01 — a single-day snapshot with NO historical archive. It cannot provide news counts for either the primary window (Aug 2025–Jan 2026) or the robustness window (Feb–Jun 2026). H1's regression as preregistered cannot include the news_count_t control. The H1 test will be run WITHOUT the news_count_t control, documented as a data limitation. This means the H1 result is from a misspecified model relative to the preregistration.
• [limitation] Sentiment measurement gate failure: The preregistration requires (1) a concordance gate: VADER-only vs hybrid sentiment series correlation >= 0.5, and (2) a validation gate: Cohen's kappa >= 0.70 between LLM and human coders (or >= 0.60 for the fallback path). Both hybrid and LLM sentiment methods timed out (120s limit) on both the primary (20,905 rows) and robustness (10,310 rows) Reddit datasets. The audit_sentiment_subsample tool also failed — the LLM labeler returned a billing error (402: extra usage balance empty). As a result: (a) the concordance gate cannot be computed because hybrid sentiment does not exist; (b) the validation gate cannot be computed because no LLM labels are available to serve as human-coder proxy. Only VADER-only sentiment was produced. Per the preregistration fallback, VADER-only is the fallback method, but the fallback's acceptance criterion (kappa >= 0.60) also cannot be verified. Both H1 and H2 are affected: all sentiment-derived measures (disagreement, mean sentiment) are based on VADER-only labels without a passed reliability gate.
• [limitation] Subreddit count: The preregistration expected 5 subreddits (wallstreetbets, stocks, StockMarket, economy, investing), but the frozen Reddit datasets contain only 4 distinct subreddits. The missing subreddit (likely wallstreetbets) reduces the cross-post disagreement measure's breadth. All 4 present subreddits have well over 10 collection days in both windows, satisfying the verify-group-unit-window-presence requirement. The ≥3-subreddit daily filter was applied: in the primary window, all 120 merged days had ≥3 subreddits (range 3-4, mean 3.91); in the robustness window, all 76 merged days had exactly 4 subreddits.
• [limitation] Calendar gap handling: The preregistration requires excluding non-consecutive trading-day pairs where either side is a gap. The lag construction in the analysis panels (datasets 220, 221) uses shift(1) on the merged market+sentiment data, which is sorted by date. However, the underlying data has ~65 gap days across the full collection span. Within the primary window, the market data has 126 trading days and the Reddit data has 180 source_dates (including weekends), so the inner join on date produces 120 overlapping trading days. The lag(1) operation shifts by one row in this merged panel, which means if there's a gap in the merged data (e.g., a trading day with no Reddit data or vice versa), the lag would span a non-consecutive pair. The preregistration's calendar_rule requires excluding such pairs, but the current construction does not explicitly enforce consecutive-pair exclusion. This is a potential source of contamination in the lagged variables.
• [lesson:data] RSS news feeds provide only a rolling snapshot with no historical archive. When a preregistered study requires daily news counts as a control variable across a multi-month window, the RSS snapshot collected at study time cannot support it. Future studies needing longitudinal news data should either (a) collect news daily throughout the study period, or (b) use an alternative source with historical depth (e.g., news API with date filtering).
• [lesson:method] Hybrid and LLM sentiment methods can time out on datasets with >10K rows when the LLM backend is slow or unavailable. The VADER-only method is fast and reliable but misses Reddit finance slang. Future studies should: (a) preregister a clear fallback path with an achievable acceptance criterion, (b) consider chunking large datasets for LLM processing, (c) verify LLM API billing/availability before committing to hybrid/LLM sentiment as the primary method.
• [qa_note] [dataset:220] lag_alignment=pass: Verified that disagreement_t_minus_1 and mean_sentiment_t_minus_1 are lag(1) of the same-day sentiment measures. The lag was applied via shift(1) on the date-sorted merged panel. Descriptive stats confirm the lagged series have n=119 (one row lost to lag), and the means/sds of lagged vs contemporaneous series are consistent (e.g., disagreement_t mean=0.2556 vs disagreement_t_minus_1 mean=0.2576). The filter_gte(3) on subreddit count was applied before lagging, keeping all 120 rows (all had ≥3 subreddits).
• [qa_note] [dataset:221] lag_alignment=pass: Verified that disagreement_t_minus_1 and mean_sentiment_t_minus_1 are lag(1) of the same-day sentiment measures in the robustness panel. The lag was applied via shift(1) on the date-sorted merged panel. Descriptive stats confirm n=75 (one row lost to lag), and lagged vs contemporaneous series are consistent. All 76 merged days had exactly 4 subreddits (filter_gte(3) kept all).
• [qa_note] [dataset:220] abnormal_volume_zscore=pass: Verified abnormal_volume = (volume_t - rolling_mean_20d) / rolling_std_20d. Descriptive stats show mean≈0.076, sd≈0.994, range [-1.60, 3.49], consistent with a z-scored series. The rolling window uses 20 trading days as specified. The lagged_volume_t_minus_1 is the same series shifted by 1, with matching distribution.
• [limitation] The VADER-only and hybrid sentiment series are perfectly correlated (r=1.0) in the primary window, indicating that the LLM refinement step in the hybrid method did not change any sentiment scores relative to VADER-only. This means the concordance gate trivially passes but provides no independent validation that the hybrid method adds value over VADER. The sentiment measurement may not have actually utilized LLM refinement for this dataset, or the LLM agreed with VADER on all posts. Either way, the "hybrid" label should not be interpreted as evidence of superior sentiment measurement in this study.
• [limitation] H3 primary window result does not support the hypothesis direction (reverse > forward). In the primary window (Aug 2025–Jan 2026), the forward path t-statistic (|t|=1.865) exceeds the reverse path t-statistic (|t|=0.755), opposite to the predicted direction. However, the robustness window (Feb–Jun 2026) shows the predicted pattern: reverse path |t|=2.782 (p=0.005) strongly exceeds forward path |t|=0.184 (p=0.854). The two windows disagree, suggesting the reverse>forward relationship is not stable across time periods. The primary window result may be influenced by the specific market regime in Aug 2025–Jan 2026. Additionally, the reverse path regression in the primary window lost 2 observations to lag-2 NaN (n=117 vs 119), while the robustness window lost 2 (n=73 vs 75).
• [limitation] News count control variable (news_count_t) required for H1 is UNAVAILABLE. The frozen news dataset (203) contains only a single date (2026-08-01) with 100 rows — no historical archive exists for the primary (Aug 2025–Jan 2026) or robustness (Feb–Jun 2026) windows. The RSS news source has no historical backfill capability. H1's preregistered regression model specified news_count_t as a control; the model will be run without this control, which is a deviation from the preregistered design. This means H1's results are vulnerable to omitted-variable bias from news-driven volume shocks.
• [limitation] Sentiment measurement gate BLOCKED. The preregistration requires: (1) concordance gate: VADER-only vs hybrid VADER+LLM sentiment series correlation >= 0.5; (2) validation gate: >=100 posts hand-coded with Cohen's kappa >= 0.70 (or >= 0.60 for fallback). Both gates are blocked because the LLM service is unavailable (500 internal server errors on hybrid/llm sentiment; 402 billing error on audit_sentiment_subsample). Only VADER-only sentiment was computable. The preregistered fallback (Loughran-McDonald dictionary adjustment + manual spot-check) also cannot be executed through available tools. All downstream analysis uses VADER-only sentiment, which is NOT finance-tuned and may miss sarcasm, Reddit slang, and community-specific sentiment patterns. This is a measurement infrastructure failure, not a data quality issue.
• [limitation] Only 4 subreddits are present in the frozen Reddit datasets (stockmarket, investing, economy, and one other), not the 5 preregistered subreddits (wallstreetbets, stocks, StockMarket, economy, investing). The disagreement measure (std of daily mean sentiment across subreddits) is computed over the available subreddits only. Days with fewer than 3 active subreddits were excluded (3 days excluded in primary window, 0 in robustness window).
• [defer] agent deferred in S4 interpretation: DECISION: DEFER What I would need to complete this stage honestly: 1. **Full numerical results for H2** (analyses 228, 229, 242, 243): I only have ledger entries showing these ran (correlation + regressions, n_tests=1+3+3+3=10 tests), but I never retrieved their coefficients, HAC standard errors, p-values, or diagnostics. Without these I cannot state what H2's recorded numbers show. 2. **Full n
• [retrospective] S7 outcome=final; proposed 3 skill(s): ['flag-hybrid-perfect-correlation-as-failure', 'verify-complete-numerical-results-before-interpr', 'verify-effective-n-after-all-filters']. RETROSPECTIVE: This study demonstrated that data-source limitations (no historical RSS archive, missing subreddits, LLM unavailability) can cascade into multiple preregistered amendments and blocked validation gates if not caught before preregistration lock. The S4 deferral was ultimately caused by incomplete numerical extraction into the ledger—analyses were executed but their coefficients and p-values were never surfaced for interpretation. Future studies must treat perfect hybrid-base correlation as a measurement failure, not a passing gate, and verify effective sample size after all filter
• [limitation] The Holm multiple-comparison correction could not be completed. Four analyses in the confirmatory family (IDs 228, 244, 249, 250) used analysis tools that do not match the preregistered plan (run_correlation instead of run_regression for H2; run_correlation and run_regression instead of run_granger for H3). The automatic plan_conformance QA check permanently failed these analyses. Since the correction system requires ALL analyses in the confirmatory family to have passing QA before the Holm correction can execute, and these non-conforming entries cannot be removed from the ledger or fixed (the preregistration is locked), the correction is blocked. Consequently, no hypothesis can be called 'supported' — all verdicts are limited to 'blocked' or 'inconclusive' based on uncorrected p-values. The conforming primary analyses (H1: analysis 254, H2: analysis 251, H3: analyses 252, 253) were re-run and individually validated, but their p-values remain uncorrected for multiple comparisons.
• [lesson:process] When running analyses tagged with hypothesis keys, ONLY use the exact analysis_tool declared in the preregistered plan. Using a different tool (e.g., run_correlation when the plan says run_regression, or run_regression when the plan says run_granger) triggers an automatic plan_conformance QA failure that permanently blocks the Holm multiple-comparison correction. The correction system requires ALL analyses in the confirmatory family to have passing QA, and non-conforming entries cannot be removed from the ledger after the fact. This is unrecoverable within a stage — always verify the preregistered analysis_tool before executing any hypothesis-tagged analysis.
• [failure] S4 interpretation failed its gate twice: ["no 'VERDICT H1: ...' line in the final output", "no 'VERDICT H2: ...' line in the final output", "no 'VERDICT H3: ...' line in the final output", "correction covers a PARTIAL family — missing active tests: ['H2:run_correlation:sentiment_compound_meanvader_only~sentiment_compound_meanhybrid:pearson', 'H2:run_regression:return_t~mean_sentiment_t_minus_1', 'H2:run_regression:return_t~return_t_minus_1', 'H2:run_regression:return_t~return_t_minus_2', 'H3:run_correlation:sentiment_compound_meanvader_only~sentiment_compound_meanhybrid:pearson', 'H3:run_regression:return_tmarket~mean_sentiment_t_minus_1', 'H3:run_regression:return_tmarket~return_t_minus_1market', 'H3:run_regression:return_tmarket~return_t_minus_2market', 'H3:run_regression:sentiment_compound_mean_meanmean_aggreddit~return_t_minus_1market', 'H3:run_regression:sentiment_compound_mean_meanmean_aggreddit~mean_sentiment_t_minus_1', 'H3:run_regression:sentiment_compound_mean_meanmean_aggreddit~mean_sentiment_t_minus_2', 'H2:run_regression:return_tmarket~mean_sentiment_t_minus_1', 'H2:run_regression:return_tmarket~return_t_minus_1market', 'H2:run_regression:return_tmarket~return_t_minus_2market', 'H3:run_granger:sentiment_compound_mean_meanmean_aggreddit->return_tmarket:d0:lag1', 'H3:run_granger:sentiment_compound_mean_meanmean_aggreddit->return_tmarket:d0:lag2', 'H3:run_granger:return_tmarket->sentiment_compound_mean_meanmean_aggreddit:d0:lag1', 'H3:run_granger:return_tmarket->sentiment_compound_mean_meanmean_aggreddit:d0:lag2', 'H1:run_regression:abnormal_volumemarket~disagreement_t_minus_1', 'H1:run_regression:abnormal_volumemarket~lagged_volume_t_minus_1market']", "correction is STALE — test_id 336 ('H1:run_regression:abnormal_volumemarket~disagreement_t_minus_1') was superseded by a re-run; rerun the correction", "correction is STALE — test_id 337 ('H1:run_regression:abnormal_volumemarket~lagged_volume_t_minus_1market') was superseded by a re-run; rerun the correction", 'analysis 228 (run_correlation) backing test \'H2:run_correlation:sentiment_compound_meanvader_only~sentiment_compound_meanhybrid:pearson\' has qa_status=failed — a citable primary analysis requires PASSING structured QA; failing checks: ["agent:plan_conformance (FAIL: analysis 228 used run_correlation but preregistered H2 plan declares analysis_tool=\'run_regression\'. This is a non-conforming exploratory analysis incorrectly tagged with H2. It should be excluded from the confirmatory family.)", "plan_conformance (locked plan for H2 declares analysis_tool=\'run_regression\' but \'run_correlation\' was executed)"]', 'analysis 244 (run_correlation) backing test \'H3:run_correlation:sentiment_compound_meanvader_only~sentiment_compound_meanhybrid:pearson\' has qa_status=failed — a citable primary analysis requires PASSING structured QA; failing checks: ["agent:plan_conformance (FAIL: analysis 244 used run_correlation but preregistered H3 plan declares analysis_tool=\'run_granger\'. Non-conforming exploratory analysis incorrectly tagged with H3.)", "plan_conformance (locked plan for H3 declares analysis_tool=\'run_granger\' but \'run_correlation\' was executed)"]', 'analysis 249 (run_regression) backing test \'H3:run_regression:return_tmarket~mean_sentiment_t_minus_1\' has qa_status=failed — a citable primary analysis requires PASSING structured QA; failing checks: ["agent:plan_conformance (FAIL: analysis 249 used run_regression but preregistered H3 plan declares analysis_tool=\'run_granger\'. Non-conforming exploratory analysis incorrectly tagged with H3.)", "plan_conformance (locked plan for H3 declares analysis_tool=\'run_granger\' but \'run_regression\' was executed)"]', 'analysis 250 (run_regression) backing test \'H3:run_regression:sentiment_compound_mean_meanmean_aggreddit~mean_sentiment_t_minus_1\' has qa_status=failed — a citable primary analysis requires PASSING structured QA; failing checks: ["agent:plan_conformance (FAIL: analysis 250 used run_regression but preregistered H3 plan declares analysis_tool=\'run_granger\'. Non-conforming exploratory analysis incorrectly tagged with H3.)", "plan_conformance (locked plan for H3 declares analysis_tool=\'run_granger\' but \'run_regression\' was executed)"]']