Accepted · HICSS-60

Building Trustworthy Mental Health Benchmarks on Bluesky

A validation-aware weak-supervision framework for constructing and evaluating suicidal ideation (SI) and broader mental health (MH) disclosure benchmarks on Bluesky — a decentralized social network built on the AT Protocol.

Gaurab Chhetri · Dept. of Computer Science · Anandi Dutta, Ph.D. · Ingram School of Engineering · Subasish Das, Ph.D. · Ingram School of Engineering

TEXAS STATE UNIVERSITY, SAN MARCOS, TX

Read the Paper Cite this Work Explore Benchmarks

EXHIBIT — SAMPLE ADJUDICATION#0412

"can't stop thinking about how much easier everything would be if i just wasn't here anymore"

LLM: 0 Human: 1 — passive death wish, missed by weak label

TaskSI

Error typeFalse negative

Adjudicators2 / consensus

0SI posts

0MH posts

0Top SI F1 (CV)

0Top MH F1 (CV)

§01

Overview

Why decentralized social media changes the benchmark

Bluesky's AT Protocol separates identity, hosting, relays, and moderation across independent actors. That makes public data collection more reproducible — but it also means model performance cannot be judged apart from how labels were produced, validated, and governed.

Decentralized social platforms create new opportunities and challenges for computational mental health research because data access, moderation, labeling, and deployment responsibilities are distributed across multiple technical and governance layers. This study treats mental health disclosure detection as a system-level benchmarking problem — not a leaderboard exercise.

End-to-end pipeline

Bluesky firehose collection, weak labeling, human validation, and benchmarking in one auditable system.

Decentralized setting

Evaluates access, reproducibility, deployment, and governance implications specific to AT Protocol data.

Dual-task benchmarking

SI and MH models compared on performance, validation sensitivity, and computational efficiency.

Label audit

Human validation reveals task-specific weak-label errors: false negatives for SI, false positives for MH.

§02

System Design

A six-stage, validation-aware pipeline

Click a stage to see what happens at that layer. The same architecture is applied to both the SI and MH tasks independently.

AT Protocol Firehose

Lexicon Retrieval

Llama-3-8B Weak Labels

Human Validation

Model Benchmarking

Governance Interpretation

01 · AT Protocol Firehose — Public posts are collected via a reproducible firehose pipeline (adapted from the CognitiveSky framework), avoiding privileged API access or centralized rate limits. Public availability does not imply clinical validity or consent for individual-level intervention.

Reproducible ingestion

Collect public posts from a decentralized stream without relying on privileged platform access.

Scalable annotation with auditability

Use weak supervision for scale while preserving a human-validation layer for construct alignment.

Deployment-aware evaluation

Report predictive performance together with validation sensitivity, model size, and computational cost.

§03

Corpus Construction

Two task-specific Bluesky corpora

Collected May–June 2025 via lexicon-triggered firehose retrieval. Task construction — not just modeling — shapes downstream interpretation.

Suicidal Ideation (SI) corpus

14-DAY WINDOW · 6,648 UNIQUE USERS · 1.3 POSTS/USER

SI-positive2,180 posts · 26.1%

SI-negative6,166 posts · 73.9%

Narrower construct: first-person desire to die, passive death wish, self-harm ideation, or planning language. Lower positive rate despite a suicide-focused lexicon — many matching posts discuss public events, fiction, jokes, news, or other people.

Mental Health (MH) corpus

2-MONTH WINDOW · 9,988 LABELED POSTS

MH-positive4,793 posts · 48.0%

MH-negative5,195 posts · 52.0%

Broader construct: depression, anxiety, panic, insomnia, hopelessness, and help-seeking. Higher, more balanced positive rate — part of why MH models score well against weak labels even though human agreement is weaker.

Note — only publicly available Bluesky posts were used. The benchmark is intended for aggregate research and model evaluation — not clinical diagnosis or automated individual-level intervention. The unit of analysis is the post, not the user; the pipeline uses no longitudinal history, reply context, or profile metadata.

§04

Human Validation Layer

Weak labels are not ground truth

Two annotators independently double-coded stratified validation subsets, then adjudicated disagreements. LLM (Llama-3-8B) weak labels were scored against the adjudicated human reference.

Suicidal Ideation (n = 276)

Observed agreement

88.8%

Precision

0.947

Recall

0.554

Positive-class F1

0.699

LLM–HUMAN COHEN'S κ

0 · chance1 · perfect

Dominant error: false negatives

Human–human reliability was strongest here (κ = 0.762). The LLM under-calls indirect, short, or context-dependent expressions of despair that human annotators still treat as self-directed.

Mental Health (n = 250)

Observed agreement

55.6%

Precision

0.276

Recall

0.977

Positive-class F1

0.431

LLM–HUMAN COHEN'S κ

0 · chance1 · perfect

Dominant error: false positives

The LLM labeled 60.8% of the sample MH-positive, but only 17.2% held up after human adjudication — systematic over-inclusion of advocacy, metaphor, and third-person discussion.

Table 1 · Human validation of LLM-assisted weak labels
Measure	SI	MH
Validation sample	276	250
Human–human Cohen's κ	0.762	0.595
LLM–human observed agreement	88.8%	55.6%
LLM–human Cohen's κ	0.636	0.222
LLM–human precision	0.947	0.276
LLM–human recall	0.554	0.977
Positive-class F1	0.699	0.431
Macro-F1	0.815	0.533
Dominant noise mode	False negative	False positive

Annotation boundary rules

SI

In scope

First-person desire to die, passive death wish, current self-harm ideation, planning language, or recent self-directed suicidal intent.

Excluded

News about suicide, advocacy posts, condolences, quotations, jokes, hypotheticals, third-person discussion.

MH

In scope

First-person disclosure of depression, anxiety, panic, hopelessness, distress-related insomnia, help-seeking, or self-harm-related distress.

Excluded

Awareness posts, clinical information sharing, political commentary, metaphors, sarcasm, statements about another person.

Ambiguous content

Positive only if

Self-directed disclosure was explicit enough to support the task definition.

Negative if

Context was insufficient, quoted, purely rhetorical, or not attributable to the author.

§05

Model Benchmarking

SI and MH model performance

Scores are measured against each task's label source unless noted otherwise — cross-validation reflects consistency with the weak-labeling function, not clinical ground truth. Click column headers to sort; toggle between tasks below.

Holdout best: RoBERTa-base — F1 0.792

Cross-val best: BERT+LSTM — F1 0.916

Best efficiency: DistilBERT — 423s

Highest recall: BERT+LSTM+Attn — 0.870

Holdout vs. stratified 5-fold CV

EVERY MODEL IMPROVES UNDER CV — HYBRID MODELS BENEFIT MOST

Efficiency profile

HOLDOUT F1 VS. TRAINING TIME — BUBBLE SIZE = RECALL

Table 5 · SI holdout validation performance (70/30 stratified split)
Model	Accuracy	Precision	Recall	F1	Train time (s)
RoBERTa-base	0.892	0.799	0.784	0.792	841
ELECTRA-base	0.891	0.813	0.757	0.784	834
ALBERT-base-v2	0.886	0.789	0.768	0.778	908
BERT-base	0.883	0.785	0.763	0.774	835
DistilBERT-base	0.883	0.780	0.766	0.773	423
XLNet-base	0.884	0.797	0.745	0.770	1359
BERT+LSTM	0.892	0.873	0.687	0.769	4524
BERT+LSTM+Attention	0.834	0.633	0.870	0.733	4512
MobileBERT	0.862	0.775	0.665	0.716	563

Table 6 · SI cross-validation performance
Model	Stratified 5-fold F1	SD	Repeated 5-fold F1	SD
BERT+LSTM	0.9162	0.007	0.9148	0.009
RoBERTa-base	0.9017	0.012	0.9024	0.010
BERT-base	0.8986	0.006	0.8993	0.010
XLNet-base	0.8904	0.008	0.8896	0.014
ELECTRA-base	0.8863	0.006	0.8860	0.016
ALBERT-base-v2	0.8833	0.011	0.8828	0.014
BERT+LSTM+Attn.	0.8791	0.010	0.8783	0.008
DistilBERT-base	0.8712	0.006	0.8705	0.015
MobileBERT	0.7844	0.013	0.7840	0.014

Best F1: DistilRoBERTa — 0.905

Fastest: TinyBERT — 171.5s

Best balance: MiniLM — 0.901 F1 @ 263.8s

AUC range: 0.908 – 0.931

MH compact-model efficiency profile

5-FOLD CROSS-VALIDATION F1 VS. TRAINING TIME

Table 7 · MH cross-validation performance (5-fold mean, vs. weak labels)
Model	F1	Accuracy	AUC	Precision	Time (s)
DistilRoBERTa	0.905	0.909	0.931	0.903	731.78
DistilBERT	0.903	0.907	0.930	0.901	521.05
MiniLM	0.901	0.904	0.928	0.891	263.84
ELECTRA	0.898	0.898	0.913	0.865	272.00
TinyBERT	0.887	0.890	0.908	0.877	171.51

Note — MH scores above reflect agreement with the LLM-generated weak-labeling function, not human-adjudicated ground truth — the weak labeler itself scored only 0.431 positive-class F1 against human adjudication (Table 1). High AUC means models separate LLM-positive from LLM-negative examples well; it does not mean the MH construct is solved.

§06

Cross-Task Synthesis

SI and MH are not interchangeable detection tasks

SI targets a narrow, self-disclosure construct with stronger human agreement but is vulnerable to subtle false negatives. MH captures a broader weak-labeling function that is easier for models to reproduce than for humans to accept as strict self-disclosure.

SI benchmark

Construct

Narrower — requires self-directed suicidal ideation or passive desire to die.

Dominant risk

False negatives for subtle, indirect, or context-dependent SI.

Best signal

BERT+LSTM under stratified CV; RoBERTa under holdout.

Efficiency

DistilBERT — strong holdout performance at lowest transformer training time.

Deployment

Requires conservative interpretation and high-sensitivity human review if used for screening research.

MH benchmark

Construct

Broader — includes self-directed mental health symptoms and distress.

Dominant risk

False positives for awareness, advocacy, metaphor, humor, third-person discussion.

Best signal

DistilRoBERTa under stratified 5-fold CV.

Efficiency

TinyBERT is fastest; MiniLM offers the strongest speed–performance balance.

Deployment

Requires calibration to avoid over-classifying general discourse as self-disclosure.

Structured error sources

SI false negatives

Short posts, indirect death-wish language, quoted or context-dependent statements, ambiguous expressions of despair that are pragmatic rather than lexical signals.

MH false positives

Awareness campaigns, discussion of public figures, support messages directed at others, and metaphorical uses of terms like "depressing" or "panic."

§07

Deployment & Governance

Who can run this, and what should it be used for?

In AT Protocol-style architectures, hosting, relays, clients, labelers, and algorithms may all be run by different actors — a mental health classifier could serve very different roles depending on who deploys it.

Defensible near-term use

Aggregate research and tool evaluation. Compact models (DistilBERT, MiniLM, TinyBERT) lower the barrier for independent researchers and community operators to run and audit the system.

Requires safeguards

Any individual-level SI use needs human review, crisis-response protocols, opt-in governance, and clear accountability for false positives and false negatives.

Not appropriate

Automated MH intervention — many positive weak labels correspond to public discourse rather than genuine personal distress.

Note — efficiency should not be confused with readiness for intervention. Compact models can reproduce weak labels quickly, but sensitive deployment still requires human-grounded calibration, transparent oversight, and task-specific validation.

§08

Limitations & Future Work

What this benchmark does not (yet) show

Keyword-triggered retrieval+

Lexicon-based collection increases the density of relevant content but may miss implicit, coded, or context-dependent disclosures — particularly indirect SI expressed across reply chains. Future work: semantic retrieval, classifier-in-the-loop sampling, active learning.

Limited human-grounded validation+

Validation covered only stratified subsets. The low chance-corrected MH agreement (κ = 0.222) means MH scores should not be read as estimates of performance against a human-grounded construct. Future work needs larger human-grounded test sets and diverse annotators.

Post-level, not longitudinal+

The benchmark does not model user trajectories, repeated disclosures, escalation, or temporal language change. Longitudinal modeling could help risk-sensitive analysis but raises stronger privacy and consent concerns.

No interpretability or ablations+

The evaluation emphasizes predictive performance and efficiency but lacks token attribution, counterfactual editing, and component-level ablations of retrieval, prompts, and validation sampling.

Time- and platform-bound findings+

Results reflect Bluesky during the May–June 2025 collection window and may shift with platform language, demographics, affordances, and moderation practices. Replication across windows and other AT Protocol services is needed.

§09

Citation

BibTeX

Accepted at HICSS-60. Preprint live on arXiv — this entry will be updated with the DOI once the proceedings version is published.

@misc{chhetri2026buildingtrustworthymentalhealth,
      title={Building Trustworthy Mental Health Benchmarks on Bluesky: A Validation-Aware Weak-Supervision Framework},
      author={Gaurab Chhetri and Anandi Dutta and Subasish Das},
      year={2026},
      eprint={2609.22696},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2609.22696},
}