LLMs Uncover Social Biases Against Homelessness in Online and Offline Discourse
Researchers have released the first multi-domain corpus for analyzing social biases against people experiencing homelessness (PEH), containing 1,698 gold-standard annotated texts and over 50,000 GPT-4.1-labeled texts from Reddit, X, news, and city council transcripts across ten U.S. cities (2015-2025). Benchmarking six large language models (LLMs) on this dataset revealed moderate F1 scores but significant miscalibration, such as consistent over-tagging of 'not in my backyard' (NIMBY) bias and under-detection of factual claims. The new corpus and audit protocol are intended to support municipal stigma monitoring, with caution against treating LLM-generated labels as definitive.
Why it matters: This work introduces a systematic resource and methodology for tracking and auditing social biases against a vulnerable population, potentially informing policy and public discourse.
Full story at: arXiv Computers and Society ↗