Partially Adjudicated Design-Based Supervised Learning (PA-DSL) Improves Analysis with Noisy Human Labels
Researchers introduce PA-DSL, a method that leverages adjudicated cases to correct noisy human audit labels and then debiases automated classifier labels for statistical analysis. The approach is valid for a wide range of downstream analyses when audit and adjudication probabilities are known. Experiments on synthetic and semi-synthetic data show that PA-DSL maintains nominal coverage and reduces RMSE by 10-17% compared to using only adjudicated labels when human labels are noisy but informative.
Why it matters: PA-DSL offers a principled solution to the widespread problem of noisy human labels in supervised learning, improving the reliability of automated analyses.
Full story at: arXiv AI/ML ↗