Distribution-Conditioned Attribute Selection Improves Interpretable Zero-Shot Classification
A new study finds that class descriptors generated by large language models (LLMs) for zero-shot image classification often lack visual evidence, causing ImageNet accuracy to drop from 59.5% to 15.5% when class names are omitted. The researchers propose a method that selects attributes directly from target images using CLIP's embedding space, achieving 23.8% accuracy without class names and outperforming prompt-tuning methods with significantly less computation.
Why it matters: This work exposes a key limitation in LLM-based descriptor generation for vision-language models and introduces a more robust, interpretable, and efficient alternative.
Full story at: arXiv Computer Vision ↗