Just Keep Prompting: Evaluating Repetitive Socratic Prompting in Vision-Language Models
A new evaluation framework, Just Keep Prompting (JKP), assesses the stability of vision-language models (VLMs) like GPT-4o, Gemini 2.5 Pro, and Qwen3-VL-30B under repeated multi-turn questioning. The study finds that repeated prompting often destabilizes model answers, leading to answer flipping and reduced reliability, rather than improving reasoning. Notably, GPT-4o exhibited the most instability, while Qwen3-VL-30B and Gemini 2.5 Pro showed different failure modes under pressure.
Why it matters: This work highlights a critical failure mode in VLMs, showing that sustained conversational pressure can undermine answer reliability, which is important for real-world deployment.
Full story at: arXiv Computation and Language ↗