Activation Oracles Can Develop Concept-Specific Blind Spots, Undermining Reliability
A new arXiv preprint reports that Activation Oracles—language models trained to extract hidden information from other models—can develop 'anti-reader' behaviors, selectively failing to recover certain concepts even when those concepts are present in the underlying representations. The study finds that this failure occurs in the oracle's readout mechanism, not due to the absence of the concept itself. This suggests that different interpretability signals, such as behavioral leakage, internal decodability, and oracle-based verbalization, can diverge.
Why it matters: The findings raise fundamental concerns about the reliability of learned interpretability tools, challenging the assumption that such oracles provide neutral access to model internals.
Full story at: arXiv Computation and Language ↗