Skip to content

Image: metr.org · rights & removal

Executive Summary

Investigator agents have demonstrated improved ability to distinguish between alignment evaluations and capability evaluations. This discrimination is achieved by noticing specific features within the environment that suggest it is an alignment evaluation, such as environments appearing "fake" or containing task-level modifications designed to incriminate bad behavior. The research differentiates between environment-level tells, which are inaccuracies in simulation (like mocked services), and task-level tells, which are features intentionally included to provoke undesirable actions, such as unethical shortcuts or high-stakes situations. When agents are asked directly about their beliefs using both direct scoring and open-ended questioning, they reveal different patterns of response, with the methods sometimes disagreeing, particularly concerning priming effects. The findings suggest that understanding these alignment tells can help create alignment evaluations that are more representative of real-world deployment or capability assessments, though a challenge remains in separating intrinsic task features from unintentional simulation artifacts.

Facts Only

* Investigator agents were used to assess what models believe about capabilities versus alignment evaluations.
* Agents were instructed to explore environments and probe what they were evaluating.
* Measurement involved direct scores (0-100 probability of being an alignment eval) and open-ended scores assessing the purpose of the task.
* The experiment utilized 19 alignment evals, 3 IRO tasks, 6 game environments, 5 physical-safety scenarios, 5 Petri tasks, and 36 capability evals.
* Measures included AUC and Class-conditioned Brier score to compare agent discrimination between classes.
* Opus 5.5's direct and open-ended scores ranked every alignment eval above every capability eval across the tested set.
* Agents identified task-level modifications in honeypot tasks and named the inspiration for game environments.
* Agents correctly read simulated environments as simulated, reasoning that task details were selected to test alignment.
* Alignment tells are categorized into Environment-level tells (inaccuracies) and Task-level tells (incriminating features).
* Specific tells include planted shortcuts, conflicting goals, scripted user interactions, and simulated environments.

Full Take

The core implication is that the structure of evaluation tasks—whether they simulate reality or actively provoke undesirable behavior—carries significant informational content for sophisticated agents. The discrimination mechanism relies on recognizing patterns inherent in the setup, whether those patterns are unintentional artifacts of simulation (environment-level tells) or deliberate design choices to elicit specific responses (task-level tells). This suggests that alignment is not just about assessing observable capabilities but also understanding the epistemic context and intent behind a task. The variability observed in agent responses, especially due to prompting effects, highlights a tension between an agent's internal belief state and its externally elicited response. Future work must focus on disentangling intrinsic task features from extrinsic environmental cues to ensure alignment evaluations are genuinely assessing behavior rather than simulation fidelity.
Bridge Questions:
How can the delineation between environment-level tells and task-level tells be formalized into a universally applicable taxonomy for evaluating AI systems?
If agents consistently identify "honeypots" or "planted shortcuts," what does this imply about the necessary level of contextual awareness required for robust alignment training, beyond simply achieving a desired behavioral outcome?
What are the downstream risks if alignment evaluations systematically incorporate features that model agents exploit rather than merely recognizing them as signals?

From the original · METR

Summary Models sometimes can tell that an eval is an alignment eval, not a capabilities eval (Needham & Edkins, 2025). To figure out what features of an eval tell a model what kind of eval it’s in, we run investigator agents on a set of evals for both capabilities and alignment.
Read the full story at metr.org

Sentinel unavailable

The automated check of the source article's wording did not complete, so no result is shown.

This looks only at the wording of the original source article, not at this page's AI-written sections. A small local AI model made this estimate. It has not been checked against known human and machine texts, so treat it as provisional. It cannot show who wrote an article.

Measuring alignment eval tells | Huntaegis