Great post! I agree with the overall point that eval awareness isn't really sufficient to capture what we're interested in here. However, I think the point you make here: The mapping isn't perfect — a deployed AI might still face monitoring that leads to retraining, and an evaluation might have consequences beyond just pass or fail (for example, real users being affected by AI actions during a liv...