For example: If you ask an AGI for an ASI alignment plan, and its ruthless goal is something like “the humans will be pleased with my answer in the short-term”, then it will be trying to please and convince you (at best), not to be scrupulously correct, and we have abundant real-world evidence that people can be convinced by incorrect alignment plans (otherwise we wouldn’t be in a situation where ...