知情的造神者

我今天从 Anthropic 辞职了。过去三年我在 OpenAI 和 Anthropic 做预训练（pretraining）研究。两家公司都没有在负责任地行事。它们正在直线竞速冲向自我改进的超级智能，拿我们的生命做赌注。更多想法见下。

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.

不要低估这项技术的力量。这些系统很快就能入侵任何东西、一夜之间革新任何领域、并获取真实的权力和资源。我们每个人都见证了这些领域的进展，而且进展没有放缓。

Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.

造 AI 的人是真心相信这东西可能在本年代末杀死我们所有人。这不是营销话术。恰恰相反——很多高管和资深研究员在媒体上会把措辞收敛得体面，但我听到同样的人在私下表达恐惧。没有任何其他人类活动构成这个量级的危险。

The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.

一个常见的反应是："如果他们真信这个，为什么还在造？"在 OpenAI，很多人并没有真正内化这场文明级的赌注。在 Anthropic，赌注被充分理解了，但他们被困在一场"必须第一个到终点"的竞赛里——他们相信没有别人会负责任地做这件事，所以必须自己来，哪怕冒着风险。

A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.

接受这场竞赛、进入"残局"，是一个傲慢的豪赌——这种事不应该从一家私人公司的 Slack 里发起。试图"速通对齐"，应该要求有非同寻常的信心：确信不存在更好的路径。

Accepting this race and entering the “endgame” is a hubristic gamble that should not be launched from a private company’s Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.

我对协调的可能性保持乐观。像 Hugging Face 攻击事件这样的警告枪，已经让美国各实验室之间达成配速协议变得更现实。但我不觉得我们正走在阻止一场全球竞赛的轨道上——这可能需要代价高昂的行动，比如暂时禁止提升模型能力。

I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.

如果你也是实验室研究员，我恳请你想想未来几年实际会是什么感受。你愿意在没有严格理解其心智的情况下，启动一次超级智能的 RL 训练吗？你应该因为"反正它会发生"就埋头继续——还是抓住这个时刻，呼吁改变条件？

If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because “it’s happening anyway” - or take this moment to call for different conditions?

Jacob 说的是对的——我们真的真心相信 AI 可能杀死所有人类！我个人认为未来十年内的概率超过 10%。我相信 Anthropic 已经在尽全力，但我们目前没有解决超级智能对齐的方案，也显然没有走在解决它的轨道上。

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.