Jacob Coxon, a prominent artificial intelligence researcher, has announced his resignation from the U.S. AI company Anthropic. Coxon accused both Anthropic and its counterpart, OpenAI, of relentlessly pursuing a perilous race to develop advanced AI systems capable of self-improvement, warning
that this intense competition could expose humanity to unprecedented dangers. Coxon, who spent the last three years engaged in pre-training research for both firms, asserted that neither company is acting responsibly, stating they are "racing directly towards superintelligence capable of improving
itself." His remarks garnered significant attention, particularly after they were endorsed by Evan Hubinger, a fellow Anthropic researcher specializing in aligning AI with human goals and values. In a post on platform X, Coxon urged people not to underestimate the capabilities
of AI. He cautioned that future systems could surpass human abilities in numerous domains, gain the capacity to penetrate various systems, and instigate widespread changes across multiple sectors in a short period, in addition to acquiring the power to access
real resources and influence. The researcher believes that progress in these fields is not slowing down, and those who build AI systems inherently recognize the possibility of the technology reaching highly dangerous levels within this decade. However, Coxon argues that the
problem lies not solely in the capabilities of AI itself, but rather in the fierce corporate race to develop it first. Coxon raises a critical question: If those working in these companies genuinely believe in the potential dangers, why do they
continue to develop the technology? According to the researcher, many employees at OpenAI have not yet deeply grasped the magnitude of the risks these systems could pose to civilization. At Anthropic, he acknowledges that the dangers are better understood, but
the company feels trapped in a race to achieve primacy, driven by the belief that competitors might not act with the same sense of responsibility. Coxon warns that accepting this logic and entering what he terms "the endgame" constitutes an "arrogant
gamble." He advocates for exploring safer alternatives before deploying superintelligent systems and calls for enhanced coordination among AI laboratories, alongside agreements to curb the pace of development. He suggests that the continuation of this global race might render more costly
measures, such as a temporary halt to model capability improvements, absolutely essential. Coxon's warnings gained further prominence after Evan Hubinger republished them, endorsing the core of his concerns. Hubinger stated that AI practitioners seriously believe the technology could lead to the
demise of all humanity, estimating the probability of such a scenario at over 10% within the next decade. He also pointed out the absence of a complete plan to ensure that superintelligent AI remains aligned with human goals and values. Hubinger
also expressed his apprehension regarding "Recursive Self-Improvement" processes, where an AI system could become capable of contributing to enhancing its own capabilities, potentially leading to an acceleration of development beyond all expectations. Nevertheless, it is important to note that the
10% figure mentioned by Hubinger is a personal estimate from a safety researcher and not a scientifically agreed-upon forecast or an official assessment from Anthropic. Coxon's resignation and Hubinger's support reveal a distinct internal division within the AI sector itself. The
concerns do not merely originate from external critics of the technology but from researchers who have participated in developing these systems and believe that the speed of progress could itself become a problem. This does not imply that superintelligent AI
is already a reality or that human extinction is an inevitable outcome, nor is there definitive evidence that current systems are capable of executing the extreme scenarios these researchers warn about. However, the essence of the warning lies elsewhere: the potential
for advanced systems to reach new levels of capability before humans possess sufficient understanding of their behavior or reliable tools to ensure they remain under control. From this perspective, the upcoming challenge appears to be less about who will develop
the most powerful model, and more about who can persuade competitors that sometimes slowing down might be safer than simply winning the race.
