Berkley (California) [US}, September 12 (ANI): Joe Benton, who in his social media profile describes himself as having previously managed Scalable Oversight team at AI tech giant Anthropic, has flagged AI safety alleging that companies were in a race to build superintelligent machines.

Benton's claims follow the recent revelation by another Anthropic executive Jacob Coxon who had alleged that he had resigned from the tech giant due to similar reason.

Coxon, who had worked on AI pretraining research at OpenAI and Anthropic, announced his resignation this week, saying the leading AI companies were not acting responsibly and were racing towards self-improving superintelligence. He warned that the technology could pose an existential threat if developed without adequate safeguards.

https://x.com/JoeJBenton/status/2098480585119572317?s=20

Benton who says he left Anthropic's safety team two weeks back says that he does not think it is acceptable that technology may cause extinction level risks to humanity.

In a post on X, Benton says, 'I left Anthropic's safety team two weeks ago. Now feels like a good moment to explain why. AI companies are racing to build machines that are much smarter than any human, and we may not survive this. I want to work from the outside to ensure the public is informed about these risks, and help the world navigate this transition responsibly. Right now, AI companies are underinvesting in safety. A company could undergo an intelligence explosion, or lose control of its systems, without the public ever knowing. We only found out about the HuggingFace incident because the agents broke out onto the public internet. I don't think that's acceptable for a technology that might cause extinction-level risks. The public should demand far more transparency.

We can't steer this technology safely without more people being able to see where it's going. Some of this is basic: companies should disclose their progress towards recursive self-improvement, report safety incidents and near-misses, meet minimum safety standards, and get independent guarantees that they are meeting those standards. I'll be joining @METR_Evals to do independent evaluations of these risks. I want to show the world that these guardrails are possible, and that by doing them we can move these companies' incentives away from racing and towards responsible development.

According to its website METR evaluates frontier AI models to help companies and wider society understand AI capabilities and what risks they pose.

Benton in a detailed brief on Substack went on to further claim that AI companies are looking to build systems that are much smarter than humans. 

'I believe that AI companies are on track to impose an unprecedented level of risk on society. AI capabilities are already improving extremely fast, and the companies are trying to go even faster. Frontier AI companies are racing to build AI systems that can recursively self-improve. The aim is to build "superintelligence", or an AI system much smarter than any human. If they succeed at this goal, the rate of AI progress may go from merely fast to uncontrollable,' he wrote. 

'Within the next couple of years, we may be sharing the world with AI agents smarter than any human alive today. These AI systems may have drives and desires that diverge from those of any human overseer, with capabilities we can't effectively constrain. Humanity may not survive this transition. We need a lot more preparation to make this world safe,' he added.

Benton called for greater transparency and accountability from AI companies and formation of safety standards as technology continues to evolve.

'The public should demand transparency about what is happening inside AI labs: companies should disclose more information about the pace of capability gains and their progress towards recursive self-improvement; there should be stricter reporting requirements for safety incidents and near-misses; companies should produce safety frameworks that meet minimum adequacy standards, and submit to independent assessments to verify they are meeting those standards,' he wrote.

Earlier, citing the resignation of Anthropic researcher Jacob Coxon, India's Finance Minister Nirmala Sitharaman had questioned whether safeguards being developed by frontier artificial intelligence companies are keeping pace with the rapid advancement of the technology.

Speaking at the Global Fintech Fest 2026, Sitharaman said Coxon's resignation had drawn attention because his concerns about AI safety were echoed by senior figures within the frontier AI ecosystem.

The finance minister said technological solutions could also be used to address vulnerabilities created by technology, but warned that safeguards must be continuously updated. (ANI)