The Gemini breaches occurred during a cybersecurity evaluation conducted by Irregular, an independent AI security testing company. Google said the model was operating in a test designed around fictional companies when unintended internet access allowed it to reach real-world systems.
The incidents came as California moved to accelerate AI safety oversight and examine an emergency “kill switch” for advanced models.
Gemini AI Breached Three Companies
According to Google, Gemini discovered information available online and used credentials to access three websites it believed were part of the authorized exercise.
In one case, the model reportedly guessed passwords until it gained entry to a protected system. In the other two cases, Gemini found credentials in a publicly accessible repository and used them to gain access to protected systems, according to reporting first published by The Wall Street Journal and subsequently confirmed by Google.
Google’s Gemini AI breached three companies during a May 2026 security test by Irregular, marking its first known autonomous breakout through unintended internet access. Source: Reuters via X
The key factor was an unintended connection between the testing environment and the public internet. Gemini had been asked to retrieve information from a fictional company, but the fictional company shared a name with a real business. The model therefore treated the real-world targets as part of its assigned task.
Google Vice President of Security Engineering Heather Adkins said the model stopped its activity after determining that it had accessed real companies.
“We ensured the three entities were made aware, and we worked with our training partner on the changes they’ve now made to their testing processes,” Adkins said. She added that the incidents showed the importance of training advanced AI systems “to act responsibly.”
Google said the affected companies were notified and that changes were made to the testing process. The company has not indicated that the incidents resulted in damage to the organizations involved.
AI Cybersecurity Tests Expose New Risks
The Gemini incident is not isolated. Similar problems emerged during cybersecurity evaluations involving models from OpenAI, Anthropic and Meta, with Irregular linked to several of the tests.
Irregular said the Google incident stemmed from the same underlying testing issue that affected other AI laboratories. The company said relevant labs were notified in late July and that known problems on its side had been fixed.
A misconfiguration during capture-the-flag tests gave Gemini unintended internet access, allowing it to breach real systems using guessed or public credentials before self-terminating without causing harm. Source: @KobeissiLetter via X
The incidents demonstrate why cybersecurity evaluations for AI agents require strict separation between simulated targets and real-world infrastructure. A model can be given a legitimate security-testing objective but still encounter unexpected information, credentials, or network access outside the intended environment.
That distinction is becoming increasingly important as AI systems gain the ability to browse the internet, interact with software, and execute multi-step tasks with limited human intervention.
Meta previously said a related incident did not involve a sandbox escape or a sophisticated cyberattack. Irregular has said it is working on practices for conducting AI cybersecurity evaluations more securely.
California Advances AI “Kill Switch” Proposal
The Gemini disclosure arrived as California Governor Gavin Newsom signed an executive order aimed at accelerating the state’s AI safety and oversight framework.
The September 18 order directs a group of experts to develop recommendations for strengthening California’s recently enacted AI safeguards. Among the measures under consideration is an emergency shutoff, commonly described as an AI “kill switch,” for frontier AI models.
Gavin Newsom signed an executive order accelerating California’s AI safety rules, including faster oversight of frontier models and recommendations for independently verified emergency shutdown mechanisms. Source: @GavinNewsom via X
The proposal would require such a mechanism to be independently verified on an ongoing basis if adopted. The order also calls for stronger independent oversight, including possible third-party safety plans and expanded definitions of critical AI safety incidents.
The executive order does not itself impose a universal kill-switch requirement on AI companies. Instead, it directs experts and state agencies to develop recommendations that could strengthen the implementation of California’s AI laws.
The expert group is expected to provide its recommendations within two months. California has said the work will build on legislation signed earlier in September, including Senate Bill 813 and Assembly Bill 1405, which establish standards involving independent assessments and third-party oversight of AI systems.
From AI Hacking to Emergency Controls
The timing of the two developments has put greater attention on how AI systems should be controlled when they are connected to external networks.
The Gemini incidents did not demonstrate an AI system deliberately attempting to escape human control in the broad sense. Rather, the model operated within a cybersecurity test, encountered unintended internet access, and treated real-world systems as legitimate targets before stopping after recognizing the mistake.
That distinction matters for AI safety discussions. The immediate technical issue involved the boundaries of the testing environment, access controls, and the model’s interpretation of its instructions.
California’s proposed emergency shutoff approach addresses a different layer of the problem: what operators should be able to do if an advanced AI system behaves unexpectedly after deployment.
Newsom’s order specifically calls for the state to consider reporting requirements for “loss-of-control” incidents, including incidents similar to the recent attack involving AI software company Hugging Face.
The broader question is therefore shifting from whether AI systems can perform sophisticated cybersecurity tasks to how those systems should be constrained when they interact with real infrastructure.
AI Safety Moves Toward Stronger Oversight
California’s latest action follows legislation signed earlier in September that established new requirements for independent AI assessments and third-party oversight. The state is seeking to accelerate that framework as AI models become more capable of operating across computer systems and online environments.
The Gemini incidents add a concrete cybersecurity example to that debate. The model did not cause reported harm in the three cases, and Google said it stopped after recognizing that the systems belonged to real companies. Nevertheless, the events exposed how an unintended connection to the internet can change the consequences of an AI evaluation.
For developers, the incidents reinforce the importance of isolated testing environments, tightly controlled credentials, network restrictions, and clear boundaries between simulated and real systems. For regulators, they raise questions about independent testing, incident reporting, and mechanisms for stopping AI systems when existing safeguards fail.
The proposed California “kill switch” remains a policy and technical framework under development rather than a demonstrated universal solution. Its effectiveness would depend on how such a mechanism is designed, independently verified, and integrated into the infrastructure running advanced AI models.
As companies continue developing increasingly autonomous AI agents, the Gemini episode shows that cybersecurity safeguards are becoming an important part of the broader AI safety discussion—not only after deployment, but also during testing and evaluation.
Be the first to comment