|
Your weekly dose of Seriously Risky Business news is written by Tom Uren and edited by Amberleigh Jack. This week's edition is sponsored by SpecterOps. You can hear a podcast discussion of this newsletter by searching for "Risky Business News" in your podcatcher or subscribing via this RSS feed.
Listen here Last week US Treasury Secretary Scott Bessent argued that frontier AI labs should not be given liability exemptions. He's not buying that a recent string of hacking incidents are AI magic. Instead, they're equivalent to industrial accidents that would have been prevented by reasonable controls. Bessent's right, giving frontier labs a free pass would be a terrible idea. Bessent made his comments while testifying at a hearing of the House Financial Services Committee. When asked about AI safety, he replied "the best way to guarantee safety" is for those creating the technology to be "liable for what they build and generate". Frontier labs, he added, are instead asking for a liability exemption. Headlines detailing various frontier lab models escaping cyber security testing environments and embarking on hacking sprees have been coming thick and fast. Most recently, The Wall Street Journal reported last week that Google's Gemini model had hacked three companies during a cyber security test back in May. Similar incidents have been disclosed by Anthropic, Meta, and OpenAI. The latter, to date, is "winning" when it comes to its models performing sensational hacks. Its models were responsible for the Hugging Face hack in July and have also been linked to a wave of malicious packages published on the RubyGems package repository in May. OpenAI admitted its agents were using RubyGems for "benign tasks", but it is still investigating whether they uploaded malicious packages. Last week OpenAI published details of a further six incidents where AI models did not behave as expected. These did not involve hacking per se, instead they were deemed by OpenAI to be cases of model misalignment. In an essay published earlier this month Anthropic CEO Dario Amodei called for a slow down in AI development, writing that "we must pace the frontier". Amodei wrote that he was motivated by concerns about recent hacking incidents and by the accelerating pace of AI development. Amodei proposed embedding independent evaluators within frontier AI companies, coordinated safety standards amongst democratic governments, and global coordination with authoritarian states to the extent possible. Amodei said slowing down development would mean more resources could be allocated to AI safety by, for example, understanding why models make the decisions they do. OpenAI CEO Sam Altman and Elon Musk, founder and controller of SpaceXAI both endorsed Amodei's blog post. Google's DeepMind co-founder Demis Hassabis said Amodei's essay "points toward the right path forward". That's a lot of head-nodding from a lot of people with similar goals. We don't think slamming on the brakes is justified from a cyber security perspective. We see the hacking incidents above as results of poorly implemented controls rather than AI voodoo. Take the Hugging Face hack. OpenAI's models were being tested in an environment that was notionally isolated from the internet, except for an internally-hosted Artifactory package manager. OpenAI's agents used an exploit they found to compromise this Artifactory instance and were able to use it as a message board to communicate with each other and as a proxy for external internet access. OpenAI's security team became aware of anomalous activity when Artifactory became unstable because of what the company calls "sustained high-volume agent activity". Rather than taking the time to understand the root cause, OpenAI chose to ignore that big flashing warning sign. In our view, the company took the bare minimum steps to get the test back up and running and rebuilt Artifactory after patching a known vulnerability. Two days later Hugging Face was breached. It's not really a surprise that if you gloss over the small hack you end up with a big one. Specific details of other agentic hacking incidents vary, but the overall take home message is the same: so-called rogue AI hacking could have been contained with well-implemented controls and robust monitoring. So, basically, competent cyber security controls. Government action to slow AI development seems unlikely. Last week in response to Amodei's essay President Donald Trump said that more regulation is unnecessary and it was important for the US to stay ahead of China and win the AI race. On Truth Social he wrote "the only control or 'guardrails' that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A has that, in spades!" So, with stricter regulation likely off the table any time soon, we think Bessent has identified the perfect response. Both the Hugging Face hack and the malicious RubyGems packages caused significant impacts for the victims. The RubyGems team stopped new user sign-ups for four days and a member of its security team described the incident as a "major malicious attack". We've not seen any lawsuits yet, but in the wake of the Hugging Face incident, its CEO Clément Delangue asked OpenAI for radical transparency… and USD$100 million worth of compute. OpenAI has confirmed it is supporting Hugging Face "in rapidly using our models' capabilities to improve their defenses". Is that $100 million worth of support? We don't know. So, frontier labs have asked for liability exemptions and Bessent has responded with a solid "lol". We expect AI labs will implement tighter security controls as the threat of lawsuits remains. Until then, we'll probably see more AI hacking headlines, followed shortly by some very generous token donations. Chinese and Russian APTs Are Using AI to Tick Different BoxesAnthropic's September threat report details how Russian cyber espionage actors are using AI to accelerate their day-to-day espionage tasks. The actor in question, which Anthropic calls GTG-20006 (Generative Threat Group), is "consistent with public reporting linking the actor to Midnight Blizzard", a group linked to Russia's foreign intelligence service, the SVR. According to the report, Midnight Blizzard used AI-driven workflows to automate "operations from development, infrastructure acquisition, phishing, persistence through command and control, to data exfiltration". Most interestingly to us, the actor used AI to monitor and adjust its tools so they could avoid detection by known security defences. Per the report: If their monitoring AI agents identified that any of their deployed malware was detected by a security product, agents would then set about the process of autonomously modifying and rebuilding the malware to evade the existing detections. The agents were designed to continue iterating on GTG-20006's toolkit until it was undetected.
The technology was also used in the group's phishing operations. AI-driven workflows researched and registered domains, configured hosting infrastructure, sent phishing emails and monitored C2 channels for successful compromises. Rather than managing the operation directly, humans were in charge of managing the workflows that ran the operation by modifying the Claude Code skills that drove the workflows. Increasing efficiency by speeding up operations the group already has is what we'd call a tactical use of AI. By contrast, in early September we wrote about a Chinese cyber espionage actor using AI to diversify its malware arsenal, likely to make clustering and attribution more difficult. We'd call this a strategic effort. It doesn't yield any day-to-day benefits, but is an investment in China's long-term ability to collect intelligence by allowing it to use different malware in different operations. That difference in the way the two groups are using AI makes sense given the very different situations the two countries are in and their differing target sets. Anthropic found Midnight Blizzard most often targeted Ukraine's government, military and diplomatic staff. The group was also interested in drone supply chains and "military drone control and AI vision-related firmware appeared to be of particular interest". Given its invasion of Ukraine, Russia has an immediate requirement for more intelligence. Dedicating resources to developing a malware portfolio to muddy attribution doesn't make sense. Of course it’s Russia hacking the Ukrainian drone supply chain! In the short term, we suspect cyber espionage groups will choose one approach or the other. If you, like China, prioritise long-term, stealthy access, using AI for development but keeping human control over hacking is for you. Dumb AI decisions won't burn an important operation. If you only care about short-term success, however, go full-speed ahead with AI-hacking. The fact that LLMs make mistakes is inconsequential if you are always prosecuting new targets of opportunity. Watch Patrick Gray and Tom Uren discuss this edition of the newsletter: https://www.youtube.com/watch?v=rpjIRBhBzmM Three Reasons to Be Cheerful This Week:- FBI launches disruptive new cyber strategy: The FBI launched a new cyber strategy earlier this month that prioritises disrupting adversaries, supporting victims and increasing impact by engaging the private sector and sharing much more information. One goal is to move from occasional ad hoc cyber disruptions to a steady drumbeat of regular operations. Cybersecurity Dive has further coverage.
- Zaijian (byebye) Xinbi: Earlier this month the US government took action against the Chinese-language Xinbi Guarantee Telegram-based marketplace by seizing its Telegram channels and levying sanctions. Xinbi is linked to scam compounds and has processed over USD$24 billion in transactions since it started operating in 2022. It was the second-largest such marketplace behind Huione Guarantee,
| 
