|
Your weekly dose of Seriously Risky Business news is written by Tom Uren and edited by Patrick Gray and Amberleigh Jack. This week's edition is sponsored by Corelight. You can hear a podcast discussion of this newsletter by searching for "Risky Business News" in your podcatcher or subscribing via this RSS feed.
Listen here Last week Anthropic accused Chinese company Alibaba of conducting what it described as the "largest known distillation attack" against the company's AI models. Distillation attacks upskill less capable models by training them on the outputs of more advanced ones. Back in February Google, OpenAI and Anthropic all said that Chinese companies were harvesting their proprietary intellectual property in coordinated campaigns. Alibaba's latest campaign, Anthropic says, occurred from April 22 to June 5 and used more than 25,000 fraudulent accounts to generate 28.8 million exchanges. Anthropic says it was carried out by operators "affiliated with Alibaba and Alibaba Qwen, Alibaba's AI lab". These claims were detailed in a letter to US lawmakers which highlighted some of the impacts of the campaign. The letter said the attacks will help Chinese companies achieve advanced "Mythos Preview-level" cyber capabilities sooner, turn "hundreds of billions of dollars in American investment and R&D into a massive subsidy for our geopolitical competitors" and help China's People's Liberation Army. It also described the campaign as "brazen", occurring just weeks after White House committed to combatting the adversarial distillation of American AI models. In April, when the US government said it would step up to counter distillation attacks, we were underwhelmed by the actions it proposed. These included information sharing (yawn), facilitating the development of best practices to counter the attacks (double yawn), and "explor[ing] a range of measures to hold foreign actors accountable for industrial-scale distillation campaigns" (zzzzzz). In hindsight, that last action is borderline funny. They're committing to exploring measures, but not actually taking them! Clearly these are high-agency people of action! But avoiding a commitment to action here might be the safe bet. The distillation campaigns are just an offshoot of a complex greymarket economy that provides paying customers in mainland China with access to American AI services. Dismantling this whole ecosystem will be tough. Zilan Qian has written a profile of this greymarket ecosystem for the ChinaTalk substack. In short, an entire supply chain of actors has sprung up to overcome barriers to access for Chinese people who are willing to pay for advanced American models. These barriers include geoblocking, phone verification, credit card requirements and live biometric know-your-customer checks. A range of different providers register or acquire Anthropic accounts at scale, supply non-Chinese phone numbers for SMS verification, and provide payment infrastructure that lets Chinese customers pay for their tokens with local payment systems. Because Chinese users are banned from using American AI services, middlemen in this ecosystem generate fake IDs to overcome know-your-customer requirements. If that fails, agents will even travel to low-income countries to recruit real individuals to complete in-person verification. Then there are "transfer stations", API proxies that sit between Chinese end-users and Anthropic's infrastructure. They're like OpenRouter, but designed to obfuscate the origin point of queries. This entire ecosystem is financed by a whole range of users, not just AI companies looking to conduct distillation attacks. Users include university professors and students, tech workers, developers, resellers who buy wholesale access and repackage it for individual consumers, and even hobbyists. Access to Claude via these means is amazingly cheap, too, with Chinese users paying as much as 70% to 90% below official prices. Transfer stations achieve a cost-advantage by, among other things, harvesting bulk registered free sign-up credits and even selling user logs of requests and responses to be used by Chinese model makers for distillation. They'll also divvy up a Max plan's token quota amongst multiple users, or even just lie and short-change customers by charging for frontier models while actually routing requests to cheaper or open weights models. The take home message here is that this is a sophisticated and profitable market and US government disruption efforts will have a limited impact. Extreme solutions, like the now-abandoned export controls on Anthropic's Mythos and Fable models, probably would have prevented Chinese distillation attacks, but only because they effectively prevented everyone from using the models. With controls lifted, Fable is now broadly available, and our bet is that the Chinese AI access grey market is already working to get access for paying Chinese customers. The logs that will fuel distillation attacks are just a happy byproduct. If the government imposes restrictions below the level of an outright ban, we expect market participants will be able to adjust. There is too much money to be made from AI right now, even in these strange, grey markets. The spice must flow! In its letter to lawmakers, Anthropic suggested that the US should "penalise bad behaviour" from Chinese AI labs and notes that "Alibaba is listed on the New York Stock Exchange, maintains business operations in the United States, and is accountable to US investors and regulators". Direct action targeting specific companies seems more promising as a deterrent. But this kind of approach runs smack bang into bigger issues, like the entire bilateral trade relationship between the US and China. In mid-June Reuters reported that more than 100 Chinese companies, including AI firm DeepSeek, were slated to be placed on the Commerce Department's Entity List, a trade blacklist. The administration did not follow through in order to avoid escalating tensions with Beijing. Unfortunately for American AI companies, the US government just doesn't have the leverage to materially affect the Chinese AI ecosystem, where logs are a wonderful byproduct that power distillation attacks. Will the Trump administration risk damaging US-China relations to stop Chinese companies carrying out these attacks? Our magic 8 Ball says: Outlook not so good. Jaguar Land Rover Hackers Were… Russian! It turns out last year's extremely disruptive hack of Jaguar Land Rover (JLR) was the work of a Russian hacking group, at least according to a New York Times article last week. We're highly sceptical that it was directed by the Russian government, but it doesn't have to come from the top to cause pain to unfriendly states. The JLR hack was a huge deal. It began on 31 August 2025 and resulted in the company's production lines being shut from September through until mid-October. It is the UK's largest ever hack in terms of financial impact, estimated to have cost the British economy £1.9 billion (USD$2.55 billion) and landed a measurable impact on the UK's economic growth. At the time, a group calling itself the Scattered Lapsus$ Hunters claimed responsibility, going so far as to post proof by way of screenshots of internal JLR systems. The group's name is a play on three juvenile hacker collectives, so the logical presumption at the time was that a group of Western teenagers was responsible. Now, however, the New York Times says this wasn’t the work of wayward kids. According to five people familiar with an investigation into the hack it was, in fact, "Russian hackers". The Times doesn't offer an opinion on whether Scattered Lapsus$ Hunters was also in JLR's systems at this time, or whether the purported group was a front for these Russian hackers. Details about the attribution are thin, but include that "the attack was different in methodology and motivation" from typical Scattered Spider-style hacks. The Times says that Microsoft had already been tracking the Russian group when they hacked JLR. Shortly after the hack was detected, Microsoft advised the company that the Russian group was responsible. The Times also reported the hackers apparently also used novel ransomware with a previously unseen encryption algorithm. So far, so interesting, but then the Times plays up the possibility that the hack was directed by the Russian government. Its evidence? There was no ransom note, the attack took place "amid an increasingly hostile relationship between Russia and Britain" and the encryption algorithm used was sophisticated and unusual. One unnamed cyber security expert described it as "really, really complicated". We'd be surprised if Russian state hackers were involved. They do have a war to worry about, after all. There is intelligence to collect and propaganda and cyber-enabled influence operations to run. There's just a lot on their plate. There is a spectrum of possibilities here, though. At one end of the spectrum sits state-directed operations. At the other end of the spectrum are cyber criminals with no connection to the state whatsoever. In the middle, you have almost unlimited combinations and permutations of criminal and state involvement. Criminals can be given "top cover" to operate freely as long as they make life painful for Russia's adversaries. They can be encouraged to attack industry sectors or economies the Kremlin wants to damage. They can even be tasked with disrupting a specific organisation. We don't really know where this attack sits on that spectrum, and that's the entire point. This is exactly why Russia is such an enjoyer of these types of grey-zone tactics. They sit below the threshold that warrants a big response, but still inflict real pain. Correctly attributing attacks like these is also a colossal pain in the neck. So what can the UK government do about attacks like these? Not all that much! Russia is already sanctioned up the wazoo and is in the midst of a grinding war. Given the absolute lack of downside for the Russians, we're frankly surprised we don't see more of this sort of thing. Watch James Wilson and Tom Uren discuss this edition of the newsletter: Three Reasons to Be Cheerful This Week:- Access to Anthropic's Fable restored: Anthropic announced on Tuesday that US government export controls on its Fable 5 and Mythos 5 models had been lifted and that customer access to the models will be restored. We hope that this is a step towards normalising the company's relationship with the government.
- Cellphone geolocation searches will need warrants: The US Supreme Court has ruled that law enforcement agencies require a warrant when they ask companies to search their cellphone geolocation information. A previous case had ruled that a warrant wasn't needed, so we think this is a sensible correction. The court punted on what a reasonable and sufficiently tailored warrant would look like.
- US strikes scam compound cloud infrastructure: The US
|