|
Your weekly dose of Seriously Risky Business news is written by Tom Uren and edited by Amberleigh Jack. This week's edition is sponsored by runZero. You can hear a podcast discussion of this newsletter by searching for "Risky Business News" in your podcatcher or subscribing via this RSS feed. Listen here The US government has committed to countering Chinese 'distillation attacks' which are being used to steal the proprietary capabilities of American frontier AI models. We love a little governmental fist-shaking, but we don't think its plan will have China's AI labs shaking in their boots. Distillation attacks, also known as model extraction attacks, upskill less capable models on the cheap by training them on the outputs of more advanced models. Back in February, OpenAI, Google and Anthropic each said that they had been victims of distillation attacks. Anthropic said that Chinese labs had collectively generated "16 million exchanges" with Claude, across 24,000 fraudulent accounts. Google cited an attack that involved 100,000 queries to Gemini. Last week, a memo released by the White House acknowledged the problem: …foreign entities, principally based in China, are engaged in deliberate, industrial-scale campaigns to distill US frontier AI systems. Leveraging tens of thousands of proxy accounts to evade detection and using jailbreaking techniques to expose proprietary information, these coordinated campaigns systematically extract capabilities from American AI models, exploiting American expertise and innovation.
The memo also promised action. It's great to get a rapid response from the government, but when it comes to the specific actions the administration has committed to we are, sadly, underwhelmed. According to the memo the administration will: Share information about distillation attacks; enable private sector coordination against attacks; facilitate development of best practices to "identify, mitigate, and remediate" distillation attacks and "explore a range of measures" to hold actors accountable. These feel an awful lot like the ineffective measures the US government used throughout China's 20 years of IP theft. But wait, it's not just information sharing, coordination and best practices… We have a strongly worded letter to add to the mix as well! Reuters reported the State Department has cabled diplomatic posts and directed them to raise "concerns over adversaries' extraction and distillation of US AI models" in their host countries. The cable also said that a "demarche request and message has been sent to Beijing". That demarche is a formal raising of concerns with the Chinese government. In response, the Chinese embassy in Washington said the White House's accusations of AI intellectual property theft were "pure slander". We're shocked, too. To be fair, the Trump administration's promised actions are reasonable and will make some difference. But they won't stop Chinese distillation attacks. The technology is such a game changer that China's AI labs will be fully committed to overcoming the US government's countermeasures. The disappointing thing about the memo is that it made no mention of strengthening the most effective tool the US government has: semiconductor export restrictions. There is already evidence that chip export restrictions are hampering the development of Chinese models. The release of Chinese AI company DeepSeek's latest V4 model was significantly delayed because it unsuccessfully tried to train the model on Huawei's Ascend processor. DeepSeek eventually reverted to using chips from American company Nvidia to train the model. This was possible despite restrictions because Chinese companies do have access to older chips, plus… export restrictions have been leaky. DeepSeek is using Huawei chips to actually run the model to answer queries, a process known as inference. Additionally, a Chinese tech blogger reported that "constraints on computing power and cash" is why DeepSeek V4 is a text-only model rather than being multimodal. In its V4 technical report DeepSeek itself says the model "trails state-of-the-art frontier models by approximately three to six months". Huawei's Ascend can theoretically be produced by Chinese firms, but historically the bulk of Ascend production has occurred in Taiwan in violation of sanctions. Ascend chips are not as performant as Nvidia's leading chips and, at least so far, Chinese firms cannot produce as many chips as Nvidia. Suffice it to say that export controls are complementary to measures that counter distillation attacks. They must be part of the solution to maintaining America's AI advantage. Leading AI firms have also argued for stronger export controls. It was counterproductive to loosen controls when access to chips is the only structural advantage that the US has in the AI technology race. There may be good political reasons export restrictions were not mentioned in the White House's memo. In just over two weeks, President Donald Trump is scheduled to meet China's President Xi Jinping. A blow up over chips could upset that meeting. Still, China has spent the last 20 years pillaging intellectual property from advanced economies, using a comprehensive range of techniques that covered the gamut from economic inducements through to cyber espionage. It would be an absolute tragedy if key technologies for the next 20 years were stolen as well. Good News Everyone! Chinese Hackers Adopt BotnetsChinese threat actors are moving en masse to using botnets of compromised smart devices to facilitate their operations. This makes a network defender's job more difficult, but presents opportunities for government disruption. Last week, the UK's NCSC and a host of international cyber security authorities jointly released an advisory detailing a "major shift" in the way Chinese cyber actors are operating. These actors have shifted from rolling their own individual infrastructure to using large-scale networks that are sometimes managed by third parties. The networks are primarily made up of compromised SOHO routers and IoT devices. The NCSC refers to them as "covert networks". They are commonly known as botnets. The report says these networks are used for all aspects of a cyber operation including reconnaissance, malware delivery, command and control, data exfiltration and deniable internet browsing such as researching exploitation techniques. The covert networks are a low-cost, low-risk way to disguise the origin of malicious activity. Multiple covert networks have been created, they are constantly being updated and any one network could be used by multiple actors. The NCSC believes "the majority of China-nexus threat actors" are using them. Leveraging botnet-based networks isn't a new idea in the world of state-sponsored espionage. Russian agencies have a very long history of doing exactly that. The FSB ran the Snake malware network for 20 years from 2003 and the GRU created the botnets known as VPNFilter and Cyclops Blink. It does, however, appear that China's equivalents are commercial endeavours rather than being created and run by a state intelligence organisation. Take the Raptor Train botnet for example. In 2024 the US announced a court-authorised takedown of the Chinese botnet that Lumen Technologies had analysed and named. That botnet was formed in 2020, contained more than 60,000 devices at its peak and had, over time, compromised more than 200,000 devices including SOHO routers, DVRs and IP cameras. The botnet was run by the hacking group Flax Typhoon, which the US government linked to a Beijing-based cyber security firm, Integrity Technology Group. Rather than being a top-down mandate from the Ministry of State Security (MSS) or Ministry of Public Security (MPS), Dakota Cary, a China-focused consultant at SentinelOne, told Seriously Risky Business the shift towards using botnets was likely "led by market forces". Eugenio Benincasa, who authored various reports into the Chinese cyber espionage ecosystem, told SRB the system is both "competitive and collaborative". He believes the companies that build these covert networks could sell them to multiple customers. Additionally, the provincial arms of the MSS and MPS often work "semi-independently", and Benincasa pointed to the the i-SOON data leak as showing a single firm "working with dozens of [MPS or MSS] bureaus across more than 30 provinces". This fragmentation makes it less likely that the adoption of covert networks is the result of centralised direction. A 2024 Mandiant report described the beginning of this shift. It detailed what it called a "growing trend among China-nexus cyber espionage" towards covert networks. At the time, Mandiant thought the trend was motivated by "rais[ing] the cost of defending an enterprise's network and shift[ing] the advantage toward espionage operators by evading detection and complicating attribution". From a government perspective, botnets actually present a disruption opportunity, one that the US has done a half-decent job taking advantage of. In 2024, in addition to Raptor Train, it also disrupted the Chinese KV botnet and the criminal "911 S5" service. In March this year, it announced that it had disrupted four DDoS botnets. So, although these networks have proven worth to adversaries, they are also a point of vulnerability. Adversaries are on a treadmill that involves constantly maintaining, developing and renewing these networks. The US disruption track record is, as we said, half decent. But we'd like to see a constant drumbeat of takedowns. Authorities need to increase their pace. Watch Amberleigh Jack and Tom Uren discuss this edition of the newsletter: Three Reasons to Be Cheerful This Week:- US launches scam crackdown: The US government announced a multipronged action that targets the entire lifecycle of scam operations. These actions include levying criminal charges against two Chinese nationals, sanctions targeting scammers in Cambodia, disrupting recruitment and 'restraining' cryptocurrency to prevent it from being moved. Chainalysis has further coverage.
- Alleged Silk Typhoon hacker extradited to the US: Xu Zewei, a Chinese national has been extradited to the US from Italy, where Xu and his wife were arrested while vacationing in Milan. The US alleges Xu is a member of the Chinese hacking group known as Silk Typhoon (previously Hafnium) and was involved in the 2021 mass exploitation of Microsoft Exchange servers among other things. The Record has further coverage.
- Another Scattered Spider Arrest: A 19-year-old dual United States and Estonian citizen has been arrested and is facing charges related to being a member of the Scattered Spider cybercrime group. The Chicago Tribune has further details based on court records that were temporarily unsealed.
| 
