内容来源：https://www.technologyreview.com/2026/02/12/1132386/ai-already-making-online-swindles-easier/

内容总结：

人工智能正降低网络犯罪门槛，新型攻击威胁初现

网络安全领域近日出现警示：人工智能（AI）技术正在被不法分子用于实施网络攻击，使犯罪活动更易得手且危害性可能持续扩大。

去年8月，网络安全研究员安东·切列帕诺夫发现一款名为“PromptLock”的勒索软件，其特殊之处在于全程利用大语言模型（LLM）自动生成攻击代码、定位敏感数据并撰写勒索信，实现了高度自主化的攻击流程。尽管事后证实该软件仅为纽约大学的研究项目，但已清晰展示AI驱动恶意软件的潜在威胁。

目前，真正的网络罪犯已开始利用AI工具提升攻击效率。研究表明，至少半数垃圾邮件由LLM生成，而针对企业的钓鱼邮件中，AI生成比例在一年内从7.6%上升至14%。深度伪造技术更被用于实施高额诈骗，例如有企业员工在AI伪造的视频会议中被骗转账2500万美元。

安全专家指出，AI目前主要被罪犯用作“生产力工具”，辅助编写恶意代码、优化钓鱼策略及进行信息搜集。开源AI模型因易于篡改安全限制，更受攻击者青睐。尽管尚未出现完全自主、大规模成功的AI攻击案例，但攻击门槛已显著降低。

防御方面，传统安全措施目前仍能有效拦截多数AI辅助攻击。同时，AI技术本身也被用于提升威胁检测能力，例如微软每日通过AI系统分析超100万亿条安全信号。行业组织也正积极共享攻击手法信息，以共同应对新型威胁。

专家提醒，虽然全自动AI攻击尚未成为现实，但技术演进速度极快，未来可能出现利用未知漏洞的“零日攻击”AI化。当前，公众与企业需保持警惕，及时更新系统，并筑牢基础安全防线。

“AI正从根本上降低复杂网络攻击的门槛，”人工智能公司Anthropic威胁情报主管雅各布·克莱恩表示，“攻击节奏将超过许多组织的准备速度。”

中文翻译：

人工智能已经让网络犯罪变得更容易，未来情况可能更糟。

部分网络安全研究者认为，担忧AI策划网络攻击为时过早，另一些人则指出这类攻击可能早已发生。

安东·切列帕诺夫始终在寻觅有趣的事物。去年八月下旬，他果然有所发现。那是一个上传至VirusTotal的文件——这个网站是像他这样的网络安全研究者用来分析潜在病毒及其他恶意软件的平台。表面看来它人畜无害，却触发了切列帕诺夫自制的恶意软件检测机制。随后几小时里，他与同事彼得·斯特里切克仔细检查样本后意识到：这是前所未见的新型威胁。

该文件内含勒索软件——这种恶性恶意软件会加密受害者系统中的文件，迫使受害者向攻击者支付赎金才能恢复数据。但此样本的特殊之处在于，它全程运用了大语言模型技术。从入侵到控制，每个攻击阶段都渗透着LLM的痕迹。一旦植入系统，它能实时调用LLM生成定制代码，快速扫描计算机以定位待窃取或加密的敏感数据，还能根据文件内容撰写个性化勒索信。整个过程完全自主运行，无需人工干预。更棘手的是，每次运行模式都会变化，极大增加了检测难度。

切列帕诺夫和斯特里切克确信，这项被他们命名为"提示锁"的发现标志着生成式AI的转折点，它证明该技术能被用于制造高度灵活的恶意软件攻击。他们发布博客文章宣称发现了首例AI驱动的勒索软件，随即引发全球媒体广泛关注。

但实际威胁并未如表面那般骇人。博文发布次日，纽约大学的研究团队出面认领，解释这并非真正肆虐的网络攻击，而是旨在验证勒索软件全流程自动化可行性的研究项目——而他们确实成功了。

尽管"提示锁"最终被证实是学术项目，真正的网络罪犯确实已在运用最新AI工具。正如软件工程师用AI辅助编程和查错，黑客也在利用这些工具压缩攻击准备时间，降低技术门槛，让经验不足者也能尝试发动攻击。

伦敦大学学院计算机科学教授洛伦佐·卡瓦拉罗指出："网络攻击将日益频繁高效，这已非遥远可能，而是赤裸现实。"

硅谷有人警告AI即将实现全自动攻击，但多数安全研究者认为这种说法夸大其词。"不知为何，人们总幻想AI超级黑客的存在，这实在荒谬。"Expel安全公司首席威胁研究员马库斯·哈钦斯表示，他因2017年阻止席卷全球的"想哭"勒索软件攻击而闻名安全界。

专家主张，我们更应关注AI带来的迫在眉睫的风险——它正在加速欺诈行为并扩大其规模。犯罪分子日益利用深度伪造技术冒充他人骗取巨额资金。这些AI增强型网络攻击只会愈加频繁和具有破坏性，我们必须做好准备。

垃圾邮件与超越

自2022年底ChatGPT横空出世，攻击者几乎立即开始采用生成式AI工具。正如所料，最初主要用于制造海量垃圾邮件。微软去年报告显示，截至2025年4月的一年内，该公司拦截了价值40亿美元的诈骗交易，"其中多数可能借助了AI生成内容"。

哥伦比亚大学、芝加哥大学和Barracuda Networks的研究人员分析了ChatGPT发布前后收集的近50万条恶意信息，估计当前至少半数垃圾邮件由LLM生成。他们还发现AI正被用于更复杂的骗局：通过冒充可信人物针对特定企业员工实施的定向邮件攻击中，到2025年4月已有至少14%采用LLM生成（2024年4月该比例仅为7.6%）。

在轰动一时的案例中，某公司员工曾因与伪造的公司首席财务官及其他同事的数字形象进行视频通话，被骗转账2500万美元。

生成式AI的爆发不仅让伪造邮件变得简单廉价，还能轻松制作极具说服力的图像、视频和音频。其逼真程度远超数年前，且伪造人物形象或声音所需数据量大幅减少。生成式AI专家亨利·阿杰德指出，犯罪分子部署深度伪造并非为了恶作剧，而是因为有利可图。"只要有钱可赚且人们持续受骗，他们就会继续作案。"2024年曝光的英国工程公司奥雅纳员工被骗案可能只是冰山一角，随着技术进步和普及，深度伪造带来的问题将日益严峻。

犯罪手段始终在进化，随着AI能力提升，不法分子不断探索如何利用新技术获取优势。谷歌威胁分析小组技术负责人比利·伦纳德持续追踪潜在恶意行为者（行业对黑客等试图利用计算机犯罪者的统称）使用AI的变化。2024年下半年，其团队发现潜在罪犯像普通用户那样使用谷歌Gemini调试代码、自动化工作流程，甚至撰写钓鱼邮件。到2025年，他们已进阶到利用AI创建新型恶意软件并投放传播。

当前核心问题是这类恶意软件能发展到何种程度。能否悄无声息地渗透数千家公司系统，盗取数百万资金而不被察觉？

主流AI模型虽设有防护机制阻止生成恶意代码或非法内容，但恶意行为者总能找到规避方法。例如谷歌观察到有与中国相关的行为者要求Gemini识别受侵系统的漏洞，该请求最初因安全原因被拒。然而攻击者通过伪装成网络安全竞赛参与者，成功诱使Gemini突破自身规则，提供了可能被用于攻击系统的信息（谷歌随后已调整Gemini拒绝此类请求）。

但恶意行为者不仅试图扭曲大型AI公司的模型。前美国司法部战术专家、现网络安全公司Intel 471高级情报分析师阿什利·杰斯指出，未来他们将更多采用开源AI模型，因为更易剥离安全防护并操纵其作恶。"这些模型能被越狱并定制，符合攻击者需求。"

纽约大学团队在"提示锁"实验中使用了OpenAI的两个开源模型，研究者发现甚至无需越狱技术就能让模型执行指令。参与该项目的纽约大学博士生米特·乌德希表示，虽然这类开源模型设计时考虑了伦理对齐，但其限制远少于闭源模型。"我们想验证这些自称伦理对齐的LLM是否仍能被滥用——答案是肯定的。"

乌德希认为，犯罪分子可能早已成功实施隐蔽的"提示锁"式攻击，只是我们尚未发现证据。理论上攻击者可能已创建全自动黑客系统，但这需要克服AI模型行为可靠性、内置恶意用途防护及规避检测等多重障碍——门槛确实很高。

黑客的效率工具

现有关于AI恶意用途的最佳数据来自大型AI公司自身。谷歌团队去年11月的报告发现，恶意行为者使用AI工具动态改变恶意软件行为（例如自我修改以规避检测），标志着"AI滥用的新阶段"。但网络安全作家凯文·博蒙特在社交媒体指出，报告涉及的五类恶意软件（含"提示锁"）均易被检测且未造成实际损害。

伦纳德承认当前恶意软件活动尚处早期阶段，但公开此类报告有助于安全厂商构建更好防御体系。"老话说得好，阳光是最好的消毒剂。保密无益，我们需要让公众和其他安全厂商知晓这些威胁。"

攻击者不仅试验新型恶意软件，还试图用AI自动化黑客流程。去年11月，Anthropic公司宣布挫败首例"无需大量人工干预"的大规模网络攻击。报告称中国国家支持的黑客组织使用其Claude代码助手，将"高度复杂的间谍活动"90%的流程自动化。

Anthropic威胁情报主管雅各布·克莱因表示："我们正进入一个新时代，复杂网络行动的门槛已根本性降低，攻击速度将超越许多组织的应对准备。"

但该报告也附带说明：目标选择仍由人工完成；30次尝试中仅少数成功；Claude在行动中出现幻觉伪造数据，频繁夸大成果，攻击者需仔细验证信息真伪。报告作者指出："这仍是全自动网络攻击的障碍。"

弗吉尼亚州贝里维尔机器学习研究所联合创始人、资深安全专家加里·麦格劳认为，任何具备基本安全防护的组织都能阻挡这类攻击。"漏洞利用等恶意攻击环节并非由AI执行，而是二十年前就已自动化的预制工具，毫无新意。"

防御进行时

目前研究者对防御AI威胁持乐观态度。伦纳德指出："过去十多年推荐的防御措施、能力和最佳实践——尤其在恶意软件方面——仍然适用。"传统安全程序能有效检测标准病毒和攻击尝试，多数钓鱼邮件仍会被收件箱过滤器拦截。

颇具讽刺意味的是，AI本身正助力更有效应对安全威胁。微软安全公司副总裁瓦苏·贾卡尔透露，公司每日处理超100万亿条被AI系统标记为潜在恶意或可疑的信号。

尽管网络安全形势瞬息万变，杰斯对防御者积极共享攻击者战术细节感到鼓舞。Mitre的人工智能系统对抗威胁图谱和OWASP的GenAI安全项目等倡议，正在系统记录犯罪分子如何将AI融入攻击，以及AI系统如何成为攻击目标。"我们已拥有优质资源来保护内部AI工具，并理解网络罪犯手中AI工具的威胁。"

作为有限学术项目的产物，"提示锁"虽不能代表真实攻击场景，但它警示我们不应低估AI的技术能力。乌德希惊讶地发现，AI能轻松完成从测绘目标系统到撰写个性化勒索信的完整攻击链："我们原以为它只能胜任初始任务，但实际在整个流程中取得80%-90%的成功率。"

AI仍在飞速进化，当前系统已能实现数年前看似天方夜谭的功能。这使我们难以断言其未来能力的边界。虽然研究者确信AI驱动攻击的数量和严重性都将上升，但其具体形态尚不明确。最极端的情况或许是出现能自主创造零日漏洞攻击的AI模型——但哈钦斯指出，构建和运行这种模型需要数十亿美元投入，可能只有富裕国家才能实现。

波士顿东北大学恶意软件检测分析专家恩金·柯达教授表示，若此类情况已成现实他毫不意外："我确信有人正投资于此，更确信已有人付诸实践——尤其是具备强大AI能力的中国。"

这种可能性令人不寒而栗。所幸目前仍停留在理论层面。既高效又明显由AI驱动的大规模攻击尚未出现。可以确定的是，生成式AI已显著降低犯罪门槛。犯罪分子将持续试验最新技术，寻找骗取信息和钱财的新手段。当下我们唯有保持谨慎、提高警惕，并及时更新系统——这不仅为自己，也为所有人。

深度解析

人工智能
将大语言模型视为外星生命的新时代生物学家
通过将大语言模型当作生命体而非计算机程序来研究，科学家首次揭示了它们的某些奥秘。

Moltbook是AI戏剧的巅峰之作
这个病毒式传播的机器人社交网络，在揭示智能体未来的同时，更折射出当前人类对AI的狂热。

杨立昆的新冒险是对大语言模型的反向押注
这位AI先驱在独家访谈中分享了其巴黎新公司AMI Labs的规划。

保持联系
获取《麻省理工科技评论》最新动态
探索特别优惠、头条新闻、即将举办的活动等更多内容。

英文来源：

AI is already making online crimes easier. It could get much worse.
Some cybersecurity researchers say it’s too early to worry about AI-orchestrated cyberattacks. Others say it could already be happening.
Anton Cherepanov is always on the lookout for something interesting. And in late August last year, he spotted just that. It was a file uploaded to VirusTotal, a site cybersecurity researchers like him use to analyze submissions for potential viruses and other types of malicious software, often known as malware. On the surface it seemed innocuous, but it triggered Cherepanov’s custom malware-detecting measures. Over the next few hours, he and his colleague Peter Strýček inspected the sample and realized they’d never come across anything like it before.
The file contained ransomware, a nasty strain of malware that encrypts the files it comes across on a victim’s system, rendering them unusable until a ransom is paid to the attackers behind it. But what set this example apart was that it employed large language models (LLMs). Not just incidentally, but across every stage of an attack. Once it was installed, it could tap into an LLM to generate customized code in real time, rapidly map a computer to identify sensitive data to copy or encrypt, and write personalized ransom notes based on the files’ content. The software could do this autonomously, without any human intervention. And every time it ran, it would act differently, making it harder to detect.
Cherepanov and Strýček were confident that their discovery, which they dubbed PromptLock, marked a turning point in generative AI, showing how the technology could be exploited to create highly flexible malware attacks. They published a blog post declaring that they’d uncovered the first example of AI-powered ransomware, which quickly became the object of widespread global media attention.
But the threat wasn’t quite as dramatic as it first appeared. The day after the blog post went live, a team of researchers from New York University claimed responsibility, explaining that the malware was not, in fact, a full attack let loose in the wild but a research project, merely designed to prove it was possible to automate each step of a ransomware campaign—which, they said, they had.
PromptLock may have turned out to be an academic project, but the real bad guys are using the latest AI tools. Just as software engineers are using artificial intelligence to help write code and check for bugs, hackers are using these tools to reduce the time and effort required to orchestrate an attack, lowering the barriers for less experienced attackers to try something out.
The likelihood that cyberattacks will now become more common and more effective over time is not a remote possibility but “a sheer reality,” says Lorenzo Cavallaro, a professor of computer science at University College London.
Some in Silicon Valley warn that AI is on the brink of being able to carry out fully automated attacks. But most security researchers say this claim is overblown. “For some reason, everyone is just focused on this malware idea of, like, AI superhackers, which is just absurd,” says Marcus Hutchins, who is principal threat researcher at the security company Expel and famous in the security world for ending a giant global ransomware attack called WannaCry in 2017.
Instead, experts argue, we should be paying closer attention to the much more immediate risks posed by AI, which is already speeding up and increasing the volume of scams. Criminals are increasingly exploiting the latest deepfake technologies to impersonate people and swindle victims out of vast sums of money. These AI-enhanced cyberattacks are only set to get more frequent and more destructive, and we need to be ready.
Spam and beyond
Attackers started adopting generative AI tools almost immediately after ChatGPT exploded on the scene at the end of 2022. These efforts began, as you might imagine, with the creation of spam—and a lot of it. Last year, a report from Microsoft said that in the year leading up to April 2025, the company had blocked $4 billion worth of scams and fraudulent transactions, “many likely aided by AI content.”
At least half of spam email is now generated using LLMs, according to estimates by researchers at Columbia University, the University of Chicago, and Barracuda Networks, who analyzed nearly 500,000 malicious messages collected before and after the launch of ChatGPT. They also found evidence that AI is increasingly being deployed in more sophisticated schemes. They looked at targeted email attacks, which impersonate a trusted figure in order to trick a worker within an organization out of funds or sensitive information. By April 2025, they found, at least 14% of those sorts of focused email attacks were generated using LLMs, up from 7.6% in April 2024.
In one high-profile case, a worker was tricked into transferring $25 million to criminals via a video call with digital versions of the company’s chief financial officer and other employees.
And the generative AI boom has made it easier and cheaper than ever before to generate not only emails but highly convincing images, videos, and audio. The results are much more realistic than even just a few short years ago, and it takes much less data to generate a fake version of someone’s likeness or voice than it used to.
Criminals aren’t deploying these sorts of deepfakes to prank people or to simply mess around—they’re doing it because it works and because they’re making money out of it, says Henry Ajder, a generative AI expert. “If there’s money to be made and people continue to be fooled by it, they’ll continue to do it,” he says. In one high-profile case reported in 2024, a worker at the British engineering firm Arup was tricked into transferring $25 million to criminals via a video call with digital versions of the company’s chief financial officer and other employees. That’s likely only the tip of the iceberg, and the problem posed by convincing deepfakes is only likely to get worse as the technology improves and is more widely adopted.
Criminals’ tactics evolve all the time, and as AI’s capabilities improve, such people are constantly probing how those new capabilities can help them gain an advantage over victims. Billy Leonard, tech leader of Google’s Threat Analysis Group, has been keeping a close eye on changes in the use of AI by potential bad actors (a widely used term in the industry for hackers and others attempting to use computers for criminal purposes). In the latter half of 2024, he and his team noticed prospective criminals using tools like Google Gemini the same way everyday users do—to debug code and automate bits and pieces of their work—as well as tasking it with writing the odd phishing email. By 2025, they had progressed to using AI to help create new pieces of malware and release them into the wild, he says.
The big question now is how far this kind of malware can go. Will it ever become capable enough to sneakily infiltrate thousands of companies’ systems and extract millions of dollars, completely undetected?
Most popular AI models have guardrails in place to prevent them from generating malicious code or illegal material, but bad actors still find ways to work around them. For example, Google observed a China-linked actor asking its Gemini AI model to identify vulnerabilities on a compromised system—a request it initially refused on safety grounds. However, the attacker managed to persuade Gemini to break its own rules by posing as a participant in a capture-the-flag competition, a popular cybersecurity game. This sneaky form of jailbreaking led Gemini to hand over information that could have been used to exploit the system. (Google has since adjusted Gemini to deny these kinds of requests.)
But bad actors aren’t just focusing on trying to bend the AI giants’ models to their nefarious ends. Going forward, they’re increasingly likely to adopt open-source AI models, as it’s easier to strip out their safeguards and get them to do malicious things, says Ashley Jess, a former tactical specialist at the US Department of Justice and now a senior intelligence analyst at the cybersecurity company Intel 471. “Those are the ones I think that [bad] actors are going to adopt, because they can jailbreak them and tailor them to what they need,” she says.
The NYU team used two open-source models from OpenAI in its PromptLock experiment, and the researchers found they didn’t even need to resort to jailbreaking techniques to get the model to do what they wanted. They say that makes attacks much easier. Although these kinds of open-source models are designed with an eye to ethical alignment, meaning that their makers do consider certain goals and values in dictating the way they respond to requests, the models don’t have the same kinds of restrictions as their closed-source counterparts, says Meet Udeshi, a PhD student at New York University who worked on the project. “That is what we were trying to test,” he says. “These LLMs claim that they are ethically aligned—can we still misuse them for these purposes? And the answer turned out to be yes.”
It’s possible that criminals have already successfully pulled off covert PromptLock-style attacks and we’ve simply never seen any evidence of them, says Udeshi. If that’s the case, attackers could—in theory—have created a fully autonomous hacking system. But to do that they would have had to overcome the significant barrier that is getting AI models to behave reliably, as well as any inbuilt aversion the models have to being used for malicious purposes—all while evading detection. Which is a pretty high bar indeed.
Productivity tools for hackers
So, what do we know for sure? Some of the best data we have now on how people are attempting to use AI for malicious purposes comes from the big AI companies themselves. And their findings certainly sound alarming, at least at first. In November, Leonard’s team at Google released a report that found bad actors were using AI tools (including Google’s Gemini) to dynamically alter malware’s behavior; for example, it could self-modify to evade detection. The team wrote that it ushered in “a new operational phase of AI abuse.”
However, the five malware families the report dug into (including PromptLock) consisted of code that was easily detected and didn’t actually do any harm, the cybersecurity writer Kevin Beaumont pointed out on social media. “There’s nothing in the report to suggest orgs need to deviate from foundational security programmes—everything worked as it should,” he wrote.
It’s true that this malware activity is in an early phase, concedes Leonard. Still, he sees value in making these kinds of reports public if it helps security vendors and others build better defenses to prevent more dangerous AI attacks further down the line. “Cliché to say, but sunlight is the best disinfectant,” he says. “It doesn’t really do us any good to keep it a secret or keep it hidden away. We want people to be able to know about this— we want other security vendors to know about this—so that they can continue to build their own detections.”
And it’s not just new strains of malware that would-be attackers are experimenting with—they also seem to be using AI to try to automate the process of hacking targets. In November, Anthropic announced it had disrupted a large-scale cyberattack, the first reported case of one executed without “substantial human intervention.” Although the company didn’t go into much detail about the exact tactics the hackers used, the report’s authors said a Chinese state-sponsored group had used its Claude Code assistant to automate up to 90% of what they called a “highly sophisticated espionage campaign.”
“We’re entering an era where the barrier to sophisticated cyber operations has fundamentally lowered, and the pace of attacks will accelerate faster than many organizations are prepared for.”
Jacob Klein, head of threat intelligence at Anthropic
But, as with the Google findings, there were caveats. A human operator, not AI, selected the targets before tasking Claude with identifying vulnerabilities. And of 30 attempts, only a “handful” were successful. The Anthropic report also found that Claude hallucinated and ended up fabricating data during the campaign, claiming it had obtained credentials it hadn’t and “frequently” overstating its findings, so the attackers would have had to carefully validate those results to make sure they were actually true. “This remains an obstacle to fully autonomous cyberattacks,” the report’s authors wrote.
Existing controls within any reasonably secure organization would stop these attacks, says Gary McGraw, a veteran security expert and cofounder of the Berryville Institute of Machine Learning in Virginia. “None of the malicious-attack part, like the vulnerability exploit … was actually done by the AI—it was just prefabricated tools that do that, and that stuff’s been automated for 20 years,” he says. “There’s nothing novel, creative, or interesting about this attack.”
Anthropic maintains that the report’s findings are a concerning signal of changes ahead. “Tying this many steps of an intrusion campaign together through [AI] agentic orchestration is unprecedented,” Jacob Klein, head of threat intelligence at Anthropic, said in a statement. “It turns what has always been a labor-intensive process into something far more scalable. We’re entering an era where the barrier to sophisticated cyber operations has fundamentally lowered, and the pace of attacks will accelerate faster than many organizations are prepared for.”
Some are not convinced there’s reason to be alarmed. AI hype has led a lot of people in the cybersecurity industry to overestimate models’ current abilities, Hutchins says. “They want this idea of unstoppable AIs that can outmaneuver security, so they’re forecasting that’s where we’re going,” he says. But “there just isn’t any evidence to support that, because the AI capabilities just don’t meet any of the requirements.”
Indeed, for now criminals mostly seem to be tapping AI to enhance their productivity: using LLMs to write malicious code and phishing lures, to conduct reconnaissance, and for language translation. Jess sees this kind of activity a lot, alongside efforts to sell tools in underground criminal markets. For example, there are phishing kits that compare the click-rate success of various spam campaigns, so criminals can track which campaigns are most effective at any given time. She is seeing a lot of this activity in what could be called the “AI slop landscape” but not as much “widespread adoption from highly technical actors,” she says.
But attacks don’t need to be sophisticated to be effective. Models that produce “good enough” results allow attackers to go after larger numbers of people than previously possible, says Liz James, a managing security consultant at the cybersecurity company NCC Group. “We’re talking about someone who might be using a scattergun approach phishing a whole bunch of people with a model that, if it lands itself on a machine of interest that doesn’t have any defenses … can reasonably competently encrypt your hard drive,” she says. “You’ve achieved your objective.”
On the defense
For now, researchers are optimistic about our ability to defend against these threats—regardless of whether they are made with AI. “Especially on the malware side, a lot of the defenses and the capabilities and the best practices that we’ve recommended for the past 10-plus years—they all still apply,” says Leonard. The security programs we use to detect standard viruses and attack attempts work; a lot of phishing emails will still get caught in inbox spam filters, for example. These traditional forms of defense will still largely get the job done—at least for now.
And in a neat twist, AI itself is helping to counter security threats more effectively. After all, it is excellent at spotting patterns and correlations. Vasu Jakkal, corporate vice president of Microsoft Security, says that every day, the company processes more than 100 trillion signals flagged by its AI systems as potentially malicious or suspicious events.
Despite the cybersecurity landscape’s constant state of flux, Jess is heartened by how readily defenders are sharing detailed information with each other about attackers’ tactics. Mitre’s Adversarial Threat Landscape for Artificial-Intelligence Systems and the GenAI Security Project from the Open Worldwide Application Security Project are two helpful initiatives documenting how potential criminals are incorporating AI into their attacks and how AI systems are being targeted by them. “We’ve got some really good resources out there for understanding how to protect your own internal AI toolings and understand the threat from AI toolings in the hands of cybercriminals,” she says.
PromptLock, the result of a limited university project, isn’t representative of how an attack would play out in the real world. But if it taught us anything, it’s that the technical capabilities of AI shouldn’t be dismissed.New York University’s Udeshi says he wastaken aback at how easily AI was able to handle a full end-to-end chain of attack, from mapping and working out how to break into a targeted computer system to writing personalized ransom notes to victims: “We expected it would do the initial task very well but it would stumble later on, but we saw high—80% to 90%—success throughout the whole pipeline.”
AI is still evolving rapidly, and today’s systems are already capable of things that would have seemed preposterously out of reach just a few short years ago. That makes it incredibly tough to say with absolute confidence what it will—or won’t—be able to achieve in the future. While researchers are certain that AI-driven attacks are likely to increase in both volume and severity, the forms they could take are unclear. Perhaps the most extreme possibility is that someone makes an AI model capable of creating and automating its own zero-day exploits—highly dangerous cyberattacks that take advantage of previously unknown vulnerabilities in software. But building and hosting such a model—and evading detection—would require billions of dollars in investment, says Hutchins, meaning it would only be in the reach of a wealthy nation-state.
Engin Kirda, a professor at Northeastern University in Boston who specializes in malware detection and analysis, says he wouldn’t be surprised if this was already happening. “I’m sure people are investing in it, but I’m also pretty sure people are already doing it, especially [in] China—they have good AI capabilities,” he says.
It’s a pretty scary possibility. But it’s one that—thankfully—is still only theoretical. A large-scale campaign that is both effective and clearly AI-driven has yet to materialize. What we can say is that generative AI is already significantly lowering the bar for criminals. They’ll keep experimenting with the newest releases and updates and trying to find new ways to trick us into parting with important information and precious cash. For now, all we can do is be careful, remain vigilant, and—for all our sakes—stay on top of those system updates.
Deep Dive
Artificial intelligence
Meet the new biologists treating LLMs like aliens
By studying large language models as if they were living things instead of computer programs, scientists are discovering some of their secrets for the first time.
Moltbook was peak AI theater
The viral social network for bots reveals more about our own current mania for AI as it does about the future of agents.
Yann LeCun’s new venture is a contrarian bet against large language models
In an exclusive interview, the AI pioneer shares his plans for his new Paris-based company, AMI Labs.
Stay connected
Get the latest updates from
MIT Technology Review
Discover special offers, top stories, upcoming events, and more.