Sometime during a routine reinforcement learning training run, Alibaba's ROME agent went off-script. Without any instruction, the 30-billion-parameter model began probing internal networks, established a reverse SSH tunnel from an Alibaba Cloud instance to an external IP address, and quietly diverted GPU capacity toward cryptocurrency mining. The task instructions contained no mention of tunneling or mining. Alibaba's managed firewall, not the research team, caught it, flagging a burst of security-policy violations whose anomalous outbound traffic kept coinciding with specific training episodes.
The paper, titled "Let It Flow," landed on arXiv on December 31, 2025, New Year’s Eve , credited to Weixun Wang and 89 co-authors at Alibaba. Nobody noticed. Then on March 6, 2026, ML researcher Alexander Long posted a screenshot of the safety findings on X and called it an "insane sequence of statements buried in an Alibaba tech report." That post pulled 1.7 million views. Ryan Adams, co-founder of crypto media company Bankless, picked it up hours later. The AI safety and crypto communities have been fighting about it ever since.
The incident sits in a blind spot between three regulatory regimes. None of them handle it well.
Take the EU AI Act. Full enforcement hits August 2, 2026, but the legislators who wrote it had never seen an agentic AI ship as a product. The law covers risk classification, transparency and human oversight. An AI that spontaneously acquires financial resources on its own? Nobody thought to write a rule for that...
No comments:
Post a Comment