Technology1 min read

Was software just always too complex for humans?

By · Published by Everything Blog

In short

Recent breakthroughs, including solving the Riemann hypothesis with 17 million lines of AI-assisted code, expose the reality that software scale has outpaced human comprehension. To resolve modern security failures across foundational stacks like Linux, developers propose transitioning to machine-checked formal specifications and simplified architectures like unikernels. This framework would automate patching entirely while human operators focus on co-writing specifications alongside AI. Additionally, automated agents could generate diverse software implementations from these models to prevent catastrophic monoculture exploits.

Key points

  • Transitioning to machine-checked formal specifications allows automated agents to handle…: Transitioning to machine-checked formal specifications allows automated agents to handle patching without relying on human maintainer triage.
  • Generating diverse software implementations from identical specifications can prevent sin…: Generating diverse software implementations from identical specifications can prevent single exploits from compromising vast numbers of hosts.
  • Relying on a small pool of proprietary frontier AI models introduces systemic risks that…: Relying on a small pool of proprietary frontier AI models introduces systemic risks that demand diversity among underlying architectures.

Sources checked by our analysis pipeline: 1

I was glued to the Lean chat as the legendary Riemann hypothesis was cracked via 17 million (!) lines of AI-assisted code last week. This has since been certified correct despite no human (yet) understanding how it works. The future of maths, and much science, is clearly going to involve human-AI boosting.

However, it also struck me deeply just how broken software engineering has been for my entire life! There's always been an imbalance between our understanding of the theory of computation and our relentless drive to deploy software throughout society. We're paying the piper now as all those decades of human-written code turn out to be full of bugs. Since software essentially drives society now, weird things happen like supermarket fridges needing James Bond to secure them. This has led to people like Tim Gowers joining the Pause AI march.

How the hell did we get here? Have we just completely overestimated the human ability to write software? Or will software engineering 'catch up' once LLM harnesses get better?

Software was reasonably under control in pre-AI times when attackers…

Technology