Tracking perplexity offers a durable metric for measuring model progress, especially as conventional downstream benchmarks quickly saturate or become obsolete. Modern artificial intelligence models historically unlock step-change abilities at specific predictive milestones, ranging from coherent paragraph generation to advanced chain-of-thought reasoning. Most recently, advanced models operating below a 0.8 loss threshold have exhibited transformative aptitudes in cybersecurity and mathematics. Looking forward, these development trajectories may accelerate further if models begin directly assisting in pretraining algorithmic research.
Key points
Perplexity functions as a lasting foundational measurement that outlives rapidly saturati…: Perplexity functions as a lasting foundational measurement that outlives rapidly saturating downstream artificial intelligence benchmarks.
Successive improvements in foundational token prediction consistently unlock advanced lat…: Successive improvements in foundational token prediction consistently unlock advanced latent capabilities like in-context learning without requiring disparate, bespoke tools.
Development cycles could experience a massive shift if artificial intelligence systems be…: Development cycles could experience a massive shift if artificial intelligence systems begin contributing directly to pretraining algorithmic research.
Sources checked by our analysis pipeline: 1
From afar, it might seem like modern AI is built similarly to how software has been built in previous decades: with a diverse set of techniques targeting specific capabilities. There are indeed a variety of methods, but rather than designing something from scratch, they are mostly harnessing the intelligence coming from a much simpler and less interpretable process: training a compressor (or, equivalently(opens in a new window), predictor) on a very large corpus of data.1
I believe tracking the pace of progress on training better compressors is key to understanding the arc of advances in AI. In this post, I argue for establishing perplexity (or, equivalently, cross-entropy loss) as a metric to compare between model developers and inform shared safety standards. As a first step, I provide more transparency into pretraining progress at OpenAI. Publishing our internal perplexity measurements took some consideration; it gives competitors a clearer picture of where we stand, but that concern is outweighed by the urgency of establishing shared(opens in a new window) safety measures across the industry. We at OpenAI hope for this to be…
Originally created by a Chinese engineer as a defensive testing framework, the open-source software links prominent artificial intelligence models to existing penetration utilities. Cybercriminals have repurposed this integration to orchestrate end-to-end network intrusions through simple plain-language commands, granting them rapid unauthorized administrative access. The rapid machine-speed execution presents emerging risks for interconnected digital hubs like Singapore, even where local breaches have not yet mat…
Artificial intelligence developers are confronting rapid capability growth by evaluating formal constraints on future model development. Jakub Pachocki of OpenAI suggested establishing thresholds on performance indicators to manage the trajectory of technological progress. This governance proposal seeks to proactively mitigate the impact of an anticipated flood of breakthroughs over the next three years.
Indian telecommunications operators successfully confirmed network resilience after a state-ordered evaluation designed to evaluate defense capabilities against sophisticated cyberthreats. The proactive review was triggered by the debut of advanced models like Anthropic’s Claude Mythos, which sparked concerns over potential infrastructure exposure. Consequently, domestic networks remain secure while companies establish tighter internal detection mechanisms to prevent future breaches.