Artificial Intelligence1 min read

Perplexity and Intelligence

By · Published by Everything Blog

In short

Tracking perplexity offers a durable metric for measuring model progress, especially as conventional downstream benchmarks quickly saturate or become obsolete. Modern artificial intelligence models historically unlock step-change abilities at specific predictive milestones, ranging from coherent paragraph generation to advanced chain-of-thought reasoning. Most recently, advanced models operating below a 0.8 loss threshold have exhibited transformative aptitudes in cybersecurity and mathematics. Looking forward, these development trajectories may accelerate further if models begin directly assisting in pretraining algorithmic research.

Key points

  • Perplexity functions as a lasting foundational measurement that outlives rapidly saturati…: Perplexity functions as a lasting foundational measurement that outlives rapidly saturating downstream artificial intelligence benchmarks.
  • Successive improvements in foundational token prediction consistently unlock advanced lat…: Successive improvements in foundational token prediction consistently unlock advanced latent capabilities like in-context learning without requiring disparate, bespoke tools.
  • Development cycles could experience a massive shift if artificial intelligence systems be…: Development cycles could experience a massive shift if artificial intelligence systems begin contributing directly to pretraining algorithmic research.

Sources checked by our analysis pipeline: 1

From afar, it might seem like modern AI is built similarly to how software has been built in previous decades: with a diverse set of techniques targeting specific capabilities. There are indeed a variety of methods, but rather than designing something from scratch, they are mostly harnessing the intelligence coming from a much simpler and less interpretable process: training a compressor (or, equivalently(opens in a new window), predictor) on a very large corpus of data.1

I believe tracking the pace of progress on training better compressors is key to understanding the arc of advances in AI. In this post, I argue for establishing perplexity (or, equivalently, cross-entropy loss) as a metric to compare between model developers and inform shared safety standards. As a first step, I provide more transparency into pretraining progress at OpenAI. Publishing our internal perplexity measurements took some consideration; it gives competitors a clearer picture of where we stand, but that concern is outweighed by the urgency of establishing shared(opens in a new window) safety measures across the industry. We at OpenAI hope for this to be…

Artificial Intelligence