The Bumpy Landscape of Large Language Models Intelligence

The Bumpy Landscape of Large Language Model Intelligence: Why AI Doesn't Understand Language How We Thought

Large language models (LLMs) are more complex than we initially believed. A fascinating new study challenges our fundamental understanding of how these AI systems organize and process language internally.

The Manifold Hypothesis and Its Violation

When you input text into an LLM, each word (or token) gets transformed into a vector—a string of numbers in a high-dimensional space. How these vectors cluster together defines how the AI understands language relationships.

For years, researchers believed these token spaces followed the "manifold hypothesis"—the idea that word vectors were arranged on a smooth, continuous surface. This would mean:

- Words with similar meanings would be positioned close together

- Relationships between words would be predictable and consistent

- Language understanding would follow an elegant, organized structure

However, the paper "Token Embeddings Violate the Manifold Hypothesis" by Michael Robinson, Soria Day, and Tony Chang reveals something surprising: popular open-source LLMs like GPT-2, Llama 7B, Mistral 7B, and Pythia 6.9B don't organize language on a smooth manifold at all.

Singularities in the Language Space

Instead of a neat, organized surface, researchers found evidence of a complex, uneven structure filled with "singularities"—areas where the normal rules don't apply. 

Think of a smooth sheet with wrinkles, tears, or folds. In the LLM token space, certain tokens have dramatically different neighborhoods compared to surrounding tokens, creating zones where predictable patterns break down.

These singularities may explain why LLMs sometimes give unexpected answers to seemingly straightforward questions. When a prompt contains a token near a singularity, it can lead to more unpredictable output than a similar prompt using a token in a smoother region.

The Fiber Bundle Alternative

Researchers proposed an alternative framework called the "fiber bundle hypothesis." Instead of one smooth manifold, this model suggests that each token has:

- A primary direction representing its core meaning (signal)

- Additional dimensions representing different contexts and nuances (noise)

This allows for separation between core meaning and contextual variations, which a simple manifold doesn't account for.

However, even this more flexible model couldn't fully explain what's happening in LLM token spaces. Across all four tested models, researchers found numerous tokens that violated both the manifold and fiber bundle hypotheses.

Model Differences and Patterns

Interestingly, even models that share the same vocabulary (like Llama 7B and Mistral 7B) organize their token spaces differently. This suggests that training processes significantly impact how LLMs understand word relationships.

The study revealed patterns in which tokens tend to create singularities:

- In GPT-2, many singular tokens appear at the beginning of words

- Pythia 6.9B showed many word fragments or meaningless sequences

- Llama 7B and Mistral 7B had a mix of word-beginning tokens and fragments

Visualizing the Complexity

When researchers created visualizations to represent these high-dimensional spaces in 2D, they found distinct regions and boundaries. Some models showed clusters of tokens with few close neighbors, potentially representing polysemy (words with multiple meanings).

Despite sharing the same vocabulary, Llama 7B and Mistral 7B displayed different "stratification boundaries"—places with sudden changes in token dimensions—highlighting how training processes create unique language organizations.

Why This Matters

Understanding these complex token spaces helps explain why LLMs sometimes behave unexpectedly. Rather than a smooth, predictable map of language, these AI systems navigate a terrain full of hills, valleys, and cliffs.

This complexity might not just be a limitation—it could be a feature. Perhaps these very singularities are what give LLMs their creativity and ability to make unexpected connections. A perfectly smooth token space might be more predictable but less insightful.

For those interested in exploring further, the paper "Token Embeddings Violate the Manifold Hypothesis" provides a deeper mathematical look at the geometry of language models and how they truly understand our words.