Language-Agnostic Representations Show a Shared Semantic Workspace

Recent research suggests that large language models (LLMs) may use a shared semantic workspace that is partly separate from the input language, challenging the view that they primarily operate on linguistic representations.

Details

The article discusses the emerging evidence for 'language-agnostic representations' in large language models (LLMs). Contrary to the view that LLMs primarily operate on linguistic representations, recent research indicates that their internal processing may involve a shared semantic workspace that is partly separate from the input language. Experiments show that semantically equivalent inputs in different languages tend to converge in the middle layers of LLMs, even when the surface form changes from text to code or equations. Activation patching studies further demonstrate that output language is encoded earlier than the underlying concept, and that averaging concept representations across languages can improve translation performance. This suggests that concept and language are at least partly separable within transformer-based LLMs, providing new insights into their internal representations and processing. While not proving that LLMs 'think' like humans, these findings challenge the standard view of LLMs as primarily linguistic systems and point to a more nuanced mental model where language is the input/output, while the middle layers capture a shared semantic workspace.