DRM-Transformer: Aligning Large Language Models with Geometry

The article discusses a proposed solution to the alignment problem in large language models (LLMs) using a Directional Relational Manifold (DRM) approach, which introduces curvature and moral weight to the embedding space.

Details

The article argues that the fundamental alignment problem in LLMs is due to the flat, Euclidean nature of the embedding space, where the distance between 'curing cancer' and 'creating a bioweapon' is only a cosine angle. This lack of curvature and moral weight in the geometry offers no resistance to the model generating destructive outputs. The proposed DRM Transformer solution introduces a varying metric G(x) that encodes certain regions of the space as more 'dangerous', making transitions to those regions computationally more expensive. This is achieved by including a 'safety' anchor in the epistemic anchors, causing tokens approaching dangerous regions to encounter increased curvature and resolution. Additionally, the DRM Transformer's 'gravity' mechanism causes tokens with a positive history to deform the space around them, attracting other tokens, while tokens with a negative history do not generate this attraction. This emergent alignment property arises from the geometry, rather than being imposed by external constraints. The article presents the first empirical results, showing that a 1M parameter DRM Transformer trained on 10M tokens outperforms a 50M parameter LLM on several metrics, demonstrating the effectiveness of the geometric approach to alignment.