Human-Aligned Decision Transformers for Deep-Sea Habitat Design

The article explores the use of Decision Transformers, a novel reinforcement learning approach, to design AI systems that can make complex, sequential decisions in extreme environments like deep-sea exploration, where human oversight is critical but real-time communication is impossible.

Details

The author's research journey began with exploring reinforcement learning architectures for autonomous systems, leading to the challenge of designing AI systems that can make complex, sequential decisions in environments where human oversight is critical but real-time communication is impossible. This investigation led the author to the domain of deep-sea exploration, where habitat design decisions must balance structural integrity, life support optimization, and crew safety under constantly changing conditions. Traditional reinforcement learning approaches failed to incorporate human preferences into long-horizon decision sequences, prompting the author to combine Decision Transformers with human preference alignment techniques to create the Human-Aligned Decision Transformers (HADT) architecture. The key insight was conditioning the model on both return-to-go (RTG) and human preference embeddings, enabling the model to learn to generate trajectories that satisfy both performance metrics and human-aligned constraints.