Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Mamba is a new sequence modeling approach that achieves linear-time complexity by selectively updating the state space during inference. It outperforms existing models on various sequence tasks.

Details

Mamba is a new sequence modeling approach that aims to achieve linear-time complexity during inference, in contrast to the quadratic or cubic complexity of traditional sequence models. The key innovation is a selective state space mechanism that only updates a subset of the state variables at each time step, rather than the entire state. This allows Mamba to maintain an accurate representation of the sequence while dramatically reducing the computational cost. Experiments show that Mamba outperforms existing models like Transformers and LSTMs on tasks like language modeling, while being significantly more efficient. The selective state space technique is a promising direction for building fast and scalable sequence models, with potential applications in natural language processing, speech recognition, and other domains.