Attention

Hymba is a novel architecture for small language models that combines transformer attention mechanisms with state space models (SSMs) in a hybrid-head parallel structure.