Transformers are the neural network architecture built for sequences, and nearly every current language model sits on top of it. It was introduced in the 2017 paper "Attention Is All You Need". What separates it from the recurrent networks that came before is simple: instead of reading text one step at a time, it can look at every token at once.
Self-attention is the mechanism that makes that work. Each token computes how much attention it should pay to every other token in the sequence. In "the bank was steep and muddy", the meaning of "bank" is settled by "steep" and "muddy", and the model puts its weight there. Stack the layers and the representations get more abstract. Order has to be supplied separately through positional encoding, because attention itself has no sense of sequence.
The architecture has an encoder block and a decoder block. Models like BERT use only the encoder and focus on understanding tasks, the GPT family uses only the decoder and generates, and translation models use both together.
Running in parallel is what decided its practical fate. Because all tokens are processed at once, training spreads across GPUs, which is what made much larger datasets and much larger models possible. The same architecture now carries images, audio and code as well as text.
The cost lives in attention itself. Computation grows with the square of sequence length, which makes long context expensive and is the technical reason context windows have limits.