Sequence-to-Sequence (Seq2Seq) Models

Sequence-to-Sequence (Seq2Seq) Models


Seq2seq models take a sequence as input and return another sequence as output. The two do not have to be the same length. A seven-word sentence can become a twelve-word translation, and that flexibility is the reason the architecture exists.

The structure has two parts. An encoder reads the input and compresses it into a representation, and a decoder produces the output step by step from that representation. First-generation models used recurrent networks and lost information on long sentences, because the whole input had to fit into one vector. Adding attention fixed that by letting the decoder look at different parts of the input at each step. The transformer architecture is the continuation of that line.

The task list is broad. Machine translation, summarisation, question answering, grammar correction, speech to text and code generation all fit the same mould. What they share is an ordered input and an ordered output.

A concrete case: on an ecommerce site, the messy product titles sellers type get converted into a standard form. The input is "red womens sport shoe size 38", the output is a tidy title carrying brand, category, colour and size fields. The model learns the transformation from examples with no rules written by hand.

Evaluation is awkward because there is no single correct output. Translation and summarisation use measures like BLEU and ROUGE, and neither fully captures closeness in meaning.

From generative AI strategy to custom agent development and retrieval architectures, we help you scale AI responsibly.
Discuss your AI project