Multitask Learning

Multitask Learning


Multitask learning trains one model on several tasks at the same time. The model has a separate output head per task while sharing its lower layers. Those shared layers learn what the tasks have in common and the heads make the task-specific distinctions.

The reasoning is that related tasks need similar representations. Instead of building three separate models to predict a text's sentiment, its topic and its urgency, one model learns the general structure of the language once. A task with little data benefits from the representation learned by a task with plenty. Shared training also acts as a kind of regularizer and can reduce overfitting.

Balancing causes trouble in practice. When task losses sit at different scales one drowns out the others, so the losses get weighted. If the tasks are too far apart, sharing hurts instead of helping, which is called negative transfer.

A concrete case: an ecommerce site predicts both the sentiment of a product review and which product attribute the review is about. Trained as one model, accuracy on attribute prediction rises above the standalone model despite the small amount of data available for it.

Improvement is not guaranteed. Pick the wrong combination of tasks and overall performance drops, so each task is measured separately.

From generative AI strategy to custom agent development and retrieval architectures, we help you scale AI responsibly.
Discuss your AI project