Autoregression is predicting a variable's future value from its own past values. In statistics it is the basis of time series modelling. A model that forecasts today's sales by looking at yesterday's, the day before's and older figures is an autoregressive model.
Language models apply the same idea to text. The model predicts the next token by looking at every token produced so far. That token gets appended to the input and the loop starts again. The GPT family works this way, which is why these are called autoregressive models. It is also why text is produced left to right and why the model cannot go back and revise what it already wrote.
The strength of the approach is its simplicity. A complex output is built one decision at a time, each decision resting on the ones before. The weakness sits in the same place: an error early on carries forward and compounds.
A concrete case: a retail chain forecasts weekly demand. An autoregressive model predicts next week from the sales of previous weeks. When the model feeds its own forecast in as the input for the following step, the prediction four weeks out is visibly more uncertain, because every intermediate error travels with it.
Autoregression in time series carries an assumption: that the past pattern still holds. When the underlying structure breaks, the model does not catch it straight away.