Attention: how transformers decide which words matter

To predict the next word well, a model must use context. In "The trophy did not fit in the suitcase because it was too big", the word "it" refers to the trophy, and you need earlier words to work that out. Modern language models are built on the transformer architecture, and its key ingredient for using context is attention.