The differences are pretty subtle... If you want an overview of the whole zoo, see https://lilianweng.github.io/lil-log/2020/04/07/the-transfor... https://www.pragmatic.ml/a-survey-of-methods-for-incorporati... https://arxiv.org/abs/2009.06732#google https://www.gwern.net/GPT-2#efficient-attention