Excellent build up of educational content and papers for building a solid understanding of Transformer architecture and its later developments by Jay Alammar

Start with - A pictorial tutorial for the "Attention is all you need" paper where sensational "transformer architecture" in deep learning is first described (Jay Alammar)

And then proceed to

• Read the Attention Is All You Need paper, the Transformer blog post (Transformer: A Novel Neural Network Architecture for Language Understanding), and the Tensor2Tensor announcement.

• Watch Łukasz Kaiser’s talk walking through the model and its details

• Play with the Jupyter Notebook provided as part of the Tensor2Tensor repo

• Explore the Tensor2Tensor repo.

Follow-up works:

• Depthwise Separable Convolutions for Neural Machine Translation

• One Model To Learn Them All

• Discrete Autoencoders for Sequence Models

• Generating Wikipedia by Summarizing Long Sequences

• Image Transformer

• Training Tips for the Transformer Model

• Self-Attention with Relative Position Representations

• Fast Decoding in Sequence Models using Discrete Latent Variables

• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost