Learning Transformer by Implementing

You will learn about Transformers inside out by directly implementing everything from Multi-Head Attention to the Original Transformer model, BERT, the Encoder-Decoder-based MarianMT translation model, and the Vision Transformer in code.

(5.0) 24 reviews

342 learners

Level Intermediate

Course period Unlimited

Deep Learning(DL)
Deep Learning(DL)
PyTorch
PyTorch
encoder-decoder
encoder-decoder
bert
bert
transformer
transformer
Deep Learning(DL)
Deep Learning(DL)
PyTorch
PyTorch
encoder-decoder
encoder-decoder
bert
bert
transformer
transformer

Reviews from Early Learners

5.0

5.0

김재준

100% enrolled

I took this course blindly without completing the prerequisites first. It was difficult and challenging at first, but as I kept listening, it became interesting, and now I am going to take the prerequisite courses.

5.0

승훈

60% enrolled

I am gaining a detailed understanding of Transformers thanks to the code and theoretical explanations.

5.0

jsmak

100% enrolled

I have taken several online courses on Transformers, and this one is the most detailed. Thank you for always providing such thorough explanations.

What you will gain after the course

  • Mastering Transformer's Self, Causal, and Cross Attention Mechanisms through Direct Implementation

  • Original Transformer model architecture learned by directly implementing Positional Encoding, Feed Forward, Encoder, Decoder, and more

  • Prerequisite knowledge for Transformer, including NLP foundations like tokenization and embedding, and RNN models.

  • Implement the BERT model from scratch and apply it to sentence classification training.

  • MarianMT model, an Encoder-Decoder translation model implemented directly

  • Understanding and Utilizing Hugging Face Dataset, Tokenizer, and DataCollator

  • Training of the Encoder-Decoder MarianMT Model and Greedy and Beam Search Inference

  • Implement Vision Transformer (ViT) from scratch and train an image classification model with custom data.

Master Transformer with just this one course!

In this course, you can learn Transformer, the core of modern AI, by implementing it yourself.

I have structured this course as a full Transformer curriculum where you understand the concepts by implementing them in code, covering everything from Multi-Head Attention—the core mechanism of Transformer—to the Original Transformer model, BERT, the Encoder-Decoder MarianMT translation model, and Vision Transformer.

Features of this course

💡 Learning Transformer by Implementing from Scratch with Code

From Multi-Head Attention, the core mechanism of Transformer, to the original Transformer model, BERT, and the encoder-decoder translation model MarianMT, by implementing them directly in code, you will learn about Transformer inside and out.

💡 Step-by-step learning from NLP foundations to core Transformer models

To understand Transformer, it is important to understand the foundations of NLP first.

Starting from tokenization and embedding, and RNN models prior to Transformer, all the way to Attention -> Transformer -> BERT -> MarianMT translation model -> Vision Transformer, in a single flow, the course is structured to enable step-by-step learning from a solid NLP foundation to core Transformer models.

💡 Balance between theory and implementation

We don't just focus on implementation. To ensure the core mechanisms that make up the Transformer are easy to understand and stick in your mind, I spent a lot of time brainstorming ideas and creating this course. This course, which leads from easy and detailed theoretical explanations to actual code implementation, will help your Transformer application skills improve by leaps and bounds.

💡 Presenting core NLP problem-solving processes

We specifically cover elements encountered in actual research and practice, such as embedding, padding masking, various types of attention, loss calculation for padded labels, and dynamic padding, and provide the process for solving them.


💡 Utilizing Key Hugging Face Libraries

Hugging Face is an indispensable library for utilizing Transformers. In this course, we use Hugging Face's Tokenizer, Dataset, DataCollator, and more to perform data processing for Transformer model training—such as data preprocessing, tokenization, dynamic padding, and conversion of label values and decoder input values—how easily and conveniently this can be handled, providing a detailed guide.

What you will learn

NLP Foundations and RNNs for Transformer Prerequisites

We provide a summary of the prerequisite knowledge for learning Transformers, including tokenization, embedding, RNN and Seq2Seq models, and the basics of Attention.

Transformer Core Mechanisms and Key Modules

You can gain a clear understanding of core attention mechanisms such as Self Attention, Causal Attention, and Cross Attention, as well as key Transformer modules like Positional Encoding, Layer Normalization, Feed Forward, and Encoder/Decoder Layers through detailed theory and hands-on practice.

Utilizing Hugging Face Tokenizer, Dataset, and DataCollator

We provide detailed explanations of the features, advantages, and usage of Hugging Face's Dataset, Tokenizer, and DataCollator. Additionally, you will gain hands-on experience through various exercises and examples on how to effectively perform data pipeline processing for Transformer NLP models by combining these components.

Implementation and Application of the BERT Model

You will learn BERT by directly implementing the key components of the BERT model. Furthermore, you can learn how to apply model training and inference for sentence classification using the BERT you implemented and various features provided by Hugging Face.

Implementation and Application of Encoder-Decoder based MarianMT Translation Model

You will directly implement the MarianMT model, an Encoder-Decoder-based Korean-English translation model, learn various data preprocessing methods and techniques required for training Encoder-Decoder models, and learn how to implement and apply Auto-Regressive-based Greedy Search and Beam Search.

Implementation and Training of the Vision Transformer Model

You will directly implement the Vision Transformer, which established the Transformer as a model comparable to CNNs in the vision domain, and train it using custom datasets. By implementing the key modules of the Vision Transformer yourself, you can easily learn and understand the features and mechanisms of ViT.

Things to know before taking the course

Practice Environment 💾

The practice environment is conducted using notebook kernels provided by Kaggle. After signing up for Kaggle and selecting the Code menu, you can use a P100 GPU for free for 30 hours per week in a Jupyter Notebook environment similar to Colab.


A 160-page lecture textbook is provided.

Recommended for
these people

Who is this course right for?

  • Deep learning NLP beginners who want to build a solid foundation by implementing everything from tokenization to RNNs and Transformers directly in code.

  • Those who want to gain a deep understanding of the Transformer architecture by implementing its internal mechanisms directly, rather than simply using a library.

  • Those who want to understand the core mechanisms of Transformers more easily through a balanced approach to theory and practice.

  • Developers who want to build a solid foundation in Attention and Transformer mechanisms for AI service development.

  • For those who want high-quality, end-to-end practical project experience ranging from Transformer basics to text classification and translation models.

Need to know before starting?

  • Deep Learning CNN Complete Guide - PyTorch Version

Hello
This is dooleyz3525

Inflearn Verified

28,445

Learners

1,573

Reviews

4,085

Answers

4.9

Rating

15

Courses

(Former) Encore Consulting | (Former) Oracle Korea | Author of "Python Machine Learning Perfect Guide"

AI Freelance Consultant

 

More

Curriculum

All

145 lectures ∙ (28hr 9min)

Course Materials:

Lecture resources
Published: 
Last updated: 

Reviews

All

24 reviews

5.0

24 reviews

  • vjeong71170433님의 프로필 이미지
    vjeong71170433

    Reviews 14

    Average Rating 3.9

    5

    26% enrolled

    I've taken various Transformer courses, and this one is truly the best. Unlike courses that only review papers, this one incorporates the instructor's intuitive explanations into both the code and PPT. Understanding comes so easily when viewing it alongside the code. I plan to confidently enroll in other courses in the future. This is truly a legendary course that stands out among the best.

    • dooleyz3525
      Instructor

      Oh, my heart feels so full. I think I'm going to be so happy today. Thank you for writing such a wonderful review.

  • jcy40236396님의 프로필 이미지
    jcy40236396

    Reviews 7

    Average Rating 5.0

    5

    100% enrolled

    Thank you always for the great lectures!

    • dooleyz3525
      Instructor

      You're very welcome ^^, I'm actually the one who's grateful to you for writing such a nice course review.

  • seunghun32461674님의 프로필 이미지
    seunghun32461674

    Reviews 2

    Average Rating 5.0

    5

    60% enrolled

    I am gaining a detailed understanding of Transformers thanks to the code and theoretical explanations.

    • dooleyz3525
      Instructor

      I'm glad I could be of help. Thank you so much for the great course review.

  • muscelelg님의 프로필 이미지
    muscelelg

    Reviews 8

    Average Rating 5.0

    5

    100% enrolled

    I took this course blindly without completing the prerequisites first. It was difficult and challenging at first, but as I kept listening, it became interesting, and now I am going to take the prerequisite courses.

    • jsmak6174님의 프로필 이미지
      jsmak6174

      Reviews 8

      Average Rating 5.0

      5

      100% enrolled

      I have taken several online courses on Transformers, and this one is the most detailed. Thank you for always providing such thorough explanations.

      • dooleyz3525
        Instructor

        Oh, thank you for the great review. Since the Transformer can be difficult for those encountering it for the first time, some parts of the explanation might have been a bit excessive. I will keep that in mind when creating the next lecture.

    dooleyz3525's other courses

    Check out other courses by the instructor!

    Similar courses

    Explore other courses in the same field!