From Embeddings to Transformers: The Mathematics of Recommender Systems Through Inner Products

H2. Why the Embedding Gives That Rating

When an embedding is called but you can’t explain why that rating appears, this course connects the calculation in one continuous line. Starting with principal component analysis and moving through singular value decomposition, matrix factorization, and Transformer attention, you’ll follow the process using a single movie rating. This isn’t time spent memorizing library names; it’s time spent explaining in words how the same dot product becomes a recommendation score.

H2. The Computation Connects in a Single Line

Insert the lesson sequence diagram directly below this heading. The alt text is a lesson sequence showing principal component analysis, singular value decomposition, matrix factorization, and transformers connected by arrows. The file is 인프런_수업순서_PCA_SVD_MF_Transformer.png in the Downloads folder.

H3. Principal Component Analysis, the Axis Where Information Concentrates

Components are independent variables. The degree to which the data is spread out is variance, and its force is the inner product. Covariance is calculated by transposing the data and then multiplying it. Like Pareto’s principle, information is concentrated along one or two axes. The length of each axis is its eigenvalue, and the ratio of its direction is its eigenvector. Eigenvalues represent information, while eigenvectors represent weights. We confirm this with the ellipse formed by height and weight and the petal lengths of irises. Since people can only see two axes, the graph retains only the axes with the greatest amount of information. When groups overlap, we reconstruct the data using LDA. It is an axis that makes the means far apart and the spread within each group small.

H3. Singular Value Decomposition: Reconstructing Missing Values

In movie ratings, missing values are a problem. Customers watch only some movies and leave ratings. SVD fills in those missing values with averages, then calculates the eigenvalues for the customer side and the movie side, respectively. Keeping only the axes with the most information reduces the dimensionality and generates new ratings for the previously empty cells. If a movie Hong Gil-dong has not seen is predicted to be around 4 points, it is recommended; if it falls below that, it is excluded—this is how initial recommendations are made. If the average difference between past ratings and predicted ratings is small, the system can be trusted. This method compares the actual scores of movies already watched with the reconstructed scores.

H3. Matrix Factorization: Train Only on Ratings for Movies That Were Watched

Values filled in with averages are not ratings given by that person. The training data should include only records of items actually viewed. Therefore, blank entries are excluded from training. For entities such as movies and customers whose dimensions are unknown, attributes are assigned arbitrarily. Embedding is the process of setting those numbers to be roughly distributed with a mean of 0 and a variance of 1, while making them different from one another. Multiplying the inner product of the two by a weight produces a rating. When it differs from the actual score, only the weights are adjusted to reduce the difference. The saved weights are the model. This training is called matrix factorization.

H3. Transformers and Attention

If we change the question, we arrive at the final calculation. Asking for the rating of a movie Hong Gil-dong has not watched is the Query. A person who gave similar ratings in the past is the Key. Dividing the degree of similarity from the dot product by the total sum yields probabilities, and when the numbers become large, they are reduced using softmax. The rating given by that person is the Value. Multiplying the probabilities by the ratings and adding them together produces the expected score. This is attention, and it is the same calculation as an expected value. In real-world tables with many blank cells, it is difficult to create attributes using only the movies someone has watched, so embeddings are used. Weights are applied separately to the query, key, and value, increasing the computational load. This structure is the transformer, and its evaluation method is attention-model.

H2. Recommendations Through Movie Ratings

This flow is the calculation behind movie-rating recommender systems. In the class, we follow the history of how Netflix and YouTube recommendations started here and progressed to matrix factorization and transformers. We also look at how close an error of around 0.4 is on a five-point scale, how the error decreases when the data grows from 100 to 1,000 people, and how averaging multiple errors in a batch helps avoid fitting everything to just a single rating.

H2. Who This Course Is For

This is for those who cannot explain why embedding numbers appear, those who trust recommendation results after filling missing ratings with the average, and those who know Query, Key, and Value only by name but do not know how to calculate them. It will be easier to follow if you know the rows and columns of matrices, multiplication, and transposition. If you have ever run a Python file, you can work through the exercises along with the class. The exercise code is obtained by entering the conditions mentioned in class into ChatGPT.

H2. Order Covered in the Class

The class is divided into three parts. First is principal component analysis. We cover why we start with PCA, eigenvalues and eigenvectors, dimensionality reduction, and LDA. Second is singular value decomposition. We look at SVD for filling in missing values and why we move on to matrix factorization after examining the error. Third is matrix factorization and transformers. We examine the learning curve, calculate ratings using attention, and then summarize everything.

H2. How to Watch the Class Videos

Each class video is linked through iframes in the class notes and played in the classroom. This video does not cover logging in, signing up, or building a service with Amazon data. It is the next step after understanding the calculations.