How do we compute Q, K, and V in attention?

The computation of Q (Query), K (Key), and V (Value) in attention mechanisms is primarily achieved through matrix multiplication with learnable weight matrices. In this context, the input embedding vector, which represents the data, is transformed into the Q, K, and V spaces by multiplying it with different weight matrices specifically designed for each component: W_Q for queries, W_K for keys, and W_V for values.

This approach allows the model to learn the optimal representations necessary for attention mechanisms during the training process. By adjusting these weight matrices, the model can adaptively learn how to focus on different parts of the input based on the task at hand. The use of matrix multiplication to derive Q, K, and V ensures a linear transformation, making it computationally efficient while enabling the model to capture complex relationships within the data.

In contrast, adding the original embedding vector to a weight matrix does not appropriately separate the different roles of Q, K, and V, as this method would yield overlapping representations rather than distinct, interpretable roles. Using a fixed encoding for each input implies that the model does not learn from the data, which limits its capability. Deriving them from an external data source would neglect the internal representations that the model requires during