Embedding Layer

The embedding layer is the first learned component in a language model. It maps each integer token ID to a dense vector of fixed dimensionality — the model's hidden size. These vectors are the continuous representations the rest of the network operates on. The embedding weights are learned during pretraining; the layer essentially serves as a lookup table where semantically or statistically related tokens tend to occupy nearby regions of the embedding space.