What kind of values does each token receive during tokenization?

During tokenization, each token is typically assigned numerical values that correspond to its representation in a predefined vocabulary or dictionary. This process is essential in natural language processing (NLP) where words or phrases (tokens) from text are converted into numerical format so that they can be effectively used in machine learning models.

The vocabulary is created based on the dataset and generally maps each unique token to a specific numerical identifier. This mapping allows the AI model to process and analyze text data mathematically, which is critical for tasks like language understanding, translation, or sentiment analysis.

In contrast to the numerical values assigned during tokenization, other types of values such as textual values, null values, and random values do not appropriately represent the structural and semantic significance necessary for AI models. Hence, they do not serve the purpose of facilitating computational processing in the same effective manner as numerical representations from a dictionary would.