Text Data Analysis II: Language Models

TF-IDF stands for Term Frequency–Inverse Document Frequency. It is a numerical statistic used to reflect how important a word is to a document within a collection. The formula combines two components: TF (how often the term appears in a specific document) and IDF (the inverse of how many documents contain that term across the whole corpus). Words appearing in many documents get a lower IDF, naturally down-weighting common words like "the" and boosting distinctive ones.

A high TF-IDF score means the word is important and distinctive to that particular document. This happens when TF is high (the word appears a lot in that document) and IDF is high (the word is rare across the corpus). A word common in all documents (B) would have IDF ≈ 0, killing its TF-IDF score. Option C would give TF = 0, also zero. Option D equally low IDF. Think of a medical paper frequently using the word "myocardial" — rare in most texts, but prominent in that document → high TF-IDF.

TF-IDF's biggest weakness is that it treats every word as an isolated symbol — it has no understanding of meaning. The words "car," "automobile," and "vehicle" would be completely unrelated in TF-IDF space even though they're semantically close. It cannot handle synonymy (same meaning, different words) or polysemy (same word, different meanings). Option A is wrong — TF-IDF naturally suppresses stop words because they appear in all documents (IDF ≈ 0). Option C is partially true but not the primary limitation. Option D is wrong — TF-IDF is entirely unsupervised.

A bigram is a sequence of exactly two consecutive words (or tokens). "Machine learning" (A) is a two-word sequence — a textbook bigram. Option B, "Deep-learning," is a single hyphenated token, not two separate words. Option C, "bi-gram is," contains three tokens (bi, gram, is) making it a trigram. Option D, "Model," is a single word (a unigram). N-grams are fundamental building blocks of statistical language models, where N indicates the window size: unigram (1), bigram (2), trigram (3), etc.

N-gram models predict the next word based on a fixed window of N–1 preceding words. The fundamental limitation is that this fixed window cannot capture dependencies between words far apart in a sentence. Consider: "The cat that sat on the mat was …" — to correctly predict the next word, you might need context from much earlier. N-grams are inherently Markovian — they assume the current word only depends on the last N–1 words. Option B is wrong — N-grams explicitly model word order. Options A and C are real concerns but secondary to the core long-range dependency problem that motivates neural language models.

The defining advantage of word embeddings (e.g., Word2Vec, GloVe) is that semantically similar words end up close together in the dense vector space (A). Words that appear in similar contexts receive similar vectors, so synonyms cluster together. The classic demonstration: king − man + woman ≈ queen. Options B and C are wrong — embeddings typically require large corpora to learn good representations, and while they reduce dimensionality versus one-hot encodings, they don't "eliminate" the curse of dimensionality. Option D (positional encoding) is a feature of Transformers, not word embeddings themselves.

The dimensionality of a word embedding (e.g., 100, 300, 768) represents the number of latent semantic features the model uses to represent each word (A). These are not hand-crafted features — they're learned automatically during training. Higher dimensions can capture richer relationships but cost more compute. Contrast this with one-hot encoding, where the vector size equals vocabulary size (B) — embeddings are far smaller and denser. The corpus size (C) and document count (D) are training hyper-parameters that influence quality but don't define the vector dimension.

The fundamental architectural distinction is the use of deep learning — specifically the Transformer architecture (B). Transformers use self-attention mechanisms that can model relationships between any two positions in a sequence regardless of distance, overcoming the fixed-window limitation of N-grams. While LLMs do process large corpora (A), so did earlier neural models. LLMs are primarily self-supervised (trained on raw text without labels), making C incorrect. Option D (coherent text generation) is a capability that results from the architecture, not the defining distinction itself.

TF-IDF = TF × IDF, where IDF = log(N / df). Because "lizard" appears in all three documents, IDF collapses to zero — making TF-IDF = 0. This is exactly TF-IDF's design: words that appear everywhere carry no discriminative power. The distractor 0.477 = log₁₀(3) would apply if "lizard" appeared in only one document; 0.176 would result from a smoothed IDF formula.

Word embedding models (Word2Vec, GloVe) are built on the distributional hypothesis: "words that occur in similar contexts have similar meanings." So two words with similar contexts will produce similar embedding vectors — they'll be close together in the vector space (A). Option B is wrong: TF-IDF depends on per-document frequency distributions, so two semantically identical words appearing in different documents can have very different TF-IDF values. Options C and D confuse vector values with vector dimensions. In any given model, all words share the same fixed dimensionality (e.g., all vectors are 300-dimensional) — what differs are the actual values inside those vectors, not the size.