Text Data Analysis I: Pre-processing

Text preprocessing is the pipeline that transforms raw text into a clean, structured form before analysis. The core techniques are: Tokenization (A) — splitting text into words or sub-word units; Stemming (B) — reducing words to their root form by chopping suffixes (e.g., "running" → "run"); and Lemmatization (C) — reducing words to their canonical dictionary form using vocabulary and grammar rules. Indexing (D) is a downstream step that happens after preprocessing — it builds a searchable structure (like an inverted index) from the already-cleaned tokens.

Stop words are extremely common, low-information words like "the," "is," "and," "of," "a" that appear in almost every document and carry little discriminative value for most NLP tasks. They are removed during preprocessing to reduce noise and vocabulary size (A). They are not rare (B) — they're the most frequent words in any language. They don't mark sentence boundaries (C) — that's punctuation or sentence-boundary detectors. And stemming operates on content words, not stop words (D).

Lemmatization (A) uses a full vocabulary and applies proper morphological analysis — it knows that "better" → "good" (irregular) and "running" → "run." It always returns a valid dictionary word. Stemming (B) uses crude heuristic rules to strip suffixes, so it can produce non-words — e.g., "studies" → "studi" with the Porter stemmer. These are non-dictionary intermediate forms. Option C is false — lemmatization is generally more accurate but slower. Option D is backwards — lemmatization is actually language-dependent (it relies on a language-specific lexicon), whereas stemming algorithms are more easily adapted across languages.

Tokenization (A) is the foundational first step: it segments a continuous string of characters into discrete tokens — typically words, punctuation, or sub-word units — that all downstream processes can operate on. Without tokens, you have a raw character stream that algorithms cannot interpret. N-gram extraction (B) requires tokenization first and is a separate downstream step. Grammar checking (C) and lemmatization speed (D) are not primary purposes of tokenization.

All four are valid suffix targets in standard stemmers. "-ing" (A) removes the present participle: running → run. "-ly" (B) removes adverb forms: quickly → quick. "-able" (C) removes adjective suffixes: comfortable → comfort. "-est" (D) removes superlative forms: greatest → great. The Porter stemmer and similar algorithms have rules covering all of these common English suffixes. Stemming is rule-based and strips any recognized suffix, regardless of whether the result is a real word.

Stop word removal is implemented via vocabulary filtering (B): you maintain a list (vocabulary/lexicon) of known stop words and simply discard any token that matches an entry in that list. Pre-trained models (A) are used for embedding or fine-tuning downstream tasks, not for basic filtering. N-gram generation (C) is about creating multi-word sequences. Feature selection (D) is a broader ML concept that may or may not remove stop words, but it's not the specific technique for stop word removal itself.

Text mining refers to the discovery of patterns and knowledge from text data. Its applications include: Information extraction (A) — automatically pulling structured facts (entities, relations) from unstructured text; Machine translation (B) — translating text between languages using learned linguistic patterns; Information retrieval (C) — finding relevant documents matching a query (e.g., search engines). Speech recognition (D) is an audio signal processing task — it converts spoken audio to text, operating on acoustic waveforms rather than text data, so it is not a text mining application.

All four are genuine NLP challenges. Ambiguity (A) — words and sentences often have multiple meanings (e.g., "bank" can mean a financial institution or a riverbank). Scale (B) — the internet produces billions of text documents; processing them efficiently is a major engineering and computational challenge. Sparsity (C) — in bag-of-words models, most words don't appear in a given document, creating very sparse high-dimensional vectors that are hard to learn from. Variation (D) — the same concept is expressed in many ways (synonyms, dialects, typos, slang), making generalisation hard.

Since stop words are not removed, "and" stays in the vocabulary. The 12-word vocabulary is:

and	cuts	decapitates	eats	lizard	paper	poisons	rock	scissors	smashes	spock	vaporizes
A:	1	1	1	0	1	1	0	0	1	0	0	0
B:	1	0	0	1	1	1	1	0	0	0	1	0

A and B share: and, lizard, paper → 3 common terms. Differences: A has cuts, decapitates, scissors that B doesn't; B has eats, poisons, spock that A doesn't. That's 6 mismatches. Euclidean distance = √6 ≈ 2.449 ≈ 2.45.

Using the same 12-term vocabulary (stop words kept):

and	eats	lizard	paper	poisons	rock	scissors	smashes	spock	vaporizes
B:	1	1	1	1	1	0	0	0	1	0
C:	1	0	0	0	0	1	1	1	1	1

B and C share: and, spock → dot product = 2. |B| = √6 ≈ 2.449, |C| = √6 ≈ 2.449.
Cosine = 2 / (2.449 × 2.449) = 2 / 6 ≈ 0.333 ≈ 0.33

This is a conceptual trap. Euclidean distance measures absolute distance in space, accounting for both direction and magnitude. Cosine similarity measures only the angle between vectors, completely ignoring magnitude. Consider a simple counterexample: A = [1, 0], B = [2, 0], C = [0, 3]. Euclidean: |A–B| = 1, |A–C| = √10 ≈ 3.16, so A is closer to B. But cosine(A, B) = 1.0 (identical direction) while cosine(A, C) = 0 (perpendicular). Now try A = [1,1], B = [1.1, 1.1], C = [10, 0]. A is closer to B by Euclidean, but cosine(A,C) might differ unpredictably. Without knowing the actual vectors, you simply cannot infer cosine similarity from Euclidean distance. The correct answer is E.