Cutting the sentence into words
Before you can understand a sentence, you split it into words. A compiler's lexer does the same to code — chopping raw text into tokens like keywords, names, and symbols.
Tokens first
Lexical analysis (lexing) breaks source text into tokens — the basic units a parser can work with.