A token is a string of characters, categorized according to the rules as a symbol (e.g., IDENTIFIER, NUMBER, COMMA). The process of forming tokens from an input stream of characters is called tokenization, and the lexer categorizes them according to a symbol type. A token can look like anything that is useful for processing an input text stream or text file.
A lexical analyzer generally does nothing with combinations of tokens, a task left for a parser. For example, a typical lexical analyzer recognizes parentheses as tokens, but does nothing to ensure that each "(" is matched with a ")".
Consider this expression in the C programming language:
sum=3+2;
Tokenized in the following table:
| Lexeme | Token type |
|---|---|
| sum | Identifier |
| = | Assignment operator |
| 3 | Integer literal |
| + | Addition operator |
| 2 | Integer literal |
| ; | End of statement |
Tokens are frequently defined by regular expressions, which are understood by a lexical analyzer generator such as lex. The lexical analyzer (either generated automatically by a tool like lex, or hand-crafted) reads in a stream of characters, identifies the lexemes in the stream, and categorizes them into tokens. This is called "tokenizing." If the lexer finds an invalid token, it will report an error.
Following tokenizing is parsing. From there, the interpreted data may be loaded into data structures for general use, interpretation, or compiling.
Read more about this topic: Lexical Analysis
Famous quotes containing the word token:
“It doesn’t make any difference how much money a father earns, his name is always Dad-Can-I.... Like all other children, my five have one great talent: they are gifted beggars. Not one of them ever ran into the room, looked up at me, and said, “I’m really happy that you’re my father, and as a tangible token of my appreciation, here’s a dollar.””
—Bill Cosby (20th century)
““With the gracious consent of the audience, you will be made to don the red tophat”Ma token phrase that the courts had evolved, whose true meaning was known to every schoolboy.”
—Vladimir Nabokov (1899–1977)
“He made for the door, automatically resuming his glasses and leaving in front of her, on the floor, his right slipper in token of his speedy return. Then, his desire exposed and his eyes wicked behind their strong lenses, he attempted to push her toward the bed.”
—Vladimir Nabokov (1899–1977)