Turn source text into an inspectable token stream
This lexical analyzer scans a small C-like dialect and shows the recognized lexemes, token classes, attributes and source positions. It also builds a symbol-name catalogue, counts each displayed class and reports lexical errors with their starting line and column. The example is loaded immediately, and results update as you edit. No compile button or external service is needed to obtain the token stream.
A lexer answers what kind of token begins at the current position. It does not establish whether the entire program is syntactically or semantically valid. The sequence of a keyword and some identifiers may be tokenizable while still forming an invalid declaration. Keep that distinction when comparing lexical errors with the later parser or type-checking stages of a compiler.
Lexemes, token classes and attributes
A lexeme is the original piece of text, such as an identifier, a literal or an operator. A token class groups similar lexemes: identifiers share a class even when their names differ. An attribute supplies the extra information needed for that token. Here identifiers point to a stable symbol number, while literals carry a descriptive kind and retain their original spelling.
Numeric literals are not rounded to JavaScript numbers. Their exact text stays in the lexeme column, including large integers, exponent notation and suffixes. This is useful when studying a lexer because converting a constant to a target-language numeric value is a separate concern. String and character escapes are also retained rather than evaluated. The attribute describes classification, not a machine-specific encoded constant.
Work through the loaded position example
The source begins with a line comment, declares the names used in the assignment and updates position using initial, rate and a numeric literal. Running the same scanner as this tool gives 16 recognized tokens and 3 distinct identifier names, with no lexical errors. Comments are included in these displayed counts for teaching, although a parser would normally skip them.
| Class | Count |
|---|---|
| keyword | 1 |
| identifier | 6 |
| number | 1 |
| string | 0 |
| character | 0 |
| operator | 3 |
| punctuation | 4 |
| comment | 1 |
The identifier catalogue generated from that source is:
| Symbol | Name | Occurrences | First position |
|---|---|---|---|
| #1 | position | 2 | 2:7 |
| #2 | initial | 2 | 2:17 |
| #3 | rate | 2 | 2:26 |
The declaration and later use of a name share one lexical entry. That is a catalogue of spelling, not proof that every use refers to the same declaration in a scoped language. The scanner does not resolve shadowing, allocate storage, determine types or check whether a declaration precedes a use.
Follow longest-match operator recognition
When several operators begin with the same character, try the longest supported operator first. A compound comparison or assignment must not be split into its shorter prefixes merely because the first character is already a valid operator. The supported list includes increment/decrement, shifts, compound assignments, comparisons, logical operators, arithmetic operators and common bitwise operators.
Punctuation such as brackets, braces, parentheses, commas and semicolons has its own class in this display. Whitespace advances the reported position but produces no token. Keyword recognition takes precedence over the general identifier classification; a keyword embedded inside a longer identifier does not turn that identifier into several tokens.
Numbers and quoted text have explicit boundaries
The number rules include decimal integers, octal integers, hexadecimal integers, binary integers as a documented C-like extension, and decimal floating literals with optional exponents. Common integer U/L/LL suffixes and floating suffixes are recognized. Unsupported or malformed numeric runs receive an error instead of being silently split into a plausible number and an identifier. This dialect is not the complete set of numeric forms accepted by every C version.
Quoted strings and character literals protect their contents from ordinary operator or comment recognition. A comment opener inside a string remains text. Likewise, quote characters inside a comment do not start a literal. An escaped quote does not terminate its literal. The scanner classifies escapes without translating them into execution-character values, and it does not implement multiline literals, raw strings, wide-string prefixes or line splicing.
Locate errors without trusting a partial stream
Invalid characters, unterminated strings or characters, empty character literals, unterminated block comments and malformed numeric forms are reported with their starting positions. A string that reaches a line boundary without closing produces an error and allows scanning to resume on the next line. An unterminated block comment consumes the remaining input because there is no reliable closing boundary to resume from.
When errors exist, the tables show recognized portions only. They should not be treated as a complete valid token stream until the errors are resolved. The input has size and token-count limits so a pasted file cannot create an impractically large teaching table. Empty input is allowed and shows an invitation to enter code rather than a stale answer.
Reported columns count Unicode codepoints, and a tab advances one reported column rather than expanding to a visual tab stop. Windows-style line endings are treated as a single line break. The scanner preserves original text slices, so a token's lexeme still corresponds to what was pasted even when a quoted literal contains non-ASCII characters. Identifiers themselves use the documented ASCII identifier pattern.
Use the LEX-style pattern panel responsibly
The expandable panel shows pattern sketches for identifiers, decimal floats, integer forms, quoted literals, comments and whitespace, alongside the supported operators. These explain the scanner's categories and longest-match decisions. They are not a complete executable Flex file: a real lexer specification also needs rule actions, state handling, error recovery and integration with its parser.
Block comments do not nest. Preprocessor directives are not expanded, macros are not substituted, and the analyzer does not perform a C preprocessor's tokenization stages. Treat it as a small educational dialect with visible rules rather than a conformance checker for an entire language standard.
Continue from lexical analysis to parsing
Use the regex-to-NFA converter to see how regular token patterns lead to automata, and the NFA-to-DFA converter to follow deterministic recognition. The compiler lab connects those topics with grammar analysis and intermediate code.
Copy token analysis saves the current token rows, symbol-name entries and error descriptions as text. It does not upload the source or create a persistent shared program. Keep the dialect limits beside copied results when using them in coursework.