ToolSphere

Lexical Analyzer

Computer science

Tokenize C-like code into lexemes, classes and attributes, with a symbol-name table, class counts, LEX-style patterns and line/column error locations.

Edit the source to see its words, symbols and located errors.

16 tokens; 3 names; 0 lexical errors

Copy token analysis

A token is a classified source fragment, such as a name, number or operator. Comments are shown as tokens for teaching. This is lexical classification, not a full C parser, preprocessor or type checker. Numeric and quoted literals retain their original text.

16 recognized tokens · 3 symbol names · 0 lexical errors

Token table

Line:colLexemeClass / attribute
1:1// position updatecomment line comment — ignored by parser
2:1floatkeyword reserved word
2:7positionidentifier symbol #1
2:15,punctuation delimiter or punctuation
2:17initialidentifier symbol #2
2:24,punctuation delimiter or punctuation
2:26rateidentifier symbol #3
2:30;punctuation delimiter or punctuation
3:1positionidentifier symbol #1
3:10=operator operator
3:12initialidentifier symbol #2
3:20+operator operator
3:22rateidentifier symbol #3
3:27*operator operator
3:2960number decimal integer (text preserved)
3:31;punctuation delimiter or punctuation

Symbol-name table

SymbolNameOccurrences / first position
#1position2 · 2:7
#2initial2 · 2:17
#3rate2 · 2:26

This is a lexical name catalogue; repeated names share an entry. It does not assign scopes, declarations, storage or types.

Counts by token class

ClassCount
keyword1
identifier6
number1
string0
character0
operator3
punctuation4
comment1
LEX-style patterns
ClassPatternNotes
identifier[A-Za-z_][A-Za-z0-9_]*Classify a matching keyword before the general identifier rule.
decimal float([0-9]+\.[0-9]*|\.[0-9]+)([eE][+-]?[0-9]+)?[fFlL]? | [0-9]+[eE][+-]?[0-9]+[fFlL]?This dialect keeps the original literal text; it does not round it to a JavaScript number.
integer0[xX][0-9a-fA-F]+ | 0[bB][01]+ | 0[0-7]+ | 0 | [1-9][0-9]*Optional U/L/LL suffixes are recognised. Binary literals are a documented C-like extension.
string"([^"\\\n]|\\.)*"Quoted text and escapes are retained; no multiline literals or line splicing.
character'([^'\\\n]|\\.)+'Character literals are classified, not interpreted as an implementation-specific integer.
line comment//[^\n]*Comments are displayed for teaching but would normally be skipped by a parser.
block comment/\*([^*]|\*+[^*/])*\*+/Block comments do not nest.
whitespace[ \t\r\n]+Skipped; positions still advance. Tabs count as one reported column.
operator>>= <<= ++ -- -> += -= *= /= %= == != <= >= && || << >> &= |= ^= + - * / % = < > ! ~ & | ^ ?Choose the longest matching operator before trying its shorter prefix.

Pattern sketches describe this dialect; they are not a complete executable Flex specification. Whitespace is skipped. Reported columns count tabs as one position.

On this browser only

Give feedback

Did this tool help?

Tell us what worked, what didn’t, or what you’d like next.

Opens Google Forms in a new tab with this tool’s name filled in. No sign-in or email required. Feedback is optional.

How to use Lexical Analyzer

  1. 1Edit the loaded C-like position-update example, or paste a small source fragment into the labelled input.
  2. 2Read the token table to compare each lexeme with its class and attribute.
  3. 3Use the symbol-name table to find repeated identifiers and their first positions.
  4. 4Check lexical errors before treating the recognized tokens as a complete stream.
  5. 5Expand LEX-style patterns to study the rules, then copy the token analysis if needed.

About Lexical Analyzer

Turn source text into an inspectable token stream

This lexical analyzer scans a small C-like dialect and shows the recognized lexemes, token classes, attributes and source positions. It also builds a symbol-name catalogue, counts each displayed class and reports lexical errors with their starting line and column. The example is loaded immediately, and results update as you edit. No compile button or external service is needed to obtain the token stream.

A lexer answers what kind of token begins at the current position. It does not establish whether the entire program is syntactically or semantically valid. The sequence of a keyword and some identifiers may be tokenizable while still forming an invalid declaration. Keep that distinction when comparing lexical errors with the later parser or type-checking stages of a compiler.

Lexemes, token classes and attributes

A lexeme is the original piece of text, such as an identifier, a literal or an operator. A token class groups similar lexemes: identifiers share a class even when their names differ. An attribute supplies the extra information needed for that token. Here identifiers point to a stable symbol number, while literals carry a descriptive kind and retain their original spelling.

Numeric literals are not rounded to JavaScript numbers. Their exact text stays in the lexeme column, including large integers, exponent notation and suffixes. This is useful when studying a lexer because converting a constant to a target-language numeric value is a separate concern. String and character escapes are also retained rather than evaluated. The attribute describes classification, not a machine-specific encoded constant.

Work through the loaded position example

The source begins with a line comment, declares the names used in the assignment and updates position using initial, rate and a numeric literal. Running the same scanner as this tool gives 16 recognized tokens and 3 distinct identifier names, with no lexical errors. Comments are included in these displayed counts for teaching, although a parser would normally skip them.

Class Count
keyword 1
identifier 6
number 1
string 0
character 0
operator 3
punctuation 4
comment 1

The identifier catalogue generated from that source is:

Symbol Name Occurrences First position
#1 position 2 2:7
#2 initial 2 2:17
#3 rate 2 2:26

The declaration and later use of a name share one lexical entry. That is a catalogue of spelling, not proof that every use refers to the same declaration in a scoped language. The scanner does not resolve shadowing, allocate storage, determine types or check whether a declaration precedes a use.

Follow longest-match operator recognition

When several operators begin with the same character, try the longest supported operator first. A compound comparison or assignment must not be split into its shorter prefixes merely because the first character is already a valid operator. The supported list includes increment/decrement, shifts, compound assignments, comparisons, logical operators, arithmetic operators and common bitwise operators.

Punctuation such as brackets, braces, parentheses, commas and semicolons has its own class in this display. Whitespace advances the reported position but produces no token. Keyword recognition takes precedence over the general identifier classification; a keyword embedded inside a longer identifier does not turn that identifier into several tokens.

Numbers and quoted text have explicit boundaries

The number rules include decimal integers, octal integers, hexadecimal integers, binary integers as a documented C-like extension, and decimal floating literals with optional exponents. Common integer U/L/LL suffixes and floating suffixes are recognized. Unsupported or malformed numeric runs receive an error instead of being silently split into a plausible number and an identifier. This dialect is not the complete set of numeric forms accepted by every C version.

Quoted strings and character literals protect their contents from ordinary operator or comment recognition. A comment opener inside a string remains text. Likewise, quote characters inside a comment do not start a literal. An escaped quote does not terminate its literal. The scanner classifies escapes without translating them into execution-character values, and it does not implement multiline literals, raw strings, wide-string prefixes or line splicing.

Locate errors without trusting a partial stream

Invalid characters, unterminated strings or characters, empty character literals, unterminated block comments and malformed numeric forms are reported with their starting positions. A string that reaches a line boundary without closing produces an error and allows scanning to resume on the next line. An unterminated block comment consumes the remaining input because there is no reliable closing boundary to resume from.

When errors exist, the tables show recognized portions only. They should not be treated as a complete valid token stream until the errors are resolved. The input has size and token-count limits so a pasted file cannot create an impractically large teaching table. Empty input is allowed and shows an invitation to enter code rather than a stale answer.

Reported columns count Unicode codepoints, and a tab advances one reported column rather than expanding to a visual tab stop. Windows-style line endings are treated as a single line break. The scanner preserves original text slices, so a token's lexeme still corresponds to what was pasted even when a quoted literal contains non-ASCII characters. Identifiers themselves use the documented ASCII identifier pattern.

Use the LEX-style pattern panel responsibly

The expandable panel shows pattern sketches for identifiers, decimal floats, integer forms, quoted literals, comments and whitespace, alongside the supported operators. These explain the scanner's categories and longest-match decisions. They are not a complete executable Flex file: a real lexer specification also needs rule actions, state handling, error recovery and integration with its parser.

Block comments do not nest. Preprocessor directives are not expanded, macros are not substituted, and the analyzer does not perform a C preprocessor's tokenization stages. Treat it as a small educational dialect with visible rules rather than a conformance checker for an entire language standard.

Continue from lexical analysis to parsing

Use the regex-to-NFA converter to see how regular token patterns lead to automata, and the NFA-to-DFA converter to follow deterministic recognition. The compiler lab connects those topics with grammar analysis and intermediate code.

Copy token analysis saves the current token rows, symbol-name entries and error descriptions as text. It does not upload the source or create a persistent shared program. Keep the dialect limits beside copied results when using them in coursework.

Guides and tutorials

Frequently asked questions

What is the difference between a token and a lexeme?

A lexeme is the original source spelling. Its token class says what category it belongs to, and an attribute supplies extra information such as an identifier’s symbol number.

Does this check whether my C program is correct?

No. It performs lexical classification for the documented C-like dialect. Parsing, preprocessing, scope resolution and type checking are separate tasks.

Why are comments in the token table?

They are shown for teaching so you can inspect their boundaries and counts. A normal parser would usually receive a stream with comments and whitespace removed.

How are repeated identifiers handled?

The same spelling reuses its symbol-name entry and increments its occurrence count. The catalogue does not distinguish declarations in different scopes.

Are large numeric constants converted to numbers?

No. Their text is preserved exactly, and the attribute describes the literal category. This avoids inventing a rounded numeric value during lexical analysis.

Why does a compound operator stay in one token?

The scanner tries longer supported operators before shorter prefixes. This prevents a valid compound operator from being split into several simpler operators.

How are errors positioned?

Each error reports its starting line and column. Tabs count as one reported column, and line endings and Unicode codepoints are handled consistently with the displayed position convention.

Can I copy the patterns into Flex unchanged?

They are explanatory sketches, not a complete executable lexer specification. You still need actions, states and recovery rules for a real Flex program.

Does it understand macros and include directives?

It does not expand a preprocessor or resolve included files. It only classifies the text supplied in the input using its stated dialect.

What should I do when some tokens appear beside an error?

Those are recognized portions of the input. Resolve the reported lexical errors before treating the result as a complete stream for a parser.

Have a suggestion for Lexical Analyzer?Share your feedback ↗