Module lexer 

Tokenizer shared by the portable-core translation frontends.

Every token keeps its UTF-16 offsets so diagnostics and source mappings agree with the JavaScript runtime; comments are retained separately because JSDoc types are part of the JavaScript frontend’s input.

Mirrors js/src/translation/lexer.js.

Structs

Comment

Source

Source text addressed by UTF-16 code units, as JavaScript strings are.

TemplateExpression

A ${…} substitution in a JavaScript template literal.

TemplatePart

Token

TokenCursor

Cursor over a token list with the helpers every frontend needs.

Tokens

Enums

TokenKind

Functions

describe

JSON.stringify(token.raw), or end of input.

is_js_space

JavaScript’s /\s/u on one code unit.

json_string

JSON.stringify of a string.

matching_brace

The index of the } closing the { at open.

tokenize

Splits text into tokens and comments.

tokenize_source

tokenize over an already encoded source.