pub fn tokenize(text: &str, language: Language) -> Result<Tokens>Expand description
Splits text into tokens and comments.
ยงErrors
Returns a syntax error for malformed numbers, unsupported escapes and unterminated strings, template literals or block comments.