Skip to main content

Module lexical

meta_language::grammar::inference

Module lexical

Expand description

Lexical class inference for positive example corpora.

Structs§

LexicalConfig: Configuration for lexical class inference.
LexicalModel: Deterministic lexical model inferred from a positive corpus.
Token: One lossless token produced by lexical segmentation.

Enums§

CharCategory: Coarse Unicode-aware character category used for token segmentation.

Functions§

categorise: Categorises one character into a coarse lexical class.
infer_lexical_classes: Infers a lexical model from positive example texts.