Struct LexicalModel 

pub struct LexicalModel {
    pub alphabet: Vec<char>,
    pub classes: Vec<GrammarRule>,
    pub config: LexicalConfig,
}

Deterministic lexical model inferred from a positive corpus.

Fields

alphabet: Vec<char>

Distinct characters observed in the training corpus, sorted by scalar value.

classes: Vec<GrammarRule>

Inferred token-level rules lowered to the grammar IR.

config: LexicalConfig

Configuration used to infer this model.

Implementations

impl LexicalModel

pub fn tokenize(&self, text: &str) -> Vec<Token>

Re-tokenises text with the same category-driven segmentation policy.

Trait Implementations

impl Clone for LexicalModel

fn clone(&self) -> LexicalModel

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LexicalModel

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for LexicalModel

fn eq(&self, other: &LexicalModel) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for LexicalModel

impl StructuralPartialEq for LexicalModel