Module grammar 

Grammar intermediate representation and links encoding.

The grammar IR is a small expression algebra that can hold PEG, BNF, EBNF, ABNF, and inferred grammars without committing to one textual surface syntax. Values can be encoded as first-class grammar links in a LinkNetwork.

Example

use meta_language::{
    FromLinks, Grammar, LinkType, LinksDecoder, LinksEncoder, ToLinks,
};

let expr = Grammar::expr();
let grammar = Grammar::builder().start("word").rule("word", expr.rep1(expr.char_range('a', 'z'))).build();

let mut encoder = LinksEncoder::new();
let root = grammar.to_links(&mut encoder);
let network = encoder.into_network();
assert!(network.links().any(|link| link.metadata().link_type() == Some(LinkType::Grammar)));
let mut decoder = LinksDecoder::new(&network);
assert_eq!(Grammar::from_links(&mut decoder, root).expect("grammar decodes"), grammar);

Re-exports

pub use concepts::GRAMMAR_CONCEPTS;

pub use concepts::GrammarConcept;

pub use concepts::annotate_grammar_concepts;

pub use concepts::grammar_expr_concept_id;

pub use concepts::rule_concept_id;

pub use decorators::decorate_emitted;

pub use decorators::decorate_grammar;

pub use decorators::decorate_syntax_tree;

pub use emit::EmitReport;

pub use emit::GrammarEmitError;

pub use emit::JsParserArtifacts;

pub use emit::RustParserArtifacts;

pub use emit::emit_abnf;

pub use emit::emit_antlr;

pub use emit::emit_bnf;

pub use emit::emit_ebnf;

pub use emit::emit_gbnf;

pub use emit::emit_javascript_parser;

pub use emit::emit_lark;

pub use emit::emit_peggy;

pub use emit::emit_pest;

pub use emit::emit_rust_parser;

pub use emit::emit_tree_sitter_grammar_js;

pub use emit::emit_tree_sitter_grammar_js_with_report;

pub use emit::emit_tree_sitter_json;

pub use emit::render_rust_type;

pub use feature::ByteClassItem;

pub use feature::FEATURE_EXPRESSION_FORMS;

pub use feature::FeatureExpr;

pub use feature::FeatureForm;

pub use feature::FieldType;

pub use feature::FieldValue;

pub use feature::GrammarDeclarations;

pub use feature::GrammarMacro;

pub use feature::GrammarScanner;

pub use feature::MATCHING_MODES;

pub use feature::OPERATION_FORMS;

pub use feature::Operation;

pub use feature::OperationCategory;

pub use feature::PrecedenceEntry;

pub use feature::RuleAttributes;

pub use feature::SETTLING_STEPS;

pub use feature::UnicodeClassItem;

pub use feature::default_settling;

pub use feature::settling_problem;

pub use fidelity::FORMER_GRAMMAR_CONSTRUCTS;

pub use fidelity::GRAMMAR_CONSTRUCTS;

pub use fidelity::GRAMMAR_FORMATS;

pub use fidelity::GrammarFidelityLevel;

pub use fidelity::GrammarFormatProfile;

pub use fidelity::canonical_grammar_format;

pub use fidelity::current_grammar_construct;

pub use fidelity::grammar_format_profile;

pub use import::GrammarImportError;

pub use import::import_abnf;

pub use import::import_antlr;

pub use import::import_bnf;

pub use import::import_ebnf;

pub use import::import_gbnf;

pub use import::import_lark;

pub use import::import_pest;

pub use import::import_tree_sitter_json;

pub use inference::active::ActiveLearningConfig;

pub use inference::active::ActiveLearningError;

pub use inference::active::Dfa;

pub use inference::active::GrammarAcceptorOracle;

pub use inference::active::Oracle as ActiveLearningOracle;

pub use inference::active::ParserAcceptancePredicate;

pub use inference::active::ParserMembershipOracle;

pub use inference::active::SamplingEquivalenceOracle;

pub use inference::active::Symbol as ActiveSymbol;

pub use inference::active::clean_structural_acceptance;

pub use inference::active::learn_dfa;

pub use inference::active::learn_grammar;

pub use inference::advisor::AdviceDecision;

pub use inference::advisor::AdviceDecisionKind;

pub use inference::advisor::AdviceSource;

pub use inference::advisor::ConceptNamingAdvisor;

pub use inference::advisor::FallbackAdvisor;

pub use inference::advisor::MdlMergeAdvisor;

pub use inference::advisor::MergeAdvisor;

pub use inference::advisor::MergeCandidate;

pub use inference::advisor::MergeRequest;

pub use inference::advisor::MergeScore;

pub use inference::advisor::NameCandidate;

pub use inference::advisor::NamingAdvisor;

pub use inference::advisor::NamingRequest;

pub use inference::advisor::LlmClient;

pub use inference::advisor::LlmError;

pub use inference::advisor::LlmMergeAdvisor;

pub use inference::advisor::LlmNamingAdvisor;

pub use inference::cfg::InferenceOptions;

pub use inference::cfg::InferenceReport;

pub use inference::cfg::InferenceResult;

pub use inference::cfg::Oracle;

pub use inference::cfg::PositiveOnlyOracle;

pub use inference::cfg::infer_cfg;

pub use inference::cfg::infer_cfg_with_advisors;

pub use inference::eval::BenchmarkReport;

pub use inference::eval::EvalError;

pub use inference::eval::GOLDEN_CORPORA;

pub use inference::eval::GoldenCorpus;

pub use inference::eval::GrammarOracle;

pub use inference::eval::MembershipOracle;

pub use inference::eval::MetricScores;

pub use inference::eval::SampleConfig;

pub use inference::eval::ScoringMode;

pub use inference::eval::evaluate;

pub use inference::eval::mdl;

pub use inference::eval::run_corpus;

pub use inference::eval::run_named_corpus;

pub use inference::eval::sample;

pub use inference::eval::size_symbols;

pub use inference::lexical::CharCategory;

pub use inference::lexical::LexicalConfig;

pub use inference::lexical::LexicalModel;

pub use inference::lexical::Token;

pub use inference::lexical::categorise;

pub use inference::lexical::infer_lexical_classes;

pub use inference::minimize::Mdl;

pub use inference::minimize::MinimizeOptions;

pub use inference::minimize::MinimizeReport;

pub use inference::minimize::MinimizeResult;

pub use inference::minimize::mdl_cost;

pub use inference::minimize::minimize;

pub use inference::prior::ByteSpan;

pub use inference::prior::Delimiter;

pub use inference::prior::LeafKind;

pub use inference::prior::PriorOptions;

pub use inference::prior::SeedNode;

pub use inference::prior::SeedTree;

pub use inference::prior::StructuralPrior;

pub use inference::prior::WhitespacePolicy;

pub use inference::prior::build_structural_prior;

pub use inference::semantic::ConstraintAtom;

pub use inference::semantic::ConstraintClause;

pub use inference::semantic::ConstraintPattern;

pub use inference::semantic::LengthUnit;

pub use inference::semantic::NonTerminalRef;

pub use inference::semantic::SemanticConstraint;

pub use inference::semantic::SemanticInferenceConfig;

pub use inference::semantic::default_pattern_catalog;

pub use inference::semantic::evaluate_atom;

pub use inference::semantic::evaluate_clause;

pub use inference::semantic::evaluate_constraint;

pub use inference::semantic::evaluate_probabilistic;

pub use inference::semantic::mine_semantic_constraints;

pub use inference::sequitur::Symbol;

pub use inference::sequitur::run_sequitur;

pub use inference::state_merging::InferredAutomaton;

pub use inference::state_merging::MergeStrategy;

pub use inference::state_merging::Sample;

pub use inference::state_merging::infer_dfa;

pub use interchange::GRAMMAR_COMMAND_USAGE;

pub use interchange::GRAMMAR_EXPORT_FORMATS;

pub use interchange::GRAMMAR_IMPORT_FORMATS;

pub use interchange::GRAMMAR_LOSSLESS_FORMATS;

pub use interchange::GrammarCommandOutput;

pub use interchange::GrammarEmitter;

pub use interchange::GrammarFileReader;

pub use interchange::GrammarImporter;

pub use interchange::GrammarLayout;

pub use interchange::GrammarLayoutDefinition;

pub use interchange::GrammarLayoutImplicit;

pub use interchange::GrammarLosslessError;

pub use interchange::GrammarSourceDefinition;

pub use interchange::GrammarSourceSplit;

pub use interchange::capture_grammar_layout;

pub use interchange::deserialize_grammar;

pub use interchange::emit_grammar_lossless;

pub use interchange::grammar_emitter;

pub use interchange::grammar_importer;

pub use interchange::import_grammar_lossless;

pub use interchange::parse_grammar_layout_links;

pub use interchange::parse_grammar_links;

pub use interchange::parse_links_expression;

pub use interchange::parse_native_grammar;

pub use interchange::percent_decode_links_text;

pub use interchange::percent_encode_links_text;

pub use interchange::render_declaration_links;

pub use interchange::render_grammar_layout_links;

pub use interchange::render_grammar_links;

pub use interchange::render_links_expression;

pub use interchange::render_native_expression;

pub use interchange::render_native_feature;

pub use interchange::render_native_grammar;

pub use interchange::render_rule_fields;

pub use interchange::render_rule_link;

pub use interchange::run_grammar_command;

pub use interchange::serialize_grammar;

pub use interchange::split_grammar_source;

pub use merge::GRAMMAR_MERGE_METHOD;

pub use merge::GrammarMergeAlternative;

pub use merge::GrammarMergeAlternativeReason;

pub use merge::GrammarMergeDecision;

pub use merge::GrammarMergeDecisionKind;

pub use merge::GrammarMergeError;

pub use merge::GrammarMergeFailure;

pub use merge::GrammarMergeFailureKind;

pub use merge::GrammarMergeFailureReason;

pub use merge::GrammarMergeNomination;

pub use merge::GrammarMergeNominationBasis;

pub use merge::GrammarMergeNominationOutcome;

pub use merge::GrammarMergeOptions;

pub use merge::GrammarMergeResult;

pub use merge::GrammarMergeSource;

pub use merge::GrammarRenameError;

pub use merge::GrammarRenameErrorKind;

pub use merge::MergedGrammarGroup;

pub use merge::RenamedGrammar;

pub use merge::RuleAlias;

pub use merge::assert_merge_complete;

pub use merge::assert_merge_shares;

pub use merge::merge_grammars;

pub use merge::normalized_rule_definition;

pub use merge::rename_grammar_rule;

pub use merge::restore_source_names;

pub use merge::shared_rule_decisions;

pub use reverse::GrammarReverseConversion;

pub use reverse::GrammarReverseFailure;

pub use reverse::GrammarReverseFailureKind;

pub use reverse::GrammarReverseReport;

pub use reverse::GrammarReverseStage;

pub use reverse::GrammarReverseStatus;

pub use reverse::check_grammar_reverse_conversion;

pub use round_trip::GRAMMAR_ROUND_TRIP_MARKER;

pub use round_trip::GrammarEmitFn;

pub use round_trip::GrammarImportFn;

pub use round_trip::GrammarRoundTrip;

pub use round_trip::GrammarRoundTripError;

pub use round_trip::GrammarRoundTripFailure;

pub use round_trip::GrammarRoundTripFailureKind;

pub use round_trip::GrammarRoundTripReport;

pub use round_trip::GrammarRoundTripStage;

pub use round_trip::GrammarRoundTripStatus;

pub use round_trip::accepts_text;

pub use round_trip::canonical_rule_definition;

pub use round_trip::check_grammar_round_trip;

pub use round_trip::mutate_grammar_start_rule;

pub use runtime::GrammarParser;

pub use runtime::register_grammar;

pub use runtime::with_grammar;

pub use surface::GrammarSurfaceError;

pub use surface::grammar_from_lino;

pub use surface::grammar_to_lino;

pub use surface::parse_grammar_surface;

pub use surface::write_grammar_surface;

pub use translate::GrammarTranslateError;

pub use translate::grammar_concept_translation_rules;

pub use translate::translate_grammar_surface;

pub use validate::DiagnosticKind;

pub use validate::GRAMMAR_DIAGNOSTIC_KINDS;

pub use validate::GrammarDiagnostic;

pub use validate::RuleSpan;

pub use validate::Severity;

pub use validate::display_grammar_expression;

pub use validate::validate;

Modules

concepts

Concept-ontology alignment for grammar algebra nodes.

decorators

The grammar and syntax tree levels of DecoratorSet.

emit

Emitters for external grammar definition formats.

feature

The grammar feature union forms (docs/grammar/feature-union.md).

feature_runtime

The native executor of the grammar feature union, the Rust port of js/src/grammar-runtime.js.

fidelity

Per-format capability profiles for grammar round-trip fidelity.

import

Importers for external grammar definition formats.

inference

Grammar inference support.

interchange

Grammar interchange commands shared by the Rust and JavaScript tools.

merge

Deterministic, meaning-aware merging of grammars from several sources, and binding-aware renaming of grammar rules.

reverse

Reverse conversion of a grammar back to its source format.

round_trip

Mutation-guarded grammar interchange round trips.

runtime

Runtime parser for first-class grammar values.

surface

Meta-notation-derived textual surface syntax for grammar authoring.

translate

Concept-aligned translation of a grammar’s human-facing surface.

validate

Semantic validation for authored grammar IR values.

Structs

ExprBuilder

Ergonomic constructor for grammar expressions.

Grammar

Order-preserving grammar.

GrammarBuilder

Fluent builder for order-preserving grammars.

GrammarKind

A node kind no rule defines, with its names in the grammars it came from.

GrammarRule

A named grammar rule.

GrammarSourceName

A name a rule has in a grammar it was merged from, such as its tree-sitter or ANTLR name, kept for the reverse conversion.

RepetitionBoundsError

Counted repetition bounds canonical_repeat rejects: an upper bound below the lower one, or a bound past the JavaScript safe-integer range.

Enums

CharClassItem

One item inside a character class.

GrammarExpr

One node of the grammar expression algebra.

GrammarFormat

Origin grammar format.

RuleKind

How a rule participates in parsing.

Functions

canonical_repeat

Lowers counted repetition to its most specific expression variant.

carry_rule_docs

Copies the documentation of every rule of source onto the rule of target that rename names it.

choice

Flattens nested choices of the same ordering, as the JavaScript choice builder does: no alternatives build GrammarExpr::Empty and one alternative builds itself.

sequence

Flattens nested sequences and drops empty items, as the JavaScript sequence builder does: no items build GrammarExpr::Empty and one item builds itself.