Module benchmark 

Competitor benchmark orchestration for the E3 grammar-inference suite.

The metric math stays in the D1 evaluation module and the inferred grammar comes from the D5 infer_cfg entry point. This module owns manifest loading, vendored-corpus integrity checks, report formatting, and bar assertions. Manifest byte counts assume the repository-pinned LF checkout policy for benches/corpora/** so CI reports the same fixture totals on every OS.

Structs

BenchmarkFailure

Non-fatal suite failure collected so one run reports every problem.

CompetitorManifest

A parsed competitor corpus manifest.

CompetitorRun

One included corpus result.

CompetitorSuiteReport

Full suite result.

CorpusManifestEntry

One vendored competitor subject.

SecondaryMetricRow

Secondary metric report row.

SkippedCorpus

One skipped manifest entry rendered in the benchmark log.

Enums

BenchmarkError

Fatal setup error that prevents a suite report from being built.

Constants

D5_REQUIRED_AVG_F1

Deterministic D5 runs must meet the current top published average F1 bar.

DEFAULT_CORPORA_ROOT

Default vendored corpus root used by the bench and integration gate.

DEFAULT_CORPUS_MANIFEST

Default manifest path used by the bench and integration gate.

PUBLISHED_NATGI_AVG_F1

Published NatGI headline average F1 from the E3 competitive-analysis notes.

PUBLISHED_TREEVADA_AVG_F1

Published TreeVada headline average F1 from the E3 competitive-analysis notes.

Functions

load_manifest

Parses a competitor corpus manifest.

render_competitor_report

Formats a suite report as the CI failure log and bench output.

run_competitor_suite

Loads the default manifest and runs the default vendored competitor suite.

run_competitor_suite_from_paths

Loads a manifest and runs a vendored competitor suite from explicit paths.