Evaluation

Measure how well a model composes answers with Fable.

The repository includes a live evaluation that runs the playground agent against real prompts and checks the result.

pnpm eval                       # provider and model from FABLE_UI_EVAL_PROVIDER / FABLE_UI_EVAL_MODEL
pnpm eval openai gpt-6.1-sol    # or name them

It costs money, so it only runs through pnpm eval with a key in .env.local. FABLE_UI_EVAL_MAX_PROMPTS caps the number of cases.

What it checks

MetricMeaning
UI choiceUsed render_ui when structure helps, plain text for greetings and single facts
ValidityEvery block, including nested ones, passed strict validation
Expected blockPicked a block that fits the request (a table for a comparison, a form for a clarifying question…)
Images from toolEvery image URL came from the search_images tool, none invented
Latency, tokensWall time and usage per case

Reports are written to evals/results/<provider>-<model>-<date>.json.

Offline checks

pnpm test runs without a model. It validates every block example and playground scenario, replays each scenario at hundreds of partial-JSON prefixes to check that streaming never throws and keys stay stable, and tests the URL, image, filename and CSV policies.

Results

The latest results are in evals/results and summarized in the README. Results depend on the model; run the eval with yours before relying on them.