What it checks

Lint and type checks do not catch a query that PostgreSQL rejects, a queue that never drains, or a reranker change that makes half the answers worse. The evaluation harness checks each of those against a real database:

Layer	Where	Needs a stack
unit	packages/*/tests/	no
SQL	packages/core/tests/test_sql_db.py	database
benchmark	packages/mcp/eval/run.py	database
MCP tools	packages/mcp/tests/test_e2e.py	MCP server

The SQL layer collects every SQL literal in the sources and PREPAREs it against the migrated schema, so a new query is covered without being registered anywhere. It also drives the embedding queue through enqueue, claim, finish and fail inside a rolled-back transaction.

The eval stack

docker-compose.eval.yaml is a separate compose project, enggraph-eval. Its database lives on tmpfs, and it binds only loopback ports: 55432 for PostgreSQL and 53000 for MCP. make eval-up migrates it and indexes eval/corpus/alpha as project alpha. It never touches the main stack.

The corpus is a small synthetic service: TypeScript auth and payments code, a Python worker, tests on both sides, a Dockerfile and a compose file. Every name in it is neutral.

Running it

The suites a change reaches, in one call:

make test

The change is the working tree against HEAD, untracked files included, or the last commit when the tree is clean. A package’s tests run when the change touches that package or one it depends on, through its pyproject.toml or its tests’ imports. The eval stack runs when the change touches core, indexer, api or mcp outside their tests, eval/, docker-compose.eval.yaml or skills/. The target prints what it picked and every file no suite reads; a change to the docs alone runs nothing.

Every suite, whatever changed:

make test-all

It runs test-py, the suite that needs no stack. Then test-eval builds the images from the working tree, brings up the eval stack on them, runs the benchmark against the baseline and then the SQL and MCP tool tests, and removes the stack whether they pass or not. Each target also runs alone, and ARGS reaches pytest, in make test too:

make test-py ARGS="-k chunks"
make test-eval

To iterate on the benchmark, keep a stack up between runs. eval-up starts the images as they are, so build them first when the code changed:

make build
make eval-up
make eval
make eval-down

make eval runs the benchmark first, then eval-checks, which is what test-eval runs on its own stack. Pass benchmark flags through ARGS:

make eval ARGS=--rerank=false
make eval ARGS=--update-baseline
make eval ARGS=--tolerance=0.05

To re-record the baseline in one step - build, a fresh stack, the benchmark with --update-baseline, and the stack removed:

make eval-baseline

Reading the benchmark

Each query in eval/queries.yaml names the files a good answer must reach. The benchmark scores the reranked search_code rows and the get_context packet for every query:

Metric	Meaning
recall_at_5, 10	share of expected files in the top 5 or 10 search files
mrr	1 / rank of the first expected file the search returns
context_recall	share of expected files present in the packet
context_precision	share of packet entries that sit in an expected file
tokens_mean	tokens the packet spent
*_ms_p50, p95	latency of the search and of the packet

The results go to eval/results/<mode>.json, which git ignores. The mode is lexical when no embedding server answered and hybrid when one did, with -norerank appended for --rerank=false. Each mode has its own baseline in eval/baseline.<mode>.json.

A callers, tests or impact query that names a symbol is also asked of find_callers, find_tests or impact_analysis. The files of the answer, the definition first, are scored as recall_at_5, recall and mrr under by_tool, beside the search score for the same query. by_tool is reported, not gated.

An overview query expects directories, by their id (src/auth/), and its packet is built with detail: "summary". Every other kind is built with detail: "source", so its scores stay comparable with the baselines recorded before the summary ladder existed.

The baseline gate

The run fails when any quality metric falls more than the tolerance (0.02 by default) below the checked-in baseline. Latency and tokens are reported but never gated. An improvement does not update the baseline on its own: record it on purpose with --update-baseline and commit the file with the change that earned it.

Adding a query

Add an entry to eval/queries.yaml with an id, a kind (discovery, config, symbol, callers, tests, impact or overview), the query and the expect list. A callers, tests or impact query takes a symbol as well, which is what the matching tool is asked about. If the answer needs code the corpus lacks, add it to eval/corpus/alpha first. Then run the benchmark and update the baseline, because a new query moves every mean: make eval-baseline.

Replaying the suggestions

The corpus is small and written by hand; the questions the graph actually failed on are recorded with the gaps. A suggestion saved with query and nodes names a question and the code that should have answered it, and make replay asks every such question of the running stack again:

make replay

It prints the share of questions whose node search_code ranks in its first five and ten results, the share get_context reaches, the same for resolved suggestions alone, and each question still missed. The full result lands in eval/results/replay-<time>.json. It reads the fifty most hit suggestions and gates nothing: a resolved suggestion whose question still misses is a fix that did not land.

Holding the tools to their answers

A change that must not alter what the tools answer - a refactoring, a dependency upgrade - is checked against a recording. Record on the code as it was, then check the code as it is, each on a freshly built eval stack:

make parity-record
make parity

Both make the same calls: every tool once, then other scopes, shapes and refusals. make parity prints a diff for every answer that changed. Times and hashes are masked, and files found by text search are compared in one order, since the tree search returns them as it finds them. The recording is eval/parity/answers.json, which git ignores.

The dashboard’s /api is held the same way, on the same stack:

make parity-web-record
make parity-web

That walk asks every address once, then its refusals, then writes that put back what they changed. Its recording is eval/parity/web.json.

Where the code is

• packages/mcp/eval/run.py - the benchmark runner.
• packages/mcp/eval/replay.py - the suggestion replay.
• packages/mcp/eval/parity.py - the recorder behind make parity.
• packages/web/eval/parity.py - the recorder behind make parity-web.
• packages/mcp/tests/ - the MCP server’s unit tests and test_e2e.py, the MCP tool test, marker db.
• packages/core/tests/test_sql_db.py - every package’s SQL and the queue, marker db.
• docker-compose.eval.yaml - the throwaway stack.
• .github/workflows/test.yml - CI: unit tests on every push, then the eval stack built from the commit’s own images.