Evaluate Matching

The evaluate-matching maintenance tool checks the quality of memories that are already matched to a tagging rule.

It loads memories through the rule's filter query, resolves the active assignments for that rule, and then evaluates only the memories that are currently linked to each tag. This tool is focused on match quality, not missed coverage.

How It Runs

The tool has two modes:

• Default mode reuses the current rule-relevant assignments already stored on the fetched memories.
• --rematch reruns matching first, then evaluates those fresh assignments instead of the stored ones.

In both modes, memories are fetched through the same semantic-search path. The difference is only whether assignment data is reused or recomputed.

--rematch does not rerun matching on every fetched memory. It first takes a deterministic sample of up to 2,000 memories using a stable ID-hash strategy, then runs the assign-tags agent on that sample. This makes rematch runs repeatable for the same input set.

Rematch uses the same primary LLM-only assignment agent as production tagging assignment and does not use vector-search based propagation or filtering. Production tagging stores those primary assignments directly; maintenance rematch can still run the second-pass cleanup judge unless --disableCleanup is passed.

After assignments are resolved, the tool evaluates matched memories only. For each category, it checks a deterministic sample of assigned memories, up to 450 per category.

The evaluation samples are deterministic, not random. Within each category, the tool processes judge batches with bounded parallelism to improve throughput while keeping report contents and pass/fail rules unchanged.

What It Reports

The tool writes a matched-only report bundle with:

• *.matched.summary.csv
• *.matched.memories.csv
• *.matched.methods.csv
• *.judge-failures.jsonl when judge failures occur
• a machine-readable summary.json

The CSVs contain per-category rollups plus memory-level verdicts and evidence. The matched-methods report breaks matched results down by assignment method (llm, similarity, or keyword) when that metadata exists.

The optional judge-failures.jsonl file is a debug artifact for cases that later appear as missing_verdict in the memory CSVs. Each line records the matched evaluation phase, category, failure type, and a short message. Failure types include missing insights arrays, omitted memory rows, invalid structured rows, duplicate rows, and unknown returned IDs.

The JSON summary is intended for monitoring and automation. It includes:

• rule metadata and run mode (evaluate-only or rematch)
• total memories evaluated
• matched pass/fail counts
• decisive, incorrect, and uncertain judgment totals
• the top failing categories for matched evaluation

Pass and Fail Rules

A category only passes when all of the following are true:

• there is at least one decisive judgment
• the decisive error rate is 12.5% or lower
• the uncertainty rate is 12.5% or lower

Categories with no decisive evidence, or categories where the reviewer model stays too uncertain, are treated as failing.

Category Context

Category evaluation uses the category title and description when both exist. If the description is empty, the evaluator falls back to the title-only context instead of treating that as an error. If both title and description are empty, the fallback label is Unknown.

For matched evaluation, a memory counts as a correct match when it explicitly fits the category in a meaningful, non-negligible way. The category does not need to be the dominant theme, so multi-topic memories can still pass when the category is clearly supported. Vague, incidental, negligible, loosely related, or contradicted overlap is treated as incorrect, and genuinely incomplete or ambiguous cases remain uncertain.

For missed-match analysis on unmatched memories, use evaluate-matching-coverage.