Evaluation
Benchmarks should match the job.
How we connect retrieval research, task-specific checks, and business milestones.
A benchmark is useful when it helps answer a practical question: can this system do the work we are asking it to do?
Start with evidence
We publish memory-retrieval results with the dataset, metric, and evaluation context. Those results describe a defined test. A business workflow also needs examples drawn from the work it will actually encounter.
For a tailored evaluation, we assemble representative questions, source material, expected results, and failure cases. That might include finding a customer commitment, reconciling a document with a source, or checking whether an agent completed an authorized action.
tau-bench: agent behavior with tools and policies YouWo preliminary memory benchmark recordCheck the system, then the outcome
Compare the harness and the model baseline on the same tasks, inputs, permissions, and budget. Record completion, accuracy, elapsed time, and cost. Inspect execution traces and verify outputs against their sources; use human review where the expected result requires judgment.
The business milestone remains a separate measure. Retrieval accuracy, a correct tool call, and revenue growth describe different things. We choose the measures that fit the engagement, record a baseline, and use the results to decide what to improve or automate next.
Measure the cost of a correct result
Token count by itself is not a business result. We compare the cost and elapsed time per accepted answer, completed report, or verified action. An agent that uses more context but reduces rework may be more efficient than one that produces cheap, unusable drafts.
For each representative task, save the source evidence, baseline result, agent result, judge or reviewer decision, tool trace, time, and model cost. Publish the task set and evaluation setup before claiming a percentage improvement. We have not published a YouWo token-efficiency benchmark yet.