Evaluations let you systematically measure model quality. Run a prompt against one or more models, score the outputs with rubrics, and track how scores change over time on the Score Trends page.
Evaluations
An evaluation runs a single prompt against one or more agents and scores each response. When you run an evaluation you choose:
- The prompt to send.
- Which agents to run it against — one evaluation can compare multiple models side by side.
- Which rubrics to apply to score the outputs.
- One or more tags to label the run.
Each agent response is independently scored by every selected rubric. Scores range from
0 to 1 in 0.1 increments. The individual rubric scores are then aggregated
into an overall score for that response (binary rubrics such as Contains and Not Contains are 0/1).
Tags
Tags are how Score Trends groups and compares results. When you open Score Trends you select a tag, and every evaluation run that carries that tag is plotted together — so you can see whether quality is improving or regressing over time.
Apply at least one tag to every evaluation run, and use each tag consistently for one type of evaluation only. If you reuse a tag across evaluations that test different things, Score Trends will plot them side by side as if they measure the same quality dimension — making the chart meaningless.
Templates
A template is a saved evaluation configuration. It stores the prompt, the selected agents, the rubrics to apply, and any default tags — everything needed to rerun the exact same evaluation later.
Use templates when you want to rerun an evaluation regularly: after switching models, after changing a system prompt, or as part of a release checklist. Running from a template guarantees that nothing about the evaluation setup has drifted between runs.
Rubrics
Rubrics define how a response is scored. Each rubric produces a score between 0 and 1 (in 0.1 increments). You can attach multiple rubrics to a single evaluation; the scores are averaged to produce the final result. Weave supports five rubric types:
LLM Judge
An LLM Judge rubric sends the model's output to a separate judge model along with your scoring instructions, and the judge returns a 0–1 score. Use this for nuanced criteria that are hard to express as a simple rule — tone, helpfulness, factual accuracy, adherence to a style guide, and so on.
The judge model is independent of the model being evaluated, so you can use a strong frontier model as the judge regardless of what agent produced the response.
Contains
A Contains rubric checks whether the model's output includes a specific string. The score is 1 if the string is present and 0 if it is not. Use this to verify that a response mentions a required keyword, includes a disclaimer, or produces output in an expected format.
Does Not Contain
The inverse of Contains — scores 1 when the output does not include the specified string, and 0 when it does. Useful for checking that a model avoids certain phrases, doesn't leak sensitive strings, or refrains from hallucinating a particular value.
Regex
A Regex rubric evaluates the model's output against a regular expression. The score is 1 if the pattern matches and 0 if it does not. Use this for structured output validation — checking that a response is valid JSON, matches a date format, starts with a specific prefix, and so on.
Tool Calls
A Tool Calls rubric scores whether the model made the expected tool calls during a response. Each rubric holds one or more requirements; each requirement specifies an optional tool name and an optional count:
- No tool name, no count — the model must call any tool at least once.
- No tool name, count N — the model must make at least N tool calls in total, regardless of which tools.
- Tool name, no count — the model must call that specific tool at least once.
- Tool name, count N — the model must call that specific tool at least N times.
The rubric score is the fraction of requirements met:
requirements met ÷ total requirements. A rubric with a single requirement
scores either 0 or 1. A rubric with four requirements where three are satisfied scores 0.75.
This lets you express compound tool-use expectations — for example, requiring that the model
calls read_file at least twice and calls write_file at
least once — and get partial credit when only some are satisfied.
