Setup continuous evaluations

​

How it works

1. Sample incoming spans — A percentage of LLM spans associated with your prompt are selected based on the sample rate.
2. Run evaluators — The sampled spans’ output is evaluated against the evaluators you configured for the prompt.
3. Store results — Evaluation scores are attached to the spans and visible in the logs and charts.

​

Configure continuous evaluations

1

Select a prompt

Navigate to the Monitor section of a monitored project and select the prompt you want to continuously evaluate.

2

Set the sample rate

Define the continuous evaluation sample rate — a value between 0 and 1:

Sample rate	Behavior
0	No logs are evaluated. Continuous evaluation is disabled.
0.5	50% of incoming spans are randomly sampled and evaluated.
1	100% of incoming spans are evaluated.

Start with a low sample rate (e.g., 0.1 or 0.2) and increase it as you gain confidence in your evaluator configuration. A rate of 1.0 evaluates every single request, which provides complete coverage but incurs additional cost.

3

Configure evaluators

Set up the evaluators that will run on sampled spans. See Setup Evaluators for the available evaluator types:

• LLM-as-a-Judge — Assess response quality against a rubric.
• JavaScript — Validate output format and business logic.
• Text Matcher — Check for required or banned patterns.
• Cost, Latency, Response Length — Track operational metrics.

​

Override the sample rate

const span = trace.logSpan({
  name: "LLM Completion",
  promptId: "your-prompt-id",
  runEvaluation: true,
  tags: ["high-priority"],
});

span.update({ runEvaluation: true });

span = trace.log_span(
    name="LLM Completion",
    prompt_id="your-prompt-id",
    run_evaluation=True,
    tags=["high-priority"],
)

span.update(run_evaluation=True)

{
  "projectId": "your-project-id",
  "span": {
    "name": "LLM Completion",
    "traceReferenceId": "your-trace-ref-id",
    "promptId": "your-prompt-id",
    "runEvaluation": true,
    "content": { ... }
  }
}

adaline-span-run-evaluation: true

​

View results

​

Best practices

• Start with a low sample rate — Begin at 0.1–0.2 to validate your evaluator configuration before scaling up.
• Use representative evaluators — Choose evaluators that measure the dimensions most important to your use case.
• Monitor the eval score chart — Watch the Avg eval score chart for trend changes that indicate quality regressions.
• Combine with alerts — Set up alerts to get notified when eval scores drop below a threshold.
• Iterate on evaluators — Refine your evaluator rubrics and thresholds based on what you observe in production.

​

Next steps