Evaluations
Score every checkpoint of a training run against a benchmark, and name the best one.
An evaluation runs one or more models over a benchmark and
scores the outputs. Pass a training_run and every checkpoint that run produced is scored in
one operation, against the same cases and the same pass threshold, and the highest scoring
one is named. That is the usual shape: a run writes a checkpoint at each stage of training,
and the question is which of them to serve.
Pass a model instead to score a single model, which is how you evaluate a published base or
one checkpoint by its nucleus:// path.
A score on its own is hard to act on. Pass compare_to to score a baseline, usually the base
model you fine-tuned, on the same benchmark in the same run. The result then carries a
before-and-after pair, baseline_score beside score.
The benchmark carries the criteria and the bar, so a result reports a score per criterion and whether each checkpoint cleared the threshold. Where the benchmark is backed by an outcome source, it scores against whatever that source last read, so the same model can be scored again as new records arrive. List evaluations filtered by benchmark returns that history in order, which is how a score is tracked across releases.
Create an evaluation
/v1/evaluationsscope: inferStarts an evaluation. Evaluations run asynchronously: the response is 202 Accepted with an
operation ID. Poll it via
Poll an operation, or poll
Retrieve an evaluation until status reaches succeeded.
Pass exactly one of training_run and model.
Body parameters
benchmarkstringrequiredID of the benchmark to score against. It carries the cases, the criteria and the pass threshold.
training_runstringID of a training run. Every checkpoint the run produced is
scored in this operation. Mutually exclusive with model.
modelstringA single model to evaluate: a model ID, or a nucleus:// path such as one checkpoint's
result_model. Mutually exclusive with training_run.
metricstringHow outputs are scored: exact_match, rouge, pass_rate or grader.
compare_tostringOptional baseline model ID, scored on the same benchmark so the response includes
baseline_score alongside score.
curl https://api.nucleus-ai.io/v1/evaluations \
-H "Authorization: Bearer $NUCLEUS_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"benchmark": "bm_7Qa",
"training_run": "run_3Bz",
"metric": "grader",
"compare_to": "Qwen/Qwen3-8B"
}'{
"id": "eval_2Wp",
"status": "running",
"operation": "op_88a1"
}Retrieve an evaluation
/v1/evaluations/{evaluation_id}scope: readReturns an evaluation and, once it has succeeded, its results. score is the headline number
for the run, on a scale of 0 to 1, and n is the number of cases scored. baseline_score
appears when the evaluation was created with compare_to.
For a training_run evaluation, checkpoints carries one entry per checkpoint. Each reports
its own score, a breakdown across the benchmark's criteria, whether it cleared the
threshold, and the price of serving it per million tokens. best_checkpoint names the
highest scoring one, and the headline score is that checkpoint's score.
A checkpoint still being scored appears with status running and no score, so the response
is readable while the run is in flight.
Path parameters
evaluation_idstringrequiredID of the evaluation, from the create response.
curl https://api.nucleus-ai.io/v1/evaluations/eval_2Wp \
-H "Authorization: Bearer $NUCLEUS_API_KEY"{
"id": "eval_2Wp",
"status": "succeeded",
"benchmark": "bm_7Qa",
"training_run": "run_3Bz",
"metric": "grader",
"score": 0.942,
"baseline_score": 0.642,
"n": 1200,
"best_checkpoint": "ckpt_000675",
"checkpoints": [
{
"checkpoint_id": "ckpt_000225",
"step": 225,
"status": "succeeded",
"score": 0.886,
"passed": false,
"serving_price_per_1m": "0.31",
"run_cost": "0.53",
"criteria": [
{ "name": "Correct routing", "score": 0.901 },
{ "name": "Resolution rate", "score": 0.874 },
{ "name": "Approved tone", "score": 0.869 },
{ "name": "SLA met", "score": 0.897 }
]
},
{
"checkpoint_id": "ckpt_000450",
"step": 450,
"status": "succeeded",
"score": 0.91,
"passed": false,
"serving_price_per_1m": "0.31",
"run_cost": "0.53"
},
{
"checkpoint_id": "ckpt_000675",
"step": 675,
"status": "succeeded",
"score": 0.942,
"passed": true,
"serving_price_per_1m": "0.31",
"run_cost": "0.53"
},
{
"checkpoint_id": "ckpt_000900",
"step": 900,
"status": "succeeded",
"score": 0.924,
"passed": false,
"serving_price_per_1m": "0.31",
"run_cost": "0.53"
}
]
}List evaluations
/v1/evaluationsscope: readReturns evaluations in your account. Filter by benchmark to get one benchmark's history in
order, which is how a model's score is tracked across releases: each entry carries the run it
scored, its headline score and when it ran, so a drop reads as a number against the release
it started at. See Pagination for how limit, offset and the cursor object
behave across list endpoints.
Query parameters
benchmarkstringOnly evaluations against this benchmark.
training_runstringOnly evaluations of this training run.
orderstringdefault: descasc for oldest first, which is the order a drift chart wants, or desc for newest first.
limitintegerMaximum number of evaluations to return.
offsetintegerNumber of evaluations to skip from the start of the list.
curl "https://api.nucleus-ai.io/v1/evaluations?benchmark=bm_7Qa&order=asc" \
-H "Authorization: Bearer $NUCLEUS_API_KEY"{
"evaluations": [
{
"id": "eval_1Ka",
"training_run": "run_2Ax",
"score": 0.918,
"n": 1200,
"created_at": "2026-07-14T09:00:00Z"
},
{
"id": "eval_2Wp",
"training_run": "run_3Bz",
"score": 0.942,
"n": 1200,
"created_at": "2026-09-08T09:20:00Z"
}
]
}