Benchmarks

Define the cases a model is scored on, the criteria each answer is judged by, and the score a checkpoint has to reach.

A benchmark is a test set with a bar. It holds the cases a model is scored on, the criteria each answer is judged by, and one pass threshold, so the standard lives with the set rather than being restated on every run. Reference it from an evaluation and every checkpoint of a training run is scored against the same cases and the same bar.

The cases come from one of two places. A static file uploaded with purpose=evaluation, which is the simple case, or an outcome source, which reads the results your CRM, ticketing queue or ERP already records and refreshes on a schedule. A source-backed benchmark always scores against the current rows, which is what lets a model be re-scored as new records arrive.

If you have no cases yet, Generate a benchmark writes one for you, either from a training run you have already done or from a description of the domain.

Create a benchmark

POST/v1/benchmarksscope: train

Creates a benchmark from a file or a connected outcome source. Pass exactly one of dataset_file and outcome_source.

Body parameters

namestringrequired

A name for the benchmark, shown wherever it is referenced.

dataset_filestring

ID of an uploaded evaluation file holding the cases. Mutually exclusive with outcome_source.

outcome_sourcestring

ID of a connected outcome source. The benchmark reads whatever that source most recently wrote, so a scheduled refresh keeps the set current. Mutually exclusive with dataset_file.

criteriaarray

The criteria each answer is scored on. Each entry takes a name and the number of cases in the set that exercise it. The case counts must not exceed the size of the set. Omit this and the benchmark carries one unnamed criterion over every case.

thresholdnumber

The score a checkpoint has to reach to pass, between 0 and 1. A checkpoint below it is reported as held. Omit it and every checkpoint is scored and none is held.

project_idstring

Attach the benchmark to a project. Keys bound to a different project cannot read it.

curl https://api.nucleus-ai.io/v1/benchmarks \
  -H "Authorization: Bearer $NUCLEUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support triage",
    "dataset_file": "file_4kEv1",
    "threshold": 0.93,
    "criteria": [
      { "name": "Correct routing", "cases": 400 },
      { "name": "Resolution rate", "cases": 300 },
      { "name": "Approved tone", "cases": 300 },
      { "name": "SLA met", "cases": 200 }
    ]
  }'

200 OK

{
  "id": "bm_7Qa",
  "name": "Support triage",
  "dataset_file": "file_4kEv1",
  "cases": 1200,
  "threshold": 0.93,
  "criteria": [
    { "name": "Correct routing", "cases": 400 },
    { "name": "Resolution rate", "cases": 300 },
    { "name": "Approved tone", "cases": 300 },
    { "name": "SLA met", "cases": 200 }
  ],
  "created_at": "2026-09-08T09:12:00Z"
}

Generate a benchmark

POST/v1/benchmarks/generatescope: train

Writes a benchmark for a domain you describe, or from data you already have. Generation runs asynchronously: the response is 202 Accepted with an operation id. Poll it via Poll an operation until it succeeds, and the result carries the benchmark id.

Give it a seed or a description. A seed produces cases drawn from your own traffic, which is what makes the set specific to your domain; a description is the fallback when you have no run to seed from and produces cases from the domain as described.

Body parameters

namestringrequired

A name for the benchmark the run produces.

seed_training_runstring

ID of a training run whose data seeds the cases. One of seed_training_run, seed_file or domain is required.

seed_filestring

ID of an uploaded file to seed from, when the data did not come from a run.

domainstring

A description of the domain to write cases for, used when there is nothing to seed from. Cases generated this way test the domain as described rather than your own traffic.

sample_sizeintegerdefault: 500

How many cases to write.

criteriaarray

The criteria to score against, each a name. Case counts are assigned by the run and reported on the finished benchmark.

thresholdnumber

The pass threshold to set on the benchmark, between 0 and 1.

project_idstring

Attach the benchmark to a project.

curl https://api.nucleus-ai.io/v1/benchmarks/generate \
  -H "Authorization: Bearer $NUCLEUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support triage, generated",
    "seed_training_run": "run_3Bz",
    "sample_size": 1200,
    "threshold": 0.93,
    "criteria": [
      { "name": "Correct routing" },
      { "name": "Resolution rate" }
    ]
  }'

202 Accepted

{
  "id": "bm_9Rd",
  "status": "generating",
  "operation": "op_4f21"
}

Once the operation reaches succeeded, retrieve the benchmark to read the cases and the case counts the run assigned to each criterion. A generated benchmark is an ordinary benchmark from that point on: inspect it, evaluate against it and reference it from later runs.

List benchmarks

GET/v1/benchmarksscope: read

Lists the benchmarks in your account, newest first.

Query parameters

project_idstring

Only benchmarks attached to this project.

limitintegerdefault: 20

How many to return, up to 100.

curl "https://api.nucleus-ai.io/v1/benchmarks?limit=20" \
  -H "Authorization: Bearer $NUCLEUS_API_KEY"

200 OK

{
  "benchmarks": [
    {
      "id": "bm_7Qa",
      "name": "Support triage",
      "cases": 1200,
      "threshold": 0.93,
      "created_at": "2026-09-08T09:12:00Z"
    }
  ]
}

Retrieve a benchmark

GET/v1/benchmarks/{benchmark_id}scope: read

Returns one benchmark, including its criteria, its case counts and the source or file behind it. A source-backed benchmark also reports when that source last refreshed, which is the version any evaluation started now would score against.

Path parameters

benchmark_idstringrequired

ID of the benchmark.

curl https://api.nucleus-ai.io/v1/benchmarks/bm_7Qa \
  -H "Authorization: Bearer $NUCLEUS_API_KEY"

200 OK

{
  "id": "bm_7Qa",
  "name": "Support triage",
  "outcome_source": "osrc_2Km",
  "refreshed_at": "2026-09-08T04:00:00Z",
  "cases": 1200,
  "threshold": 0.93,
  "criteria": [
    { "name": "Correct routing", "cases": 400 },
    { "name": "Resolution rate", "cases": 300 },
    { "name": "Approved tone", "cases": 300 },
    { "name": "SLA met", "cases": 200 }
  ],
  "created_at": "2026-09-08T09:12:00Z"
}

Benchmarks

Create a benchmark

Generate a benchmark

List benchmarks

Retrieve a benchmark

On this page