API docs
API reference

Training runs

Allocate GPUs and drive the low-level training loop with forward, backward and optimizer primitives.

A training run is a live GPU allocation holding a LoRA training session against a base model. It is the entry point to the low-level path: instead of submitting a managed fine-tuning job, you drive each step yourself with Forward-backward pass and Optimizer step, saving checkpoints along the way. See the training loop guide for the full walkthrough.

Every route under a run is keyed on the training_run_id returned at creation. The training primitives take token IDs, not text: render your conversations first with Render a conversation, or tokenize yourself. See Working with tokens for why the split exists.

Runs hold GPUs until you release them. Close a training run when you are finished; your saved checkpoints survive the run by default.

Create a training run

POST/v1/training-runsscope: train

Allocates GPUs and returns the run ID that every other training route is keyed on. The body has two modes: start a new run from a base model, or resume from a saved state. Send one set of fields, not both.

Body parameters (new run)

base_modelstringrequired

The base model to train, for example Qwen/Qwen3-8B. See List base models for the catalogue.

rankintegerdefault: 32

LoRA rank of the adapter. Higher ranks give the adapter more capacity at the cost of more memory and slower steps.

seedinteger

Seed for reproducible training. Nullable; omit it to let the platform choose.

train_mlpbooleandefault: true

Whether to train the MLP layers.

train_attnbooleandefault: true

Whether to train the attention layers.

train_unembedbooleandefault: true

Whether to train the unembedding layer.

user_metadataobject

Arbitrary key-value metadata stored on the run and returned when you retrieve it.

Body parameters (resume)

from_pathstring

A nucleus:// checkpoint path to resume from, for example nucleus://run_3Bz:train:0/sampler_weights/000240. The new run inherits the base model and adapter configuration of the saved state.

restore_optimizerbooleandefault: false

Also restore the optimizer state saved with the checkpoint, so momentum and variance estimates carry over.

weights_access_tokenstring

Token authorising access when from_path points at weights owned by another account, for example a published checkpoint.

Creating a run allocates GPUs and is not idempotent: retrying a timed-out request can allocate a second run. Send an Idempotency-Key header if your client retries automatically. See Errors and retries.

curl https://api.nucleus-ai.io/v1/training-runs \
  -H "Authorization: Bearer $NUCLEUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"base_model": "Qwen/Qwen3-8B", "rank": 32}'
201 Created
{
  "training_run_id": "run_3Bz",
  "session_id": "sess_A1",
  "base_model": "Qwen/Qwen3-8B",
  "is_lora": true,
  "lora_rank": 32,
  "status": "ready"
}

List training runs

GET/v1/training-runsscope: read

Returns your training runs as a paginated list. See Pagination for how limit, offset and the cursor object behave across list endpoints.

Passing path turns the call into a reverse lookup: given a nucleus:// checkpoint path, it returns the run that owns it.

Query parameters

limitintegerdefault: 20

Maximum number of runs to return.

offsetintegerdefault: 0

Number of runs to skip from the start of the list.

access_scopestring

owned returns only runs you created; accessible includes every run your key can read.

project_idstring

Filter to runs in one project.

pathstring

A nucleus:// checkpoint path. Returns the run that owns the checkpoint.

curl "https://api.nucleus-ai.io/v1/training-runs?limit=20" \
  -H "Authorization: Bearer $NUCLEUS_API_KEY"
200 OK
{
  "training_runs": [
    {
      "training_run_id": "run_3Bz",
      "base_model": "Qwen/Qwen3-8B",
      "status": "ready"
    }
  ],
  "cursor": { "offset": 0, "limit": 20, "total_count": 41 }
}

Retrieve a training run

GET/v1/training-runs/{run_id}scope: read

Returns one run's metadata and its current step count.

Path parameters

run_idstringrequired

ID of the run, from the create response.

Query parameters

access_scopestring

owned restricts the lookup to runs you created; accessible includes every run your key can read.

curl https://api.nucleus-ai.io/v1/training-runs/run_3Bz \
  -H "Authorization: Bearer $NUCLEUS_API_KEY"
200 OK
{
  "training_run_id": "run_3Bz",
  "base_model": "Qwen/Qwen3-8B",
  "is_lora": true,
  "lora_rank": 32,
  "steps_completed": 240,
  "status": "ready"
}

Close a training run

DELETE/v1/training-runs/{run_id}scope: train

Closes the run and releases its GPUs. An open run holds its allocation whether or not you are stepping it, so close runs as soon as you are finished with them.

Saved checkpoints are kept by default and remain loadable into new runs after this one is gone.

Path parameters

run_idstringrequired

ID of the run to close.

Query parameters

keep_checkpointsbooleandefault: true

Set to false to delete the run's checkpoints along with the run.

curl -X DELETE https://api.nucleus-ai.io/v1/training-runs/run_3Bz \
  -H "Authorization: Bearer $NUCLEUS_API_KEY"
204 No Content
(empty body)

Get run metrics

GET/v1/training-runs/{run_id}/metricsscope: read

Returns the run's metric series: loss, gradient norm and learning rate per step. The platform records the full history server-side, so you do not need to reconstruct the curve from individual gradient responses, and a dropped connection loses nothing.

Path parameters

run_idstringrequired

ID of the run.

Query parameters

from_stepinteger

First step to include.

to_stepinteger

Last step to include.

keysstring

Comma-separated metric names to return, for example loss,grad_norm,lr. Omit to return all recorded metrics.

curl "https://api.nucleus-ai.io/v1/training-runs/run_3Bz/metrics?from_step=238&keys=loss,grad_norm,lr" \
  -H "Authorization: Bearer $NUCLEUS_API_KEY"
200 OK
{
  "series": [
    { "step": 238, "loss": 0.74, "grad_norm": 1.12, "lr": 8e-5 },
    { "step": 239, "loss": 0.73, "grad_norm": 1.09, "lr": 8e-5 }
  ]
}

Forward pass

POST/v1/training-runs/{run_id}/forwardscope: train

Evaluates the loss on a batch without computing gradients. Use it to measure held-out loss during a training loop, or to score data before deciding what to train on.

The batch is token IDs, not text: render your conversations with Render a conversation first. See Working with tokens.

Path parameters

run_idstringrequired

ID of the run.

Body parameters

dataarrayrequired

The batch. Each element pairs a model_input (the token IDs to run through the model) with loss_fn_inputs (the tensors the chosen loss needs, for example target tokens and per-token weights for cross-entropy).

loss_fnstringrequired

The loss to evaluate: cross_entropy, importance_sampling, ppo, cispo or dro.

loss_fn_configobject

Loss-specific settings, for example clipping thresholds.

curl https://api.nucleus-ai.io/v1/training-runs/run_3Bz/forward \
  -H "Authorization: Bearer $NUCLEUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "loss_fn": "cross_entropy",
    "data": [
      {
        "model_input": [151644, 872, 198, 3838, 374],
        "loss_fn_inputs": {
          "target_tokens": [872, 198, 3838, 374, 151645],
          "weights": [0, 0, 1, 1, 1]
        }
      }
    ]
  }'
200 OK
{
  "loss": 0.8123,
  "metrics": { "num_tokens": 4096 }
}

Forward-backward pass

POST/v1/training-runs/{run_id}/forward-backwardscope: train

The core gradient primitive. Runs the batch forward, computes the chosen loss and accumulates gradients on the run. Gradients accumulate across calls until you apply them with Optimizer step, so you can build large effective batches from several smaller requests.

Built-in loss functions cover supervised fine-tuning and reinforcement learning: cross-entropy, importance sampling, PPO, CISPO and DRO. For losses beyond these, use Custom-loss forward-backward.

As with Forward pass, inputs are token IDs. Render conversations with Render a conversation, or tokenize yourself; see Working with tokens.

Path parameters

run_idstringrequired

ID of the run.

Body parameters

dataarrayrequired

The batch. Each element pairs a model_input (the token IDs to run through the model) with loss_fn_inputs (the tensors the chosen loss needs, for example target tokens and per-token weights for cross-entropy, or advantages and reference logprobs for the RL losses).

loss_fnstringrequired

The loss to compute: cross_entropy, importance_sampling, ppo, cispo or dro.

loss_fn_configobject

Loss-specific settings, for example clipping thresholds.

curl https://api.nucleus-ai.io/v1/training-runs/run_3Bz/forward-backward \
  -H "Authorization: Bearer $NUCLEUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "loss_fn": "cross_entropy",
    "data": [
      {
        "model_input": [151644, 872, 198, 3838, 374],
        "loss_fn_inputs": {
          "target_tokens": [872, 198, 3838, 374, 151645],
          "weights": [0, 0, 1, 1, 1]
        }
      }
    ]
  }'
200 OK
{
  "loss": 0.7941,
  "metrics": { "num_tokens": 4096, "grad_norm": 1.08 }
}

Custom-loss forward-backward

POST/v1/training-runs/{run_id}/forward-backward-customscope: train

Computes gradients for a loss you define yourself, as a two-leg exchange. The first leg runs the batch forward and returns per-token log probabilities together with an exchange_id. You compute the gradient of your loss with respect to those logprobs locally, then submit the per-token gradients back against the same exchange_id; the platform completes the backward pass and accumulates the result on the run, ready for Optimizer step.

The exchange is short-lived: expires_in in the first-leg response gives the window in seconds. Submit your gradients before it lapses, or start again with a fresh first leg.

Path parameters

run_idstringrequired

ID of the run.

Body parameters

dataarrayrequired

The batch. Each element carries a model_input of token IDs, as in Forward pass.

loss_type_inputstringrequired

The quantity to return for your loss computation. Currently logprobs: the first leg returns per-token log probabilities for each sequence in the batch.

curl https://api.nucleus-ai.io/v1/training-runs/run_3Bz/forward-backward-custom \
  -H "Authorization: Bearer $NUCLEUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "loss_type_input": "logprobs",
    "data": [
      { "model_input": [151644, 872, 198, 3838, 374] }
    ]
  }'
200 OK
{
  "exchange_id": "xch_5Tn",
  "logprobs": [[-0.21, -1.84, ...]],
  "expires_in": 60
}

Optimizer step

POST/v1/training-runs/{run_id}/optim-stepscope: train

Applies the accumulated gradients with an AdamW update and clears them, advancing the run by one step. Call it after one or more forward-backward calls.

Note that weight_decay defaults to 0.0, unlike PyTorch's AdamW. Set it explicitly if you are porting a training loop that relies on PyTorch's default.

Path parameters

run_idstringrequired

ID of the run.

Body parameters

learning_ratefloatrequired

Learning rate for this step.

betas[float, float]

AdamW beta coefficients.

epsfloat

AdamW epsilon term.

weight_decayfloatdefault: 0.0

Decoupled weight decay. Defaults to 0.0, not PyTorch's 0.01.

curl https://api.nucleus-ai.io/v1/training-runs/run_3Bz/optim-step \
  -H "Authorization: Bearer $NUCLEUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"learning_rate": 8e-5}'
200 OK
{
  "step": 241,
  "lr_applied": 8e-5
}

Load state

POST/v1/training-runs/{run_id}/load-statescope: train

Swaps saved weights into the live run without closing it or opening a new one. Use it to roll back to an earlier checkpoint mid-loop, or to continue from weights produced by a different run.

Path parameters

run_idstringrequired

ID of the run to load into.

Body parameters

pathstringrequired

A nucleus:// checkpoint path to load, for example nucleus://run_2Aa:train:0/weights/000100.

restore_optimizerbooleandefault: false

Also restore the optimizer state saved with the checkpoint.

weights_access_tokenstring

Token authorising access when path points at weights owned by another account, for example a published checkpoint.

curl https://api.nucleus-ai.io/v1/training-runs/run_3Bz/load-state \
  -H "Authorization: Bearer $NUCLEUS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"path": "nucleus://run_2Aa:train:0/weights/000100"}'
200 OK
{
  "training_run_id": "run_3Bz",
  "loaded_from": "nucleus://run_2Aa:train:0/weights/000100",
  "optimizer_restored": false
}
Training runs | Nucleus AI API