Training runs
Allocate GPUs and drive the low-level training loop with forward, backward and optimizer primitives.
A training run is a live GPU allocation holding a LoRA training session against a base model. It is the entry point to the low-level path: instead of submitting a managed fine-tuning job, you drive each step yourself with Forward-backward pass and Optimizer step, saving checkpoints along the way. See the training loop guide for the full walkthrough.
Every route under a run is keyed on the training_run_id returned at creation. The training
primitives take token IDs, not text: render your conversations first with
Render a conversation, or tokenize
yourself. See Working with tokens for why the split exists.
Runs hold GPUs until you release them. Close a training run when you are finished; your saved checkpoints survive the run by default.
Create a training run
/v1/training-runsscope: trainAllocates GPUs and returns the run ID that every other training route is keyed on. The body has two modes: start a new run from a base model, or resume from a saved state. Send one set of fields, not both.
Body parameters (new run)
base_modelstringrequiredThe base model to train, for example Qwen/Qwen3-8B. See List base
models for the catalogue.
rankintegerdefault: 32LoRA rank of the adapter. Higher ranks give the adapter more capacity at the cost of more memory and slower steps.
seedintegerSeed for reproducible training. Nullable; omit it to let the platform choose.
train_mlpbooleandefault: trueWhether to train the MLP layers.
train_attnbooleandefault: trueWhether to train the attention layers.
train_unembedbooleandefault: trueWhether to train the unembedding layer.
user_metadataobjectArbitrary key-value metadata stored on the run and returned when you retrieve it.
Body parameters (resume)
from_pathstringA nucleus:// checkpoint path to resume from, for example
nucleus://run_3Bz:train:0/sampler_weights/000240. The new run inherits the base model and
adapter configuration of the saved state.
restore_optimizerbooleandefault: falseAlso restore the optimizer state saved with the checkpoint, so momentum and variance estimates carry over.
weights_access_tokenstringToken authorising access when from_path points at weights owned by another account, for
example a published checkpoint.
Creating a run allocates GPUs and is not idempotent: retrying a timed-out request can
allocate a second run. Send an Idempotency-Key header if your client retries automatically.
See Errors and retries.
curl https://api.nucleus-ai.io/v1/training-runs \
-H "Authorization: Bearer $NUCLEUS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"base_model": "Qwen/Qwen3-8B", "rank": 32}'{
"training_run_id": "run_3Bz",
"session_id": "sess_A1",
"base_model": "Qwen/Qwen3-8B",
"is_lora": true,
"lora_rank": 32,
"status": "ready"
}List training runs
/v1/training-runsscope: readReturns your training runs as a paginated list. See Pagination for how
limit, offset and the cursor object behave across list endpoints.
Passing path turns the call into a reverse lookup: given a nucleus:// checkpoint path, it
returns the run that owns it.
Query parameters
limitintegerdefault: 20Maximum number of runs to return.
offsetintegerdefault: 0Number of runs to skip from the start of the list.
access_scopestringowned returns only runs you created; accessible includes every run your key can read.
project_idstringFilter to runs in one project.
pathstringA nucleus:// checkpoint path. Returns the run that owns the checkpoint.
curl "https://api.nucleus-ai.io/v1/training-runs?limit=20" \
-H "Authorization: Bearer $NUCLEUS_API_KEY"{
"training_runs": [
{
"training_run_id": "run_3Bz",
"base_model": "Qwen/Qwen3-8B",
"status": "ready"
}
],
"cursor": { "offset": 0, "limit": 20, "total_count": 41 }
}Retrieve a training run
/v1/training-runs/{run_id}scope: readReturns one run's metadata and its current step count.
Path parameters
run_idstringrequiredID of the run, from the create response.
Query parameters
access_scopestringowned restricts the lookup to runs you created; accessible includes every run your key can
read.
curl https://api.nucleus-ai.io/v1/training-runs/run_3Bz \
-H "Authorization: Bearer $NUCLEUS_API_KEY"{
"training_run_id": "run_3Bz",
"base_model": "Qwen/Qwen3-8B",
"is_lora": true,
"lora_rank": 32,
"steps_completed": 240,
"status": "ready"
}Close a training run
/v1/training-runs/{run_id}scope: trainCloses the run and releases its GPUs. An open run holds its allocation whether or not you are stepping it, so close runs as soon as you are finished with them.
Saved checkpoints are kept by default and remain loadable into new runs after this one is gone.
Path parameters
run_idstringrequiredID of the run to close.
Query parameters
keep_checkpointsbooleandefault: trueSet to false to delete the run's checkpoints along with the run.
curl -X DELETE https://api.nucleus-ai.io/v1/training-runs/run_3Bz \
-H "Authorization: Bearer $NUCLEUS_API_KEY"(empty body)Get run metrics
/v1/training-runs/{run_id}/metricsscope: readReturns the run's metric series: loss, gradient norm and learning rate per step. The platform records the full history server-side, so you do not need to reconstruct the curve from individual gradient responses, and a dropped connection loses nothing.
Path parameters
run_idstringrequiredID of the run.
Query parameters
from_stepintegerFirst step to include.
to_stepintegerLast step to include.
keysstringComma-separated metric names to return, for example loss,grad_norm,lr. Omit to return all
recorded metrics.
curl "https://api.nucleus-ai.io/v1/training-runs/run_3Bz/metrics?from_step=238&keys=loss,grad_norm,lr" \
-H "Authorization: Bearer $NUCLEUS_API_KEY"{
"series": [
{ "step": 238, "loss": 0.74, "grad_norm": 1.12, "lr": 8e-5 },
{ "step": 239, "loss": 0.73, "grad_norm": 1.09, "lr": 8e-5 }
]
}Forward pass
/v1/training-runs/{run_id}/forwardscope: trainEvaluates the loss on a batch without computing gradients. Use it to measure held-out loss during a training loop, or to score data before deciding what to train on.
The batch is token IDs, not text: render your conversations with Render a conversation first. See Working with tokens.
Path parameters
run_idstringrequiredID of the run.
Body parameters
dataarrayrequiredThe batch. Each element pairs a model_input (the token IDs to run through the model) with
loss_fn_inputs (the tensors the chosen loss needs, for example target tokens and per-token
weights for cross-entropy).
loss_fnstringrequiredThe loss to evaluate: cross_entropy, importance_sampling, ppo, cispo or dro.
loss_fn_configobjectLoss-specific settings, for example clipping thresholds.
curl https://api.nucleus-ai.io/v1/training-runs/run_3Bz/forward \
-H "Authorization: Bearer $NUCLEUS_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"loss_fn": "cross_entropy",
"data": [
{
"model_input": [151644, 872, 198, 3838, 374],
"loss_fn_inputs": {
"target_tokens": [872, 198, 3838, 374, 151645],
"weights": [0, 0, 1, 1, 1]
}
}
]
}'{
"loss": 0.8123,
"metrics": { "num_tokens": 4096 }
}Forward-backward pass
/v1/training-runs/{run_id}/forward-backwardscope: trainThe core gradient primitive. Runs the batch forward, computes the chosen loss and accumulates gradients on the run. Gradients accumulate across calls until you apply them with Optimizer step, so you can build large effective batches from several smaller requests.
Built-in loss functions cover supervised fine-tuning and reinforcement learning: cross-entropy, importance sampling, PPO, CISPO and DRO. For losses beyond these, use Custom-loss forward-backward.
As with Forward pass, inputs are token IDs. Render conversations with Render a conversation, or tokenize yourself; see Working with tokens.
Path parameters
run_idstringrequiredID of the run.
Body parameters
dataarrayrequiredThe batch. Each element pairs a model_input (the token IDs to run through the model) with
loss_fn_inputs (the tensors the chosen loss needs, for example target tokens and per-token
weights for cross-entropy, or advantages and reference logprobs for the RL losses).
loss_fnstringrequiredThe loss to compute: cross_entropy, importance_sampling, ppo, cispo or dro.
loss_fn_configobjectLoss-specific settings, for example clipping thresholds.
curl https://api.nucleus-ai.io/v1/training-runs/run_3Bz/forward-backward \
-H "Authorization: Bearer $NUCLEUS_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"loss_fn": "cross_entropy",
"data": [
{
"model_input": [151644, 872, 198, 3838, 374],
"loss_fn_inputs": {
"target_tokens": [872, 198, 3838, 374, 151645],
"weights": [0, 0, 1, 1, 1]
}
}
]
}'{
"loss": 0.7941,
"metrics": { "num_tokens": 4096, "grad_norm": 1.08 }
}Custom-loss forward-backward
/v1/training-runs/{run_id}/forward-backward-customscope: trainComputes gradients for a loss you define yourself, as a two-leg exchange. The first leg runs
the batch forward and returns per-token log probabilities together with an exchange_id. You
compute the gradient of your loss with respect to those logprobs locally, then submit the
per-token gradients back against the same exchange_id; the platform completes the backward
pass and accumulates the result on the run, ready for
Optimizer step.
The exchange is short-lived: expires_in in the first-leg response gives the window in
seconds. Submit your gradients before it lapses, or start again with a fresh first leg.
Path parameters
run_idstringrequiredID of the run.
Body parameters
dataarrayrequiredThe batch. Each element carries a model_input of token IDs, as in Forward
pass.
loss_type_inputstringrequiredThe quantity to return for your loss computation. Currently logprobs: the first leg returns
per-token log probabilities for each sequence in the batch.
curl https://api.nucleus-ai.io/v1/training-runs/run_3Bz/forward-backward-custom \
-H "Authorization: Bearer $NUCLEUS_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"loss_type_input": "logprobs",
"data": [
{ "model_input": [151644, 872, 198, 3838, 374] }
]
}'{
"exchange_id": "xch_5Tn",
"logprobs": [[-0.21, -1.84, ...]],
"expires_in": 60
}Optimizer step
/v1/training-runs/{run_id}/optim-stepscope: trainApplies the accumulated gradients with an AdamW update and clears them, advancing the run by one step. Call it after one or more forward-backward calls.
Note that weight_decay defaults to 0.0, unlike PyTorch's AdamW. Set it explicitly if you
are porting a training loop that relies on PyTorch's default.
Path parameters
run_idstringrequiredID of the run.
Body parameters
learning_ratefloatrequiredLearning rate for this step.
betas[float, float]AdamW beta coefficients.
epsfloatAdamW epsilon term.
weight_decayfloatdefault: 0.0Decoupled weight decay. Defaults to 0.0, not PyTorch's 0.01.
curl https://api.nucleus-ai.io/v1/training-runs/run_3Bz/optim-step \
-H "Authorization: Bearer $NUCLEUS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"learning_rate": 8e-5}'{
"step": 241,
"lr_applied": 8e-5
}Load state
/v1/training-runs/{run_id}/load-statescope: trainSwaps saved weights into the live run without closing it or opening a new one. Use it to roll back to an earlier checkpoint mid-loop, or to continue from weights produced by a different run.
Path parameters
run_idstringrequiredID of the run to load into.
Body parameters
pathstringrequiredA nucleus:// checkpoint path to load, for example nucleus://run_2Aa:train:0/weights/000100.
restore_optimizerbooleandefault: falseAlso restore the optimizer state saved with the checkpoint.
weights_access_tokenstringToken authorising access when path points at weights owned by another account, for example a
published checkpoint.
curl https://api.nucleus-ai.io/v1/training-runs/run_3Bz/load-state \
-H "Authorization: Bearer $NUCLEUS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"path": "nucleus://run_2Aa:train:0/weights/000100"}'{
"training_run_id": "run_3Bz",
"loaded_from": "nucleus://run_2Aa:train:0/weights/000100",
"optimizer_restored": false
}