Tokenization
Convert text to token IDs, decode them back, and render conversations with a chat template and loss mask.
The training and sampling primitives speak token IDs, not text. Every training run route and the sampler generation routes take and return arrays of integers, so converting between text and tokens is the caller's job. These three routes do that conversion server-side with the exact tokenizer of the model you name, so you never need a local tokenizer. See Tokens for background on why the API works this way.
For chat data, skip hand-assembling prompts entirely: Render a conversation applies the model's chat template and returns a loss mask alongside the tokens, ready to feed into a training run.
Tokenize text
/v1/tokenizescope: readConverts text to token IDs using the named model's tokenizer. Use this to build inputs for the forward-backward and sampling routes, or to count tokens before committing to a sequence length.
Body parameters
modelstringrequiredBase model whose tokenizer to use, e.g. Qwen/Qwen3-8B. Tokenizers differ between models, so
always tokenize with the model you will train or sample with.
textstringrequiredThe text to tokenize.
add_special_tokensbooleandefault: trueWhether to add the tokenizer's special tokens, such as beginning-of-sequence markers.
curl https://api.nucleus-ai.io/v1/tokenize \
-H "Authorization: Bearer $NUCLEUS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "Qwen/Qwen3-8B", "text": "What is the capital"}'{
"token_ids": [3838, 374, 279, 6722],
"count": 4,
"tokenizer": "qwen3"
}Detokenize token IDs
/v1/detokenizescope: readConverts token IDs back to text. Generate samples returns token IDs, so this is how you read your model's generations.
Body parameters
modelstringrequiredBase model whose tokenizer to use. Must match the model that produced the tokens.
token_idsarrayrequiredThe token IDs to decode, as an array of integers.
curl https://api.nucleus-ai.io/v1/detokenize \
-H "Authorization: Bearer $NUCLEUS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "Qwen/Qwen3-8B", "token_ids": [3838, 374, 279, 6722]}'{
"text": "What is the capital"
}Render a conversation
/v1/renderscope: readApplies the model's chat template to a messages array and returns token_ids plus
weights, a per-token loss mask aligned index-for-index with the tokens. A weight of 1
means the token contributes to the training loss; 0 means it is ignored. Feed both arrays
straight into a forward-backward pass.
Chat templates are easy to get subtly wrong by hand, and the usual symptom is a fine-tuned
model that will not stop talking. Rendering server-side guarantees the template matches the
one the model was trained with. It works for any model whose chat_template is true in
Retrieve a base model; for other models,
tokenize your data yourself with Tokenize text.
Body parameters
modelstringrequiredBase model whose chat template and tokenizer to use.
messagesarrayrequiredThe conversation to render, as an array of messages with role and content.
train_onstringWhich tokens carry loss weight 1: all (every token), assistant (all assistant turns) or
last-assistant (only the final assistant turn).
curl https://api.nucleus-ai.io/v1/render \
-H "Authorization: Bearer $NUCLEUS_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-8B",
"messages": [
{"role": "user", "content": "Where is my order?"},
{"role": "assistant", "content": "Let me check that for you."}
],
"train_on": "assistant"
}'{
"token_ids": [151644, 872, 198, ...],
"weights": [0, 0, 0, 1, 1, 1],
"count": 96
}