Checkpoint inference requires the model snapshot hosted at ByteDance-Seed/Cola2. CUDA with bfloat16 is recommended. CPU execution is useful for the tiny test configuration but is not a practical target for the released checkpoint.
import torch
from transformers import AutoModel, AutoTokenizer
model_path = "/path/to/cola2"
tokenizer = AutoTokenizer.from_pretrained(
model_path,
trust_remote_code=True,
)
model = AutoModel.from_pretrained(
model_path,
trust_remote_code=True,
torch_dtype=torch.bfloat16,
low_cpu_mem_usage=True,
).to("cuda").eval()trust_remote_code=True executes Python shipped with the checkpoint. Review
that code first when loading an untrusted or mirrored snapshot. Installing this
repository registers the same classes locally.
The checkpoint stores the paper setting in generation_config.json:
{
"do_sample": false,
"max_new_tokens": 32,
"dit_num_steps": 8,
"dit_timestep_schedule": "low_noise",
"dit_guidance_scale": 10.0,
"decoder_guidance_scale": -0.9,
"span_causal_chunk_size": 1,
"dynamic_duration": true,
"latent_feedback": true
}The low-noise schedule uses (t_i=T(1-i/8)^2), placing more Euler nodes near
zero noise. Calling model.generate(**inputs) uses the complete checkpoint
preset.
do_sample=false controls token selection only. Each latent block still starts
from Gaussian noise, so calls are stochastic unless a generator is supplied.
prompt = "Question: What is the capital of France? Answer:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
generator = torch.Generator(device=model.device).manual_seed(42)
with torch.inference_mode():
output = model.generate(**inputs, generator=generator)Create a new generator with the same seed before each call when exact replay is required. Reusing one generator advances its state and intentionally produces a different latent sample.
Exact equality across different devices, PyTorch versions, or kernels is not guaranteed. The benchmark derives a stable seed per record so resume and sharding do not change an already defined public run.
For each block, Cola2:
- encodes the prompt and committed context;
- denoises eight latent positions with the block-causal DiT;
- predicts a dynamic token duration for every latent;
- decodes one complete latent span per decoder chunk;
- commits the selected tokens; and
- re-encodes the completed spans as latent feedback.
Decoder CFG -0.9 combines conditional and unconditional decoder logits. The negative value preserves useful latent semantics while allowing the causal decoder to supply token-level information not represented by the compressed latent.
Generation fields can be supplied directly or through a copied
Cola2GenerationConfig:
output = model.generate(
**inputs,
max_new_tokens=48,
dit_num_steps=8,
dit_timestep_schedule="low_noise",
dit_guidance_scale=10.0,
decoder_guidance_scale=-0.9,
span_causal_chunk_size=1,
dynamic_duration=True,
latent_feedback=True,
repetition_penalty=1.0,
do_sample=False,
generator=torch.Generator(device=model.device).manual_seed(42),
)Supported DiT schedules are uniform, high_noise, low_noise, and power.
The power schedule also requires a positive dit_timestep_power.
temperature, top_k, and top_p affect token decoding only when
do_sample=True.
The release supports one return sequence and no beam search. Beam search is not defined for the block-parallel span-causal generation path.
Install the package and run:
cola2-chat \
--model /path/to/cola2 \
--device cuda \
--dtype bfloat16 \
--seed 42The equivalent repository helper is:
COLA2_MODEL_PATH=/path/to/cola2 bash scripts/run_chat.shUseful options include --prompt for one-shot generation,
--max-new-tokens, --max-context-tokens, --dit-steps,
--dit-timestep-schedule, --dit-cfg, --decoder-cfg,
--decoder-chunk-spans, and --repetition-penalty.
The interactive wrapper formats local history as User: / Assistant: text.
This is only a convenience interface: Cola2 is a base model and has not been
instruction-tuned as a chat assistant.
- Missing model: set
--modelorCOLA2_MODEL_PATH. - Out of memory: shorten the prompt or output, close other GPU processes, or choose a supported lower-precision dtype. Do not silently change model architecture fields.
- Different output with the same prompt: pass a seeded generator and check device and package versions.
- Unused generation argument: use the canonical names in Configuration; unknown fields fail instead of being ignored.
- Remote-code error: install this repository or ensure the snapshot includes the advertised remote-code files.