BF16 memory at 2K

Model tensors plus live state

MiB

BananaMind-2-Pro325.1
GPT-X2.5-135M302.6
SmolLM2-135M301.6
Speck1-140M281.3
Supra2-100M-Base216.0

Runtime state at 2K

Lower is better

MiB

BananaMind-2-Pro60.0
SmolLM2-135M45.0
GPT-X2.5-135M45.0
Supra2-100M-Base24.0
Speck1-140M12.0

BF16 model tensors plus cache or state after a 2,048-token prefill at batch 1. Framework and temporary workspace memory are excluded.