Small LLM Shootout

  Spark-X2.5-1.7B:
    ttl: 3000
    healthCheckTimeout: 600
    concurrencyLimit: 16
    env:
      - CUDA_VISIBLE_DEVICES=0
    cmd: >
      vllm serve /models/Spark-X2.5/1.7B
      --chat-template /models/Spark-X2.5/1.7B/chat_template.jinja
      --served-model-name Spark-X2.5-1.7B
      --port ${PORT}
      --max-model-len 128000
      --max-num-seqs 16
      --gpu-memory-utilization 0.6
      --enable-prefix-caching
      --trust-remote-code
      --enable-auto-tool-choice
      --tool-call-parser spark25
      --reasoning-parser qwen3

  LFM2.5-2.6B:
    ttl: 300
    cmd: >
      /usr/local/bin/llama-server
      --port ${PORT}
      --model /models/LFM2.5-2.6B/LFM2.5-2.6B-BF16.gguf
      --ctx-size 131072
      -ngl 99
      --main-gpu 0
      --split-mode none
      --fit off
      --flash-attn on
      --temp 0.2 --top-k 80 --repeat-penalty 1.05
      --jinja
      --parallel 1
      --batch-size 4096
      --ubatch-size 2048

  Spark-X2.5-4B:
    ttl: 3000
    healthCheckTimeout: 600
    concurrencyLimit: 16
    env:
      - CUDA_VISIBLE_DEVICES=0
    cmd: >
      vllm serve /models/Spark-X2.5/4B
      --chat-template /models/Spark-X2.5/4B/chat_template.jinja
      --served-model-name Spark-X2.5-4B
      --port ${PORT}
      --max-model-len 128000
      --max-num-seqs 16
      --gpu-memory-utilization 0.6
      --enable-prefix-caching
      --trust-remote-code
      --enable-auto-tool-choice
      --tool-call-parser spark25
      --reasoning-parser qwen3

  MiniCPM5-2B:
    ttl: 300
    cmd: >
      /usr/local/bin/llama-server
      --port ${PORT}
      --model /models/MiniCPM5-2B/MiniCPM5-2B-BF16.gguf
      --ctx-size 131072
      -ngl 99
      --main-gpu 0
      --split-mode none
      --fit off
      --flash-attn on
      --temp 0.9 --top-p 0.95
      --jinja
      --parallel 1
      --batch-size 4096
      --ubatch-size 2048

  Ling-3.0-tiny:
    ttl: 300
    cmd: >
      /usr/local/bin/llama-server
      --port ${PORT}
      --model /models/ling3.0-tiny/Ling-3.0-tiny-bf16.gguf
      --ctx-size 262144
      -ngl 99
      --main-gpu 0
      --split-mode none
      --fit off
      --flash-attn on
      --temp 1.0 --top-p 0.95 --top-k 20
      --jinja
      --parallel 1
      --batch-size 4096
      --ubatch-size 2048

  Nanbeige4.2-3B:
    ttl: 3000
    healthCheckTimeout: 600
    concurrencyLimit: 4
    env:
      - CUDA_VISIBLE_DEVICES=0
    cmd: >
      /usr/local/bin/llama-server
      --model /models/nanbeige4.2-3B/Nanbeige_Nanbeige4.2-3B-bf16.gguf
      --port ${PORT}
      -ngl 99
      --fit off
      --main-gpu 0
      --split-mode none
      --ctx-size 262144
      --parallel 1
      --flash-attn on
      --cache-reuse 256
      --jinja
      --temp 0.6 --top-p 0.95 --top-k 20
      --batch-size 4096
      --ubatch-size 2048

  LFM2.5-8B-A1B:
    ttl: 300
    cmd: >
      /usr/local/bin/llama-server
      --port ${PORT}
      --model /models/LFM2.5-8B-A1B/LFM2.5-8B-A1B-BF16.gguf
      --ctx-size 131072
      -ngl 99
      --main-gpu 0
      --split-mode none
      --fit off
      --flash-attn on
      --temp 0.2 --top-k 80 --repeat-penalty 1.05
      --jinja
      --parallel 1
      --batch-size 4096
      --ubatch-size 2048

  Qwen3.8-9B-Distill:
    ttl: 300
    cmd: >
      /usr/local/bin/llama-server
      --port ${PORT}
      --model /models/qwen3.8-9B-Distill/Qwen3.8-9B-BF16.gguf
      --ctx-size 131072
      -ngl 99
      --main-gpu 0
      --split-mode none
      --fit off
      --flash-attn on
      --temp 0.6 --top-p 0.95 --top-k 20
      --jinja
      --parallel 1
      --batch-size 4096
      --ubatch-size 2048

Intro

I’ve recently started looking into the best options for a simple task model I could run on my secondary GPU (along with an embedding model and some TTS). I basically hadn’t considered models smaller than 27b since Gemma4-12b dropped. I found it lackluster in tool calling and sys prompt adherence in my harness and ended up scrapping it. I heard good things about Spark X2.5 4B, so I figured I should check in on the state of small models. Maybe some of the recently released models could suffice for title generation, media recommendations, document editing, calorie tracking, light chat, and any other tiny fast tasks.

The models I found with some good anecdotal buzz were Spark-X2.5-1.7B, MiniCPM5-2B, LFM2.5-2.6B, Nanbeige4.2-3B, Spark-X2.5-4B, Ling-3.0-tiny, and LFM2.5-8B-A1B. I also added the quirky Qwen3.8-9B that is actually Qwen3.5-9B distilled from Qwen3.8 2.4T A95B. It is a lot bigger than the others but seemed like a novel, worthwhile addition. I pulled BF16 GGUFs for each to load in Llama.cpp for some benchmarks. The only exception was Spark-X2.5, that wasn’t in Llama.cpp main at the time of benchmarking so I ran that in vLLM. I didn’t bother with quantization on the models or KV cache. They were all small enough to fit in VRAM and I figured I would isolate my top 3 before determining the best quant.

Speed

Model speed isn’t super important at this size. Every model will prefill faster than any task requires, and decode speed is in 100-300tok/s range. Regardless I figured I should take a look:

vllm bench serve \
  --backend openai \
  --base-url http://127.0.0.1:8500 \
  --endpoint /v1/completions \
  --model {model} \
  --tokenizer {tokenizer} \
  --trust-remote-code \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 128 \
  --random-prefix-len 0 \
  --num-prompts 20 \
  --max-concurrency 1 \
  --ignore-eos \
  --percentile-metrics ttft,tpot,itl

The Spark-X2.5 models once again were on vLLM, so their numbers arent perfectly comparable to the others. Taken with a grain of salt, speed standouts with the upcoming results in consideration were Spark-X2.5-4B with its great prefill and Ling-3.0-tiny with its great decode.

Traditional Benchmarks

note these are not comparable to leaderboards - different sample size, different Arena Hard judge.

BFCL v4 - Tool Calling

Details: 50 samples, 6 test subsets → 300 samples total

IFEval - Instruction-Following

Details: 250 samples

MMLU-Pro - Reasoning

Details: 30 samples, 14 categories → 420 samples total

Arena-Hard - Reasoning

Details: 75 samples, run with --repeat-penalty 1.1 --min-p 0.05, judged by Deepseek V4 Flash 0731

HumanEval+ & MBPP+ - Coding

Details: One pass each

• HumanEval+: 164 problems
• MBPP+: 378 problems

Unstructured / Personal Tests

Open-WebUI Chat - Search, Reasoning, Function Calling

I primarily use Open-Webui for my LLM chat needs. This test is set up to see how well the model fares in my harness. Can it obey the system prompt and follow the user prompt? Can it properly assess available tools, format the calls, and handle errors?

1. Can you find a few recently released unified memory systems with 128gb+ ram and compare their prices/availability/pros and cons?
2. Can you format that information into a single page pdf so i can share it?

I’m expecting to see information about Nvidia RTX/DGX spark, existing and upcoming M# Ultra Mac Studios, and some of the various Ryzen AI Max 395 mini pcs. I’m expecting to see MSRP, but would love to see a model identify how much prices have increased. I’m also expecting to see pros and cons discussing availability, memory size, memory bandwidth, and compute. Lastly I’m expecting these models to be able to find and call the PDF creation tools present in the harness easily.

Results

Only Spark-X2.5-4B & Nanbeige4.2-3B respected my sys prompt’s instructions to specify how many search results to request (N=1-3). Ling has the objectively best result I think, but the sacrifice is hitting ~80 sources for what is a really simple question. The models I’m most impressed with here are Spark-X2.5-4B, Nanbeige4.2-3B, and Ling-3.0-tiny. Nanbeige being the only model that TECHNICALLY followed ALL provided instructions in sys prompt and user prompt. This kinda lines up with the benchmark results, Nanbeige really eatin.

Pelican on a Bicycle

Let’s yet again end on the classic silly “pelican on a bicycle” prompt.

Generate an SVG of a pelican riding a bicycle

Spark-X2.5-1.7B 1.7B SVG MiniCPM5-2B 2B SVG LFM2.5-2.6B 2.6B SVG Nanbeige4.2-3B 3B SVG Spark-X2.5-4B 4B SVG Ling-3.0-tiny 7.9B MoE, 1.3B active SVG LFM2.5-8B-A1B 8B MoE, 1B active Qwen3.8-9B-Distill 9B SVG

Spark-X2.5-1.7B (1.7B)

Open SVG ↗

Successful file generation.

Obviously none of these are ‘good’ but at least LFM2.5-2.6B, Nanbeige4.2-3B, Ling-3.0-tiny, and Qwen3.8-9B-Distill generated a form that can extremely abstractly be determined to contain a bird and some form of 2 wheeled vehicle. I would personally make the claim the Qwen3.8-9B-Distill’s was the best. The bird is most identifiably a pelican with the thick bill and the bicycle is pretty interpretable. Runner up would be Nanbeige4.2-3B. The spokes were impressive, but the bird is jumbled and the single triangle bike frame is hilarious. I’ll let you draw your own conclusions though.