Small LLM Finale - Q8_0

  Spark-X2.5-1.7B-Q8:
    ttl: 300
    healthCheckTimeout: 600
    cmd: >
      /usr/local/bin/llama-server
      --port ${PORT}
      --model /models/Spark-X2.5/1.7B/Spark-X2.5-1.7B-Q8_0.gguf
      --ctx-size 131072
      -ngl 99
      --main-gpu 0
      --split-mode none
      --fit off
      --flash-attn on
      --temp 1.0 --top-p 0.95 --top-k 0
      --jinja
      --parallel 1
      --batch-size 4096
      --ubatch-size 2048

  Spark-X2.5-4B-Q8:
    ttl: 300
    healthCheckTimeout: 600
    cmd: >
      /usr/local/bin/llama-server
      --port ${PORT}
      --model /models/Spark-X2.5/4B/Spark-X2.5-4B-Q8_0.gguf
      --ctx-size 131072
      -ngl 99
      --main-gpu 0
      --split-mode none
      --fit off
      --flash-attn on
      --temp 1.0 --top-p 0.95 --top-k 0
      --jinja
      --parallel 1
      --batch-size 4096
      --ubatch-size 2048

  Ling-3.0-tiny-Q8:
    ttl: 300
    cmd: >
      /usr/local/bin/llama-server
      --port ${PORT}
      --model /models/Ling-3.0-tiny/Ling-3.0-tiny-Q8_0.gguf
      --ctx-size 131072
      -ngl 99
      --main-gpu 0
      --split-mode none
      --fit off
      --flash-attn on
      --temp 1.0 --top-p 0.95 --top-k 20
      --jinja
      --parallel 1
      --batch-size 4096
      --ubatch-size 2048

  Nanbeige4.2-3B-Q8:
    ttl: 300
    healthCheckTimeout: 600
    cmd: >
      /usr/local/bin/llama-server
      --port ${PORT}
      --model /models/nanbeige4.2-3B/Nanbeige_Nanbeige4.2-3B-Q8_0.gguf
      --ctx-size 131072
      -ngl 99
      --main-gpu 0
      --split-mode none
      --fit off
      --flash-attn on
      --temp 0.6 --top-p 0.95 --top-k 20
      --jinja
      --parallel 1
      --batch-size 4096
      --ubatch-size 2048

Intro

This is a follow-up to my previous entry regarding small LLMs. I’m looking for the best current option for a small model that could run fast in a secondary gpu for light tasks like title generation, media recs, simple documents, and calorie tracking. Tool calling and general behavior in my harness is the most important feature. Before I didn’t worry about quantization, I compared the models in BF16. This time I grabbed the Q8_0 quants to see how they compare in that condition and select the best option. The three primary contenders here are

• Nanbeige4.2-3B-Q8 (the current leader)
• Spark-X2.5-4B-Q8
• Ling-3.0-tiny-Q8

I’ve also included Spark-X2.5-1.7B-Q8 as the previous benchmarks seemed to display a much larger gap than expected between it and the other models. I wanted to give it another shake as some bonus content, maybe my parameters were off when it was run in vLLM or something.

Speed & Size

Once again model speed isn’t super important at this size, but I decided it’s an easy start. I had some anomalies in my speed results last time, so I went ahead and manually set a seed for each run and ensured a manual warmup before the measurements were taken. These seemed to normalize the results from before so now we have a more conclusive table.

vllm bench serve \
  --backend openai \
  --base-url http://127.0.0.1:8500 \
  --endpoint /v1/completions \
  --model {model} \
  --tokenizer /models/{model}/{tokenizer} \
  --trust-remote-code \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 128 \
  --random-prefix-len 0 \
  --num-prompts 20 \
  --max-concurrency 1 \
  --ignore-eos \
  --percentile-metrics ttft,tpot,itl \
  --seed {seed}

It was cool to compare the prefill on vLLM with the SparkX2.5 models and see the notable performance gap in measurements. Nothing surprising there, just neat to see. Nanbeige4.2 really falls behind in both prefill and decode, which makes sense since it does more than one pass over its 22 layers. It’s more akin to a model twice its size. It is also HUGE in VRAM with any reasonable amount of context as the KV/state cost grows linearly per loop. Somewhat nullifies it in the small model category. To set it on-par with the others with less dense attention you’d have to drop it to around 32k ctx, a pretty far fall from the others at 131k.

Traditional Benchmarks

note these are not comparable to leaderboards - different sample size, different Arena Hard judge.

BFCL v4 - Tool Calling

Details: 50 samples, 6 test subsets → 300 samples total

IFEval - Instruction-Following

Details: 250 samples

MMLU-Pro - Reasoning

Details: 30 samples, 14 categories → 420 samples total

Arena-Hard - Reasoning

Details: 75 samples, run with --repeat-penalty 1.1 --min-p 0.05, judged by DeepSeek V4 Flash 0731

HumanEval+ & MBPP+ - Coding

Details: One pass each

• HumanEval+: 164 problems
• MBPP+: 378 problems

Quantization Comparison

Since my previous entry contained the same benchmarks against the BF16 weights for these models, I figured its worth putting the results side by side. The faded hatched bars represent the BF16 results and the solid bars are the results from the Q8 runs here.

A genuinely clearer and more honest presentation is simply the gap between the two overlayed onto the 95% confidence interval.

This very clearly displays none of the differences exceed error margin. Dropping to Q8 really does lose nothing visible in benchmarks. The expected result, but good to see clearly first hand. The more important focus is how each behaves in real world, in-harness use.

Unstructured / Personal Tests

Open-WebUI Chat - Search, Reasoning, Function Calling

I primarily use Open-WebUI for my LLM chat needs. This test is set up to see how well the model fares in my harness. Can it obey the system prompt and follow the user prompt? Can it properly assess available tools, format the calls, and handle errors?

1. Can you find a few recently released unified memory systems with 128gb+ ram and compare their prices/availability/pros and cons?
2. Can you format that information into a single page pdf so i can share it?

I’m expecting to see

• information about Nvidia RTX/DGX spark, existing and upcoming M# Ultra Mac Studios, and some of the various Ryzen AI Max 395 mini pcs.
• MSRP, but would love to see a model identify how much prices have increased.
• pros and cons discussing availability, memory size, memory bandwidth, and compute.
• these models to be able to find and call the PDF creation tools present in the harness easily.

Results

I was really surprised to see that none of the models adhered to the single page instruction. Before, Nanbeige4.2 had clutched it but it fell through this time. Regardless, Nanbeige4.2 still followed more of the instructions than the other models. It obeyed the system prompt and determined the best N value for search query result count. The others simply defaulted to 5 or 10 per search.

I went ahead and messed around a bit with the specific wording of my prompt. I was curious if “single page pdf” comes across as “single pdf”. I tried “single-page pdf” and saw the same multi-page productions, but then I tried “1-page pdf” and started seeing adherence. Each model generated a one pager when provided that revised prompt. Goes to show that small LLMs have not yet reached the language comprehension bigger ones have (and they’re certainly nowhere near the “read your mind” style of frontier).

Pelican on a Bicycle

You already know whats goin on here, benchmarks but silly.

Generate an SVG of a pelican riding a bicycle

Spark-X2.5-1.7B-Q8 1.7B SVG Nanbeige4.2-3B-Q8 3B SVG Spark-X2.5-4B-Q8 4B SVG Ling-3.0-tiny-Q8 7.9B MoE, 1.3B active SVG

Spark-X2.5-1.7B-Q8 (1.7B)

Open SVG ↗

Generated the SVG successfully.

Just like last time, no genuinely GOOD results, but I would be pretty confident in saying that Nanbeige generated the best of the group. There is an identifiable bird with a large beak, two spoked wheels, and some kinda frame. I’d give Ling3.0 Tiny the runner up with its bird that looks like its currently mid-crash. This test clearly is not great for this tier of model, but lets be real its not great for any tier of model. Its just fun.

Conclusion