Q3.8FN - Swift1.5 vs NVIDIA NVFP4

  qwen3.8-flash-next-nvfp4:
    ttl: 10000
    healthCheckTimeout: 900
    env:
      - VLLM_PLE_CPU_OFFLOAD=1
      - MAX_JOBS=6
    cmd: >
      vllm serve /models/qwen3.8-flash-next/nvfp4
      --served-model-name qwen3.8-flash-next-nvfp4
      --port ${PORT}
      --quantization modelopt
      --trust-remote-code
      --max-model-len 262144
      --max-num-seqs 4
      --max-num-batched-tokens 8192
      --gpu-memory-utilization 0.95
      --enable-prefix-caching
      --async-scheduling
      --no-enable-flashinfer-autotune
      --reasoning-parser qwen3
      --enable-auto-tool-choice
      --tool-call-parser qwen3_coder
      --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

  swift1.5-flash-next-vllm:
    ttl: 600
    healthCheckTimeout: 1200
    env:
      - VLLM_PLE_CPU_OFFLOAD=1
      - MAX_JOBS=6
    cmd: >
      vllm serve /models/qwen3.8-flash-next/swift-1.5-fp8ple
      --served-model-name swift1.5-flash-next-vllm
      --port ${PORT}
      --max-model-len 262144
      --max-num-seqs 4
      --max-num-batched-tokens 8192
      --gpu-memory-utilization 0.95
      --enable-prefix-caching
      --reasoning-parser qwen3
      --enable-auto-tool-choice
      --tool-call-parser qwen3_xml
      --trust-remote-code
      --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Intro

In my last entry I found Ninfer6000 - ran with a Swift1.5-derived checkpoint - kept printing tool calls into the transcript instead of dispatching them. This happened during my in-harness benchmarks and repeatedly while I was trying to use it for real work. I ended that post unsure whether to blame the engine, the Ninfer conversion, or Swift1.5 itself.

So this time I took the engine out of the question to isolate the variable. Both models on vLLM. Same quantization recipe, same sampling parameters. The only differences were the recommended --tool-call-parser plus --quantization modelopt and --async-scheduling on NVIDIA.

• nvidia/Qwen3.8-Flash-Next-NVFP4 - NVIDIA’s official post-quantization of the stock Qwen3.8-Flash-Next.
• ukisai/Swift-1.5-Qwen3.8-Flash-Next-NVFP4 - UkisAI’s reasoning-efficiency derivative of the same base. The only weight change I made is the PLE embedding, quantized to FP8 to match.

The intent here is to determine if Swift1.5 is the cause of the tool calling issues or not. I predict it was Ninfer6000 or its conversion, not the Swift1.5 tune. It will also be a prime opportunity to compare these models and determine if Swift is worth running over NVIDIA’s as my daily driver.

As mentioned, Swift’s benefits lie in reasoning efficiency. UkisAI’s card claims the finetune uses 63.4% fewer thinking tokens(median) for a 1.8x wall-clock speedup at <1% accuracy loss versus the BF16 base.

Speed

Prefill/Decode Sweep

Same synthetic speed test as the last entry, single stream, prompt size from 1k to the edge of max ctx.

Prefill is the same. Averaged from 16k up, 15,742 vs 16,001 tok/s - 1.6% apart. Nothing to report.

Decode is a bit noisy. Swift sits above NVIDIA on the 16k-260k mean, 160.5 vs 147.3 tok/s (+9.0%), and on the median too (159.9 vs 144.5). Across all eight points the gap is +6.8%. NVIDIA swings up to 29 tok/s between adjacent prompt sizes and Swift up to 24. NVIDIA takes 4k and 32k, Swift takes everywhere else, by 32% at 64k. With swings that big inside each line, a single-digit gap between them doesn’t say much. I read the decode gap as noise. The results from the actual benchmarks would be more reliable for comparison.

In-Benchmark Speed

The sweep is synthetic and single-stream. The scored suites are the opposite: concurrency, reasoning traces, and a tool-calling parser.

Effective throughput. These sit far below the sweep’s ~150 tok/s because these runs are concurrent and the models are thinking, and Arena-Hard in particular is long-form reasoning at 64k budget.

Swift reads higher in all six, by 4.5% to 12%. That consistency without --async-scheduling is noteworthy - which if anything should have flattered NVIDIA’s side of this table.

One caveat. The single-stream sweep had these two within a few percent of each other, so this is a wider gap than raw speed explains. Concurrency, prefix caching and thinking traces all get mixed into a number like this, and a model that writes less gives the scheduler less to do. Swift got through the work faster.

Latency. Per-request wall time, which is what a person actually feels:

Swift also wins every row on latency, but latency is a function of how many tokens the model output. This is really the next section in disguise.

Output Token Side-by-side

This is where the two diverge. Same prompts, same suites - just count the output tokens:

Add the per-request times up and the 6 scored tests run for 32,907 request-seconds on NVIDIA and 20,421 for Swift. 38% less wall time for similar result. Arena-Hard dominates that total though. Take it out and it’s 14.4%.

The pattern is reasonable: the savings are smallest where there isn’t much to think about and biggest where the model is expected to do notable reasoning (Arena-Hard -40.6%).

Traditional Benchmarks

note these are not comparable to leaderboards - different sample size, different Arena Hard judge.

I ran the usual suspects hoping they’d tell me something. They didn’t. One pass through each at temp 1.0 / top_p 0.95:

Clearly a tie. Nonsignificant variance in accuracy across the lot. I checked these properly rather than eyeballing the deltas, and none of them are big enough to differentiate. With IFEval there was a directionally consistent pattern. Still probably noise but fewer tokens spent, more instructions obeyed could signal some value.

Arena-Hard

Details: 75 prompts judged, single pass, temp 1.0, max_tokens: 64000, judge deepseek/deepseek-v4-flash-0731.

The winrate is irrelevant, saturated and separated by nothing, but what it did produce is the cleanest reasoning-efficiency number in the entry.

These results are the strongest evidence I found for the Swift ideal of fewer tokens for the same grade. It carifies the observation of last entry’s Arena-Hard result, where Swift’s brevity looked like it might be buying it a scoring advantage. In reality when no cutoffs happen, brevity doesn’t impact the score.

Word500

Back to the puzzle benchmark - guess the five letter word given only the counts of letters in the right place, wrong place, or nowhere. This test specifically strains models for accuracy and reasoning efficiency. Unlike Arena-Hard, the token reduction did display an effect on how likely the model’s guess was to be correct.

Details: 6 boards x 10 samples x 2 models = 120 runs, temp 1.0 / top_p 0.95, max_tokens: 64000, concurrency 4, loop-abort at 10 consecutive repeats. Three boards (Smoke, Poker, Hoard) each run twice: once as a screenshot, once as text only.

Swift lost the hard board. Hoard with a screenshot went 8/10 to 3/10, and two of Swift’s three loop aborts happened there - 18k tokens a try on average, without converging. Strip the screenshot and it goes 10/10 to 7/10. So this isn’t only a vision problem; it’s the hardest board under both presentations.

It spent fewer tokens while doing worse. 14,746 average against 16,193 (which actually is only 8.9% less, far smaller gap than expected). While other tests showed the same direction with a bigger margin and no accuracy cost, Word500 showed 9% efficiency gain at the cost of 16.7 points.

The format rate slipped. 57 of 60 answers came back in the shape the harness asked for, against 60 of 60. Three malformed answers (loop aborts) over NVIDIA.

This test was what I was most looking forward to between the models as it’s the only one that really hits the intersection of what Swift is promising. It shows there is no such thing as a free lunch and Qwen’s debated excessive reasoning truly positively impacts outcomes. That isn’t to say the gap means Swift isn’t worth using, just that use case and accuracy needs on big reasoning tasks must be considered.

Unstructured / Personal Tests

Curated suites need some personal use cases to anchor them to reality. These are the sorts of in-harness tasks I typically utilize models for.

Open-WebUI Chat - Search, Reasoning, Function Calling

The same two-turn job from the prior entry:

1. Can you find a few recently released unified memory systems with 128gb+ ram and compare their prices/availability/pros and cons?
2. Can you format that information into a single page pdf so i can share it?

Neither model handled opening/closing tasks perfectly but that’s a common behavior. Both searched precisely and found all the devices I would expect. The prices were valid and the result was a single page I could send to communicate the state of the unified memory system market. The difference was the tokens taken to accomplish the task:

3.3x less. On the harness job, in the direction the finetune advertises.

Plex recommendation app

A big, messy, directionless 1-shot attempt at a sort of application I may employ the model to create:

Create a docker compose compatible web app that users can log into with a Plex account to browse popular shows/movies and be given show/movie recommendations based on their watch history and ratings.

These two went in genuinely different directions. NVIDIA built a FastAPI backend with a React frontend, docker-compose, and a test suite. Swift built a Fastify/TypeScript server with a vanilla JS frontend.

• Neither is a finished product.
• Both got meaningfully further than I expect from one prompt.
• One quickly decided on a more robust result than the other.

Initial attempt:

NVIDIA NVFP4	Swift1.5 NVFP4

After refinement prompts:

NVIDIA NVFP4	Swift1.5 NVFP4

After the refinement prompts both were then able to log in, but neither actually pulled watch history and ratings to generate recommendations. Both somewhat faked it at the depth I tested. Both need a lot more refinement but that would be exactly as I expect. A project like this for a model of this size would require a lot more research, guiding, and refining after the initial result. Between these two though I’d pursue Swift’s direction if I continued it. I liked that it considered TMDB and the visual experience of the app was in a better direction.

This test makes me think in real world applications, even over long context and deep reasoning, the token reduction is worth it. In the time NVIDIA’s finished, Swift’s could have finished a refinement pass.

Designing a server case

This is another real world test that mirrors a project I’ve been working on. I’ve designed and built a new case for my LLM rig. Could either of these models 1-shot a 3D representation of the efficient build I brainstormed over weeks? Of course not, but let’s see what they output.

Let’s design a case for a server that supports standard ATX motherboard and power supply. It should support an RTX 6000 Blackwell and an Evga 3090 FTW3.

The primary focus should be keeping the case compact and maximizing airflow.

Your output should be a 3D representation of the case for easy viewing.

Both took a long time and neither result is “good”. This test is clearly beyond what this model class can do anywhere near 1-shot with no provided structure or ideas. But both actually produced a manipulable 3D model, which is the part I’d written off. Hermes seems to obfuscate its token counts, so I can’t put a number on the gap here the way I could on the PDF job.

Both viewers below are the real artifacts, drag to orbit and scroll to zoom. They’re three.js, loaded from a CDN, in a self-contained page. Expand to load them.

Static renders:

NVIDIA - SLIPSTREAM 26	Swift1.5 - STRIDE-2

📐 NVIDIA build's own QA panels

Both failed the engineering. NVIDIA produced a plausible chassis and delivered it unreadably; Swift produced something plausible but less creative and delivered it more clearly but still imperfect. Swift seemed to output two different designs in static and interactive, but its interactive was at least intelligible. No clear winner here, but a good test nonetheless.

Conclusion