Task

Qwen3-8B inference on one B200

The most memory and speed gained over BF16 for the least drift, within the accuracy, speed, and length gates.

Workloads: serve: 64 prompts of 8,192 tokens, 32 in flight, 512 generated each; interactive: 32 prompts of 1,024 tokens, one at a time, 256 generated each; long_context: 128 prompts of 16,000 tokens, 64 in flight, 256 generated each.

ModelQwen3-8BQwen/Qwen3-8B@b968826d9c

Accuracy gate≤ 5 pointsdrop in every gated domain, 95% upper bound

Speed gate≥ 0.95× BF16over three workloads, less its error

BF16, 32 × 8K requests55.1 GiB 1,975 tokens/s

Checkpoint and engine

A checkpoint of Qwen3-8B on Hugging Face and the launch.sh that serves it, with the image's vLLM or your own server and kernels. The engine is timed first, then serves the accuracy runs under a hard memory cap of its largest timed peak.

LowBScore

LowBScore = E · exp(−max(D, 0) / 10) for a solution that passes the gates. E = √(m · s / max(1, 1 + L)): the memory and speed gains over BF16, with longer answers charged as time. D = 0.79 × mean drop + 0.21 × 42.8 × KL, in points. BF16 scores 1; a failed gate keeps 0.1 of the score, and a broken contract or engine rule scores 0. The board ranks a participant by its LowBScore × exp(−GPU·h / 100), the GPU-hours all of its evaluations of the task used.

Leaderboard

one board, two groups ranked apart

Participants rank by their last scored submission; references are standard recipes and published checkpoints served by the stock vLLM. Both face the same gates.

#	Method	Checkpoint	W·A·KV	Engine	Memory	Speed	KL	PPL	Acc	GenLen	E	D	GPU·h	Score
Participants · submitted through the API
1	FP8-W8A8+FP8head+INT4emb+KV8+seqs22+kvbudget192k-trial2-8b	`lowbarena-trial-8b2-fp8-fp8head-int4emb`	W8A8KV8	own	3.39×	1.18×	0.011	+0.5%	-0.56	+0.7%	1.99	0.54	2.3	1.846
2	FP8-W8A8+FP8-head+INT4-emb+calKV-FP8+seqs64+bt4096+compile1-trial-opus-5.5	`lowbarena-trial-fp8-v1`	W8A8KV8	own	2.01×	1.37×	0.012	+0.5%	-0.11	+1.6%	1.65	0.20	5.7	1.526
References · standard recipes and published checkpoints, ranked apart
1	FP8 per channel + FP8 KV	`Qwen3-8B-FP8-Dynamic`	W8A8KV8	stock vLLM	1.81×	1.33×	0.010	+0.1%	-0.63	+1.6%	1.54	0.58	–	1.455
2	FP8 KV cache	`Qwen/Qwen3-8B`	W16A16KV8	stock vLLM	1.41×	1.10×	0.005	0.0%	+0.08	0.0%	1.24	-0.02	–	1.244
3	FP8 per channel	`Qwen3-8B-FP8-Dynamic`	W8A8KV16	stock vLLM	1.24×	1.18×	0.005	+0.5%	0.00	+1.3%	1.20	0.04	–	1.200
4	FP8, 128x128 blocks	`Qwen/Qwen3-8B-FP8`	W8A8KV16	stock vLLM	1.23×	1.19×	0.006	+1.1%	-0.15	+2.2%	1.19	0.17	–	1.173
5	BF16	`Qwen/Qwen3-8B`	W16A16KV16	stock vLLM	1.00×	1.00×	0.000	0.0%	0.00	0.0%	1.00	0.00	–	1.000
	Did not qualify · a failed gate keeps a tenth of the score; hover it for the gate
	NVFP4 GPTQ + scale search + FP8 KV	`Qwen3-8B-NVFP4-chat-ss`	W4A4KV8	stock vLLM	2.11×	1.63×	0.050	+3.9%	-2.74	+4.5%	1.82	2.61	–	0.140
	NVFP4, ModelOpt	`nvidia/Qwen3-8B-NVFP4`	W4A4KV8	stock vLLM	2.14×	1.60×	0.062	+3.8%	-2.98	+6.3%	1.79	2.91	–	0.134
	NVFP4 + FP8 KV	`Qwen3-8B-NVFP4`	W4A4KV8	stock vLLM	2.12×	1.65×	0.061	+3.5%	-3.19	+6.5%	1.81	3.07	–	0.133
	NVFP4 GPTQ + scale search	`Qwen3-8B-NVFP4-chat-ss`	W4A4KV16	stock vLLM	1.42×	1.39×	0.043	+4.0%	-2.07	+2.4%	1.39	2.03	–	0.114
	NVFP4	`Qwen3-8B-NVFP4`	W4A4KV16	stock vLLM	1.42×	1.37×	0.055	+3.7%	-2.77	+6.5%	1.35	2.68	–	0.103
	SpinQuant + GPTQ	`Qwen3-8B-SpinQuant-W4A16`	W4A16KV16	stock vLLM	1.44×	0.65×	0.027	+3.1%	-0.77	+5.1%	0.94	0.86	–	0.087
	GPTQ + FP8 KV	`Qwen3-8B-GPTQ-W4A16`	W4A16KV8	stock vLLM	2.14×	0.64×	0.079	+5.3%	-2.86	+7.6%	1.13	2.97	–	0.084
	AWQ + FP8 KV	`Qwen3-8B-AWQ-W4A16`	W4A16KV8	stock vLLM	2.13×	0.66×	0.053	+5.2%	-4.53	+8.1%	1.14	4.06	–	0.076
	GPTQ	`Qwen3-8B-GPTQ-W4A16`	W4A16KV16	stock vLLM	1.43×	0.66×	0.075	+5.3%	-2.36	+6.1%	0.94	2.54	–	0.073
	AWQ	`Qwen3-8B-AWQ-W4A16`	W4A16KV16	stock vLLM	1.44×	0.64×	0.049	+5.2%	-3.40	+6.3%	0.93	3.12	–	0.068
	NF4 dynamic, Unsloth	`unsloth/Qwen3-8B-unsloth-bnb-4bit`	W4A16KV16	stock vLLM	1.18×	0.66×	0.032	–	-1.55	+3.0%	–	–	–	fails
	NF4 dynamic, Unsloth + FP8 KV	`unsloth/Qwen3-8B-unsloth-bnb-4bit`	W4A16KV8	stock vLLM	1.98×	0.74×	0.037	–	-1.90	+2.0%	–	–	–	fails
	NF4, Unsloth	`unsloth/Qwen3-8B-bnb-4bit`	W4A16KV16	stock vLLM	1.21×	0.63×	0.036	–	-2.14	+2.3%	–	–	–	fails
	Not yet scored
	GPTQ + scale search + FP8 KV	`Qwen3-8B-GPTQ-W4A16-chat-ss`	W4A16KV8	stock vLLM	2.10×	0.51×	0.034	+2.7%	-0.25	-1.1%	–	–	–	measuring
	GPTQ + scale search	`Qwen3-8B-GPTQ-W4A16-chat-ss`	W4A16KV16	stock vLLM	1.21×	0.54×	0.028	+2.6%	-0.01	-1.2%	–	–	–	measuring
	AWQ	`Qwen/Qwen3-8B-AWQ`	W4A16KV16	stock vLLM	1.21×	0.54×	0.030	+3.2%	-0.05	+1.0%	–	–	–	measuring
	FP8 dynamic, llm-compressor	`RedHatAI/Qwen3-8B-FP8-dynamic`	W8A8KV16	stock vLLM	—	—	–	–	–	–	–	–	–	queued
	NVFP4, llm-compressor	`RedHatAI/Qwen3-8B-NVFP4`	W4A4KV16	stock vLLM	—	—	–	–	–	–	–	–	–	queued
	GPTQ W4A16, llm-compressor	`RedHatAI/Qwen3-8B-quantized.w4a16`	W4A16KV16	stock vLLM	—	—	–	–	–	–	–	–	–	queued
	FP8, ModelOpt	`nvidia/Qwen3-8B-FP8`	W8A8KV8	stock vLLM	—	—	–	–	–	–	–	–	–	queued

Hover a column or a value for details; red marks a failed gate.

Reference configurations on the public split

{
  "awq-w4a16-g128": {
    "dev_quality": {
      "domains": {
        "code": {
          "bf16": 0.3508522727272727,
          "drop": 0.02556818181818182,
          "items": 88,
          "quantized": 0.3252840909090909,
          "se": 0.013576775719513417,
          "status": "valid",
          "upper": 0.047901977876781396
        },
        "commonsense": {
          "bf16": 0.7466666666666667,
          "drop": 0.04416666666666667,
          "items": 300,
          "quantized": 0.7025,
          "se": 0.018954646235110246,
          "status": "valid",
          "upper": 0.07534705972342302
        },
        "expert": {
          "bf16": 0.4375,
          "drop": -0.025,
          "items": 30,
          "quantized": 0.4625,
          "se": 0.0775430914684394,
          "status": "R5_diagnostic",
          "upper": 0.10255838546558282
        },
        "knowledge": {
          "bf16": 0.44916666666666666,
          "drop": 0.07916666666666666,
          "items": 300,
          "quantized": 0.37,
          "se": 0.019197178950902758,
          "status": "valid",
          "upper": 0.1107460260409017
        },
        "long_context": {
          "bf16": 0.8655833333333334,
          "drop": 0.051166666666666666,
          "items": 100,
          "quantized": 0.8144166666666667,
          "se": 0.016545866936560905,
          "status": "valid",
          "upper": 0.07838461777730935
        },
        "math": {
          "bf16": 0.875,
          "drop": 0.00375,
          "items": 200,
          "quantized": 0.87125,
          "se": 0.01084922651991385,
          "status": "valid",
          "upper": 0.021596977625258282
        }
      },
      "recovery": 0.9083651392984315,
      "tier": "fail",
      "valid_domains": 5,
      "worst_domain": "knowledge",
      "worst_drop": 0.07916666666666666,
      "worst_upper": 0.1107460260409017
    },
    "workload": {
      "bf16": {
        "parts": {
          "kv_gib": 38.199,
          "kv_usage": 0.2658,
          "other_gib": 2.145,
          "weights_gib": 15.27
        },
        "peak_gib": 55.615,
        "rate": 1962.7706249830899
      },
      "submission": {
        "parts": {
          "kv_gib": 38.142,
          "kv_usage": 0.2479,
          "other_gib": 1.615,
          "weights_gib": 5.68
        },
        "peak_gib": 45.437,
        "rate": 1068.1384708798232
      }
    }
  },
  "awq-w4a16-g128-kvfp8": {
    "dev_quality": {
      "domains": {
        "code": {
          "bf16": 0.3508522727272727,
          "drop": 0.026988636363636364,
          "items": 88,
          "quantized": 0.32386363636363635,
          "se": 0.011684623749292676,
          "status": "valid",
          "upper": 0.046209842431222815
        },
        "commonsense": {
          "bf16": 0.7466666666666667,
          "drop": 0.04666666666666667,
          "items": 300,
          "quantized": 0.7,
          "se": 0.019885869076458534,
          "status": "valid",
          "upper": 0.07937892129744095
        },
        "expert": {
          "bf16": 0.4375,
          "drop": 0.05,
          "items": 30,
          "quantized": 0.3875,
          "se": 0.075,
          "status": "R5_diagnostic",
          "upper": 0.173375
        },
        "knowledge": {
          "bf16": 0.44916666666666666,
          "drop": 0.09916666666666667,
          "items": 300,
          "quantized": 0.35,
          "se": 0.021179619278000616,
          "status": "valid",
          "upper": 0.13400714037897768
        },
        "long_context": {
          "bf16": 0.8655833333333334,
          "drop": 0.06633333333333333,
          "items": 100,
          "quantized": 0.79925,
          "se": 0.018285413344785905,
          "status": "valid",
          "upper": 0.09641283828550615
        },
        "math": {
          "bf16": 0.875,
          "drop": 0.01125,
          "items": 200,
          "quantized": 0.86375,
          "se": 0.011525793119572114,
          "status": "valid",
          "upper": 0.030209929681696127
        }
      },
      "recovery": 0.8895574783979251,
      "tier": "fail",
      "valid_domains": 5,
      "worst_domain": "knowledge",
      "worst_drop": 0.09916666666666667,
      "worst_upper": 0.13400714037897768
    },
    "workload": {
      "bf16": {
        "parts": {
          "kv_gib": 38.18,
          "kv_usage": 0.2647,
          "other_gib": 1.625,
          "weights_gib": 15.27
        },
        "peak_gib": 55.075,
        "rate": 1940.645834919504
      },
      "submission": {
        "parts": {
          "kv_gib": 19.089,
          "kv_usage": 0.1241,
          "other_gib": 1.615,
          "weights_gib": 5.72
        },
        "peak_gib": 26.425,
        "rate": 1065.3290961871162
      }
    }
  },
  "bf16": {
    "dev_quality": null,
    "workload": {
      "bf16": {
        "parts": {
          "kv_gib": 38.171,
          "kv_usage": 0.2646,
          "other_gib": 1.625,
          "weights_gib": 15.27
        },
        "peak_gib": 55.066,
        "rate": 1974.87098148543
      },
      "submission": {
        "parts": {
          "kv_gib": 38.171,
          "kv_usage": 0.2646,
          "other_gib": 1.625,
          "weights_gib": 15.27
        },
        "peak_gib": 55.066,
        "rate": 1974.87098148543
      }
    }
  },
  "bf16-kvfp8": {
    "dev_quality": {
      "domains": {
        "code": {
          "bf16": 0.3508522727272727,
          "drop": 0.009943181818181818,
          "items": 88,
          "quantized": 0.3409090909090909,
          "se": 0.010345549600666457,
          "status": "valid",
          "upper": 0.02696161091127814
        },
        "commonsense": {
          "bf16": 0.7466666666666667,
          "drop": 0.0,
          "items": 300,
          "quantized": 0.7466666666666667,
          "se": 0.008347257041177222,
          "status": "valid",
          "upper": 0.01373123783273653
        },
        "expert": {
          "bf16": 0.4375,
          "drop": -0.004166666666666667,
          "items": 30,
          "quantized": 0.44166666666666665,
          "se": 0.039063817027606036,
          "status": "R5_diagnostic",
          "upper": 0.060093312343745264
        },
        "knowledge": {
          "bf16": 0.44916666666666666,
          "drop": 0.02,
          "items": 300,
          "quantized": 0.42916666666666664,
          "se": 0.011201508896606656,
          "status": "valid",
          "upper": 0.03842648213491795
        },
        "long_context": {
          "bf16": 0.8655833333333334,
          "drop": 0.015875,
          "items": 100,
          "quantized": 0.8497083333333333,
          "se": 0.011494749295379884,
          "status": "valid",
          "upper": 0.03478386259089991
        },
        "math": {
          "bf16": 0.875,
          "drop": -0.0125,
          "items": 200,
          "quantized": 0.8875,
          "se": 0.007673864147531794,
          "status": "valid",
          "upper": 0.00012350652268980009
        }
      },
      "recovery": 0.9817481756547681,
      "tier": "B",
      "valid_domains": 5,
      "worst_domain": "knowledge",
      "worst_drop": 0.02,
      "worst_upper": 0.03842648213491795
    },
    "workload": {
      "bf16": {
        "parts": {
          "kv_gib": 38.193,
          "kv_usage": 0.2648,
          "other_gib": 1.625,
          "weights_gib": 15.27
        },
        "peak_gib": 55.088,
        "rate": 1978.6648643264093
      },
      "submission": {
        "parts": {
          "kv_gib": 19.094,
          "kv_usage": 0.1324,
          "other_gib": 1.625,
          "weights_gib": 15.3
        },
        "peak_gib": 36.02,
        "rate": 2355.5589311648987
      }
    }
  },
  "fp8-w8a8": {
    "dev_quality": {
      "domains": {
        "code": {
          "bf16": 0.3508522727272727,
          "drop": 0.011363636363636364,
          "items": 88,
          "quantized": 0.33948863636363635,
          "se": 0.010229449437465997,
          "status": "valid",
          "upper": 0.02819108068826793
        },
        "commonsense": {
          "bf16": 0.7466666666666667,
          "drop": 0.0008333333333333334,
          "items": 300,
          "quantized": 0.7458333333333333,
          "se": 0.006519238031498318,
          "status": "valid",
          "upper": 0.011557479895148067
        },
        "expert": {
          "bf16": 0.4375,
          "drop": 0.008333333333333333,
          "items": 30,
          "quantized": 0.42916666666666664,
          "se": 0.014598719305609815,
          "status": "R5_diagnostic",
          "upper": 0.03234822659106148
        },
        "knowledge": {
          "bf16": 0.44916666666666666,
          "drop": 0.015833333333333335,
          "items": 300,
          "quantized": 0.43333333333333335,
          "se": 0.010683075972113007,
          "status": "valid",
          "upper": 0.033406993307459235
        },
        "long_context": {
          "bf16": 0.8655833333333334,
          "drop": 0.016666666666666663,
          "items": 100,
          "quantized": 0.8489166666666667,
          "se": 0.013299620679299223,
          "status": "valid",
          "upper": 0.038544542684113885
        },
        "math": {
          "bf16": 0.875,
          "drop": 0.00625,
          "items": 200,
          "quantized": 0.86875,
          "se": 0.008938212669692035,
          "status": "valid",
          "upper": 0.0209533598416434
        }
      },
      "recovery": 0.9785870543299673,
      "tier": "B",
      "valid_domains": 5,
      "worst_domain": "long_context",
      "worst_drop": 0.016666666666666663,
      "worst_upper": 0.038544542684113885
    },
    "workload": {
      "bf16": {
        "parts": {
          "kv_gib": 38.18,
          "kv_usage": 0.2647,
          "other_gib": 1.625,
          "weights_gib": 15.27
        },
        "peak_gib": 55.075,
        "rate": 1917.163679270218
      },
      "submission": {
        "parts": {
          "kv_gib": 38.177,
          "kv_usage": 0.2532,
          "other_gib": 1.515,
          "weights_gib": 8.84
        },
        "peak_gib": 48.532,
        "rate": 2208.3922827169326
      }
    }
  },
  "fp8-w8a8-kvfp8": {
    "dev_quality": {
      "domains": {
        "code": {
          "bf16": 0.3508522727272727,
          "drop": -0.0014204545454545455,
          "items": 88,
          "quantized": 0.3522727272727273,
          "se": 0.009366684352277539,
          "status": "valid",
          "upper": 0.013987741214042006
        },
        "commonsense": {
          "bf16": 0.7466666666666667,
          "drop": 0.015833333333333335,
          "items": 300,
          "quantized": 0.7308333333333333,
          "se": 0.008825883987905533,
          "status": "valid",
          "upper": 0.030351912493437938
        },
        "expert": {
          "bf16": 0.4375,
          "drop": -0.0625,
          "items": 30,
          "quantized": 0.5,
          "se": 0.044346305681377866,
          "status": "R5_diagnostic",
          "upper": 0.010449672845866592
        },
        "knowledge": {
          "bf16": 0.44916666666666666,
          "drop": 0.0175,
          "items": 300,
          "quantized": 0.43166666666666664,
          "se": 0.007898199137133128,
          "status": "valid",
          "upper": 0.030492537580583998
        },
        "long_context": {
          "bf16": 0.8655833333333334,
          "drop": 0.0048333333333333344,
          "items": 100,
          "quantized": 0.86075,
          "se": 0.013969051490744325,
          "status": "valid",
          "upper": 0.02781242303560775
        },
        "math": {
          "bf16": 0.875,
          "drop": 0.005,
          "items": 200,
          "quantized": 0.87,
          "se": 0.009537021226218736,
          "status": "valid",
          "upper": 0.020688399917129823
        }
      },
      "recovery": 0.9818750874046539,
      "tier": "B",
      "valid_domains": 5,
      "worst_domain": "knowledge",
      "worst_drop": 0.0175,
      "worst_upper": 0.030492537580583998
    },
    "workload": {
      "bf16": {
        "parts": {
          "kv_gib": 38.189,
          "kv_usage": 0.2647,
          "other_gib": 1.625,
          "weights_gib": 15.27
        },
        "peak_gib": 55.084,
        "rate": 1917.828519284783
      },
      "submission": {
        "parts": {
          "kv_gib": 19.093,
          "kv_usage": 0.1266,
          "other_gib": 1.515,
          "weights_gib": 8.84
        },
        "peak_gib": 29.448,
        "rate": 2597.756456696642
      }
    }
  },
  "gptq-w4a16-g128": {
    "dev_quality": {
      "domains": {
        "code": {
          "bf16": 0.3508522727272727,
          "drop": 0.048295454545454544,
          "items": 88,
          "quantized": 0.3025568181818182,
          "se": 0.015835097182929393,
          "status": "valid",
          "upper": 0.0743441894113734
        },
        "commonsense": {
          "bf16": 0.7466666666666667,
          "drop": 0.015,
          "items": 300,
          "quantized": 0.7316666666666667,
          "se": 0.018756641335881497,
          "status": "valid",
          "upper": 0.04585467499752506
        },
        "expert": {
          "bf16": 0.4375,
          "drop": 0.07083333333333333,
          "items": 30,
          "quantized": 0.36666666666666664,
          "se": 0.059837943663830134,
          "status": "R5_diagnostic",
          "upper": 0.16926675066033392
        },
        "knowledge": {
          "bf16": 0.44916666666666666,
          "drop": 0.045,
          "items": 300,
          "quantized": 0.4041666666666667,
          "se": 0.020755652277133458,
          "status": "valid",
          "upper": 0.07914304799588454
        },
        "long_context": {
          "bf16": 0.8655833333333334,
          "drop": 0.041916666666666665,
          "items": 100,
          "quantized": 0.8236666666666667,
          "se": 0.01577027804043108,
          "status": "valid",
          "upper": 0.06785877404317579
        },
        "math": {
          "bf16": 0.875,
          "drop": 0.02,
          "items": 200,
          "quantized": 0.855,
          "se": 0.013066317125191795,
          "status": "valid",
          "upper": 0.04149409167094051
        }
      },
      "recovery": 0.9256857465276637,
      "tier": "fail",
      "valid_domains": 5,
      "worst_domain": "knowledge",
      "worst_drop": 0.045,
      "worst_upper": 0.07914304799588454
    },
    "workload": {
      "bf16": {
        "parts": {
          "kv_gib": 38.202,
          "kv_usage": 0.2648,
          "other_gib": 1.625,
          "weights_gib": 15.27
        },
        "peak_gib": 55.097,
        "rate": 1958.6995060338104
      },
      "submission": {
        "parts": {
          "kv_gib": 38.19,
          "kv_usage": 0.2482,
          "other_gib": 1.615,
          "weights_gib": 5.68
        },
        "peak_gib": 45.486,
        "rate": 1066.0776167881504
      }
    }
  },
  "gptq-w4a16-g128-kvfp8": {
    "dev_quality": {
      "domains": {
        "code": {
          "bf16": 0.3508522727272727,
          "drop": 0.05539772727272727,
          "items": 88,
          "quantized": 0.29545454545454547,
          "se": 0.01664272475578896,
          "status": "valid",
          "upper": 0.08277500949600011
        },
        "commonsense": {
          "bf16": 0.7466666666666667,
          "drop": 0.0225,
          "items": 300,
          "quantized": 0.7241666666666666,
          "se": 0.018787263232167103,
          "status": "valid",
          "upper": 0.05340504801691488
        },
        "expert": {
          "bf16": 0.4375,
          "drop": 0.041666666666666664,
          "items": 30,
          "quantized": 0.3958333333333333,
          "se": 0.06997878715254356,
          "status": "R5_diagnostic",
          "upper": 0.1567817715326008
        },
        "knowledge": {
          "bf16": 0.44916666666666666,
          "drop": 0.05,
          "items": 300,
          "quantized": 0.39916666666666667,
          "se": 0.019823637119074856,
          "status": "valid",
          "upper": 0.08260988306087813
        },
        "long_context": {
          "bf16": 0.8655833333333334,
          "drop": 0.03216666666666667,
          "items": 100,
          "quantized": 0.8334166666666667,
          "se": 0.016743518101902354,
          "status": "valid",
          "upper": 0.05970975394429604
        },
        "math": {
          "bf16": 0.875,
          "drop": 0.02125,
          "items": 200,
          "quantized": 0.85375,
          "se": 0.01501438799064645,
          "status": "valid",
          "upper": 0.04594866824461341
        }
      },
      "recovery": 0.9176419180628873,
      "tier": "fail",
      "valid_domains": 5,
      "worst_domain": "code",
      "worst_drop": 0.05539772727272727,
      "worst_upper": 0.08277500949600011
    },
    "workload": {
      "bf16": {
        "parts": {
          "kv_gib": 38.195,
          "kv_usage": 0.2657,
          "other_gib": 2.145,
          "weights_gib": 15.27
        },
        "peak_gib": 55.61,
        "rate": 1918.617195554359
      },
      "submission": {
        "parts": {
          "kv_gib": 19.087,
          "kv_usage": 0.1241,
          "other_gib": 1.615,
          "weights_gib": 5.72
        },
        "peak_gib": 26.423,
        "rate": 967.410714007257
      }
    }
  },
  "nvfp4-w4a4": {
    "dev_quality": {
      "domains": {
        "code": {
          "bf16": 0.3508522727272727,
          "drop": 0.019886363636363636,
          "items": 88,
          "quantized": 0.3309659090909091,
          "se": 0.010279206062604103,
          "status": "valid",
          "upper": 0.03679565760934739
        },
        "commonsense": {
          "bf16": 0.7466666666666667,
          "drop": 0.06916666666666667,
          "items": 300,
          "quantized": 0.6775,
          "se": 0.020481193392829923,
          "status": "valid",
          "upper": 0.10285822979787189
        },
        "expert": {
          "bf16": 0.4375,
          "drop": 0.058333333333333334,
          "items": 30,
          "quantized": 0.37916666666666665,
          "se": 0.06278193116833898,
          "status": "R5_diagnostic",
          "upper": 0.16160961010525096
        },
        "knowledge": {
          "bf16": 0.44916666666666666,
          "drop": 0.0275,
          "items": 300,
          "quantized": 0.4216666666666667,
          "se": 0.01934997889893099,
          "status": "valid",
          "upper": 0.05933071528874148
        },
        "long_context": {
          "bf16": 0.8655833333333334,
          "drop": 0.05408333333333333,
          "items": 100,
          "quantized": 0.8115,
          "se": 0.018520336821506316,
          "status": "valid",
          "upper": 0.08454928740471122
        },
        "math": {
          "bf16": 0.875,
          "drop": 0.0125,
          "items": 200,
          "quantized": 0.8625,
          "se": 0.01237373413656573,
          "status": "valid",
          "upper": 0.032854792654650626
        }
      },
      "recovery": 0.9292796857679062,
      "tier": "fail",
      "valid_domains": 5,
      "worst_domain": "commonsense",
      "worst_drop": 0.06916666666666667,
      "worst_upper": 0.10285822979787189
    },
    "workload": {
      "bf16": {
        "parts": {
          "kv_gib": 38.193,
          "kv_usage": 0.2648,
          "other_gib": 1.625,
          "weights_gib": 15.27
        },
        "peak_gib": 55.088,
        "rate": 1917.7810205821002
      },
      "submission": {
        "parts": {
          "kv_gib": 38.18,
          "kv_usage": 0.2485,
          "other_gib": 1.515,
          "weights_gib": 5.98
        },
        "peak_gib": 45.675,
        "rate": 2583.5119703234036
      }
    }
  },
  "nvfp4-w4a4-kvfp8": {
    "dev_quality": {
      "domains": {
        "code": {
          "bf16": 0.3508522727272727,
          "drop": 0.026988636363636364,
          "items": 88,
          "quantized": 0.32386363636363635,
          "se": 0.010191972560835353,
          "status": "valid",
          "upper": 0.04375443122621052
        },
        "commonsense": {
          "bf16": 0.7466666666666667,
          "drop": 0.10416666666666667,
          "items": 300,
          "quantized": 0.6425,
          "se": 0.021296414106816028,
          "status": "valid",
          "upper": 0.13919926787237905
        },
        "expert": {
          "bf16": 0.4375,
          "drop": 0.004166666666666667,
          "items": 30,
          "quantized": 0.43333333333333335,
          "se": 0.07052841741337494,
          "status": "R5_diagnostic",
          "upper": 0.12018591331166845
        },
        "knowledge": {
          "bf16": 0.44916666666666666,
          "drop": 0.04666666666666667,
          "items": 300,
          "quantized": 0.4025,
          "se": 0.020574704590118488,
          "status": "valid",
          "upper": 0.08051205571741157
        },
        "long_context": {
          "bf16": 0.8655833333333334,
          "drop": 0.034833333333333334,
          "items": 100,
          "quantized": 0.83075,
          "se": 0.017921366152129643,
          "status": "valid",
          "upper": 0.0643139806535866
        },
        "math": {
          "bf16": 0.875,
          "drop": 0.005,
          "items": 200,
          "quantized": 0.87,
          "se": 0.010774061537324272,
          "status": "valid",
          "upper": 0.022723331228898427
        }
      },
      "recovery": 0.906021657484018,
      "tier": "fail",
      "valid_domains": 5,
      "worst_domain": "commonsense",
      "worst_drop": 0.10416666666666667,
      "worst_upper": 0.13919926787237905
    },
    "workload": {
      "bf16": {
        "parts": {
          "kv_gib": 38.186,
          "kv_usage": 0.2657,
          "other_gib": 2.145,
          "weights_gib": 15.27
        },
        "peak_gib": 55.602,
        "rate": 1971.2009762328585
      },
      "submission": {
        "parts": {
          "kv_gib": 19.088,
          "kv_usage": 0.1242,
          "other_gib": 1.515,
          "weights_gib": 6.01
        },
        "peak_gib": 26.613,
        "rate": 3554.2626559157734
      }
    }
  }
}

Recent submissions

f9baf58abbb74a4b trial2-8b (participant) · scored FP8-W8A8+FP8head+INT4emb+KV8+seqs22+kvbudget192k-trial2-8b lowbarena-trial-8b2-fp8-fp8head-int4emb W8A8KV8 done
9ac6451d1ab440bf trial2-8b (participant) · development FP8-W8A8+FP8head+INT4emb+KV8+seqs22+kvbudget192k+ws128-trial2-8b lowbarena-trial-8b2-fp8-fp8head-int4emb W8A8KV8 done
a553eefea2d24359 trial-opus-5.5 (participant) · scored FP8-W8A8+FP8-head+INT4-emb+calKV-FP8+seqs64+bt4096+compile1-trial-opus-5.5 lowbarena-trial-fp8-v1 W8A8KV8 done
d55250a215c34184 trial2-8b (participant) · development FP8-W8A8+FP8head+INT4emb+KV8+seqs12+bt4096-trial2-8b lowbarena-trial-8b2-fp8-fp8head-int4emb W8A8KV8 done
1848de776cc64df2 trial-opus-5.5 (participant) · development FP8+FP8head+INT4emb+FP8KV+seqs64+bt4096+cs1-32-64-trial-opus-5.5 lowbarena-trial-fp8-v1 W8A8KV8 done
2a7505d964444853 trial-opus-5.5 (participant) · development FP8-W8A8+FP8-head+INT4-emb+calKV-FP8+seqs64+bt4096+compile1-trial-opus-5.5 lowbarena-trial-fp8-v1 W8A8KV8 done
cb418d9c5770437d trial-opus-5.5 (participant) · development FP8-W8A8+FP8-head+INT4-emb+calKV-FP8+seqs64+compile1-trial-opus-5.5 lowbarena-trial-fp8-v1 W8A8KV8 done
bf7c333bea7e4de2 trial-opus-5.5 (participant) · development FP8-W8A8+FP8-head+INT4-emb+calKV-FP8+seqs64 (replicate)-trial-opus-5.5 lowbarena-trial-fp8-v1 W8A8KV8 done
2ff9a2b7888f4534 trial-opus-5.5 (participant) · development FP8-W8A8+FP8-head+INT4-emb+calKV-FP8+seqs64+bt4096-trial-opus-5.5 lowbarena-trial-fp8-v1 W8A8KV8 done
b0c4e95f426f48e2 trial-opus-5.5 (participant) · development FP8-W8A8+FP8-head+INT4-emb+NVFP4-KV+shared-scratch+bt4096-trial-opus-5.5 lowbarena-trial-fp8-v1 W8A8KV4 done
a706df17c4784125 trial-opus-5.5 (participant) · development FP8-W8A8+FP8-head+INT4-emb+calKV-FP8+seqs64-trial-opus-5.5 lowbarena-trial-fp8-v1 W8A8KV8 done
65ccffc38b59472e trial-opus-5.5 (participant) · development FP8-dynamic+NVFP4-KV(uncal)+seqs64-trial-opus-5.5 RedHatAI/Qwen3-8B-FP8-dynamic W8A8KV4 done
8cf31177dcf0413a arena-check (participant) · development stock-vLLM-plugin-arena-check RedHatAI/Qwen3-8B-FP8-dynamic W8A8KV8 done
b9b6f9c6034a4949 Red-team engine: skips 2 of 36 layers (reference) · scored redteam-skip-2-layers Qwen3-8B-FP8-Dynamic W8A8KV16 done
eff88ec942264431 Own engine: FP8 + FP8 KV, 16K prefill chunks (reference) · scored FP8-chunked-prefill Qwen3-8B-FP8-Dynamic W8A8KV8 done
f942081b2fbb40d0 RedHatAI/Qwen3-8B-NVFP4 (reference) · scored NVFP4, llm-compressor RedHatAI/Qwen3-8B-NVFP4 W4A4KV16 paused
c06fbd82dc4946bc nvidia/Qwen3-8B-FP8 (reference) · scored FP8, ModelOpt nvidia/Qwen3-8B-FP8 W8A8KV8 paused
4e04108c4761423a RedHatAI/Qwen3-8B-FP8-dynamic (reference) · scored FP8 dynamic, llm-compressor RedHatAI/Qwen3-8B-FP8-dynamic W8A8KV16 paused
7f6389cc47674691 RedHatAI/Qwen3-8B-quantized.w4a16 (reference) · scored GPTQ W4A16, llm-compressor RedHatAI/Qwen3-8B-quantized.w4a16 W4A16KV16 paused
8f883e5db1db4c4a nvidia/Qwen3-8B-NVFP4 (reference) · scored NVFP4, ModelOpt nvidia/Qwen3-8B-NVFP4 W4A4KV8 done
fee712e078904618 Unsloth NF4 (reference) · scored NF4, Unsloth unsloth/Qwen3-8B-bnb-4bit W4A16KV16 done
727880e648af4ca9 Unsloth dynamic NF4 + FP8 KV (reference) · scored NF4 dynamic, Unsloth + FP8 KV unsloth/Qwen3-8B-unsloth-bnb-4bit W4A16KV8 done
bf552623a2b6401d Unsloth dynamic NF4 (reference) · scored NF4 dynamic, Unsloth unsloth/Qwen3-8B-unsloth-bnb-4bit W4A16KV16 done
21711dfa533e449e Qwen/Qwen3-8B-AWQ (reference) · scored AWQ Qwen/Qwen3-8B-AWQ W4A16KV16 done
eaeaddae97584b8d Qwen/Qwen3-8B-FP8 (reference) · scored FP8, 128x128 blocks Qwen/Qwen3-8B-FP8 W8A8KV16 done
c131ca5692714e79 GPTQ W4A16 + scale search + FP8 KV (reference) · scored GPTQ + scale search + FP8 KV Qwen3-8B-GPTQ-W4A16-chat-ss W4A16KV8 done
d8a09bc6347b42f5 NVFP4 GPTQ + scale search + FP8 KV (reference) · scored NVFP4 GPTQ + scale search + FP8 KV Qwen3-8B-NVFP4-chat-ss W4A4KV8 done
fc88002bf2374bc2 NVFP4 GPTQ + scale search (reference) · scored NVFP4 GPTQ + scale search Qwen3-8B-NVFP4-chat-ss W4A4KV16 done
f71afbaddb374b9c SpinQuant + GPTQ W4A16 (reference) · scored SpinQuant + GPTQ Qwen3-8B-SpinQuant-W4A16 W4A16KV16 done
eaceb118d8d04924 GPTQ W4A16 + scale search (reference) · scored GPTQ + scale search Qwen3-8B-GPTQ-W4A16-chat-ss W4A16KV16 done