[George-Avila]

Add IQ Routing submission

Router

IQ Routing (https://iq-routing.com) is a closed-source, general-purpose production
gateway. It is not a research prototype built for this benchmark — its main workloads are coding
turns and cache protection, not QA benchmarks. The answers in this submission were generated by
IQ's routing pipeline, executed in an offline experiment harness against IQ's own provider
accounts. The gateway build carrying this
configuration is being deployed separately.

For this benchmark, each prompt was answered by exactly one of the five pool models, chosen by
IQ's routing. One routing decision and one generation per query — no cascade, no answer-side
ensembling, and no re-routing on a wrong answer.

The one exception is a provider-side content block: on 4 of the 8,400 rows (four NarrativeQA
queries) the selected model returned HTTP 400 for prohibited content before generating anything, and
IQ's fallback rung (the same rung the production gateway uses) served gpt-5.6-luna instead. The
blocked attempt produced no tokens and no cost; those rows are billed only at the model that actually
answered, which every row reports in model_used.

After the first /evaluate run flagged 17 rows with no valid generation, the 15 that had failed
transiently were re-run once through the same routing policy and the prediction file was updated in place (commit b9ce3f0); 13 now
carry valid generations and 2 still have no valid generation. The two GeoBench image rows stay failed.

Robustness file recomputed (2026-09-22). iq-routing-robustness.json was regenerated after correcting a
defect in how one of our routing features read reworded prompts. The 8,400 predictions in
iq-routing.json are unchanged: the corrected feature makes the same decisions on all 8,400 original
prompts, so the served model, accuracy and cost on every full-split row are untouched. Only the
robustness split moves, recomputed offline from the robustness pass's already-recorded routing decisions
with no new generations (commit dab1fe6). The same feature version was deployed in the production
gateway on 2026-09-22.

All five pool models were served internally, against IQ's own provider accounts (BYOK), by that
same pipeline. Costs on this leaderboard are therefore computed by RouterArena from reported token
counts at public list prices, not from anything IQ was billed.

Pool model
gemini-3-flash-preview
gemini-3.8-flash
gpt-5-nano
gpt-5.6-luna
gpt-5.6-terra

Files

File	Rows	Notes
router_inference/predictions/iq-routing.json	8,400	full split, with generated_result and router_usage
router_inference/predictions/iq-routing-robustness.json	420	routing-only, no generated_result (unchanged)
router_inference/config/iq-routing.json	—	5-model pool
model_cost/model_profiles.yaml	+1 alias	bare gpt-5.6-terra on openai/gpt-5.6-terra
model_cost/model_cost.json	regenerated	python scripts/pricing/build_model_cost.py
universal_model_names.py	+1	gpt-5.6-terra

The branch is merged with current main (44cc1ff), and the pricing comes from main's table.
gemini-3-flash-preview, gemini-3.8-flash, gpt-5-nano, gpt-5.6-luna and gpt-4o-mini already
had profiles there. The only missing spelling was the bare gpt-5.6-terra, which is now an alias of
openai/gpt-5.6-terra ($2.00/$12.00 per 1M). The earlier price rows this PR added are gone.

Routing calls (router_usage)

IQ makes one LLM call per query to decide the route. Every row now records that call in
router_usage as {"model": "openai/gpt-4o-mini", "input_tokens", "output_tokens", "purpose": "routing"}.
The values come from the run's per-query routing logs, matched to rows by global index.
output_tokens is the provider's completion_tokens, which includes any reasoning. The routing model
reported 0 reasoning tokens on every call. Nothing else in the file changed: with router_usage
removed, the prediction file is byte-identical to the previously evaluated commit dab1fe6.

• 8,397 rows have one routing call: 15,408,927 input and 645,732 output tokens in total.
• 3 rows have router_usage: []:
  • GeoBench_841 and GeoBench_1080 made no routing call. The gateway returned the
    no-image notice before routing.
  • LiveCodeBench_375: the only routing attempt timed out (see below), so the row took the default route.

Two routing attempts failed with no usage returned. We don't have their token counts, so we
recorded no count for them. We did not enter zero:

Row	Error	What happened
LiveCodeBench_375	TimeoutError	No routing call completed; router_usage is []
PubMedQA_311	CancelledError (run interrupted, then resumed)	Re-routed; the completed call is recorded in router_usage

Cost on current main

scripts/pricing/compute_cost.py iq-routing on this branch:

	USD
Answers	5.918364
Router calls	2.698778
Total (all 8,400 rows)	8.617142 ($1.0259 per 1K)

The $8.35 I posted earlier ($5.650297 answers + $2.698778 routing) was a historical figure. It used
the earlier gemini-3.8-flash price of $0.75/$3.75. main now prices that model at $0.825/$4.125.

Evaluator mismatch: please check. compute_cost.py bills every row. The official score
(llm_evaluation/run.py) skips rows that have no valid generation, so it bills neither their answer nor their router
calls. Four rows have no valid generation here. The two GeoBench rows have no cost either way. The two
cap-truncated QANTA rows (QANTA_Fine Arts_1531, QANTA_Literature_620) each made one real routing
call, and that call stays in router_usage. The two methods give these totals:

Method	Total	Per 1K
compute_cost.py (all rows)	$8.617142	$1.0259
Official scorer (valid generations only)	$8.599339	$1.0237

The $0.017803 gap is $0.000606 of routing and $0.017197 of answer tokens on those two QANTA rows.
We left the usage as recorded. Whether to charge failed rows is your call.

Two GeoBench queries (GeoBench_841, GeoBench_1080) reference an image that the text-only
dataset does not carry. The production gateway answers those with a "no image attached" notice
rather than guessing. They are submitted honestly as failed rows: success: false,
empty generated_answer, error: "no_image_attached", zero tokens. They score 0 and contribute
no cost. We did not substitute a guessed answer.

Evaluation Integrity

• No component of IQ was trained or fitted on RouterArena data or its label files. There is no
  learned router, no classifier fitted to RouterArena labels, and no weights of any kind derived
  from this dataset. The label files were used for scoring only.
• IQ is a closed-source, general-purpose production gateway. Its main workloads are coding
  turns and cache protection, not QA benchmarks. The outputs submitted here come from a fixed
  routing configuration of that gateway, served internally through IQ's own provider accounts.
• Costs are recomputed by the board. Nothing here reports what IQ was billed: RouterArena
  recomputes cost from the reported token counts at public list prices.

Reproducibility

Every row carries the provider-reported model_used, provider, and token counts from the
original generation. token_usage reports input_tokens, output_tokens, and total_tokens;
total == input + output on every row, so RouterArena's reasoning-token residual is zero by
construction. Routing calls are recorded separately in router_usage, as described above.

Prompts are byte-identical to dataset/router_data.json. The routing-only robustness file was
produced by the same routing, over the 420 prompts of dataset/router_robustness.json.

[George-Avila]

/evaluate

[github-actions]

Router Evaluation Results

Router: iq-routing
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7631
Accuracy	79.79%
Total Cost	$5.611358
Avg Cost per Query	$0.000668
Avg Cost per 1K Queries	$0.6680
Number of Queries	8400
Abnormal Entries	17
Robustness Score	0.3833

⚠️ 17 of 8400 queries (0.2%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

---

Evaluation completed by RouterArena automated workflow

[George-Avila]

/evaluate

[github-actions]

Router Evaluation Results

Router: iq-routing
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7636
Accuracy	79.86%
Total Cost	$5.650297
Avg Cost per Query	$0.000673
Avg Cost per 1K Queries	$0.6727
Number of Queries	8400
Abnormal Entries	4
Robustness Score	0.3833

⚠️ 4 of 8400 queries (0.0%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

---

Evaluation completed by RouterArena automated workflow

[George-Avila]

/evaluate

[github-actions]

Router Evaluation Results

Router: iq-routing
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7636
Accuracy	79.86%
Total Cost	$5.650297
Avg Cost per Query	$0.000673
Avg Cost per 1K Queries	$0.6727
Number of Queries	8400
Abnormal Entries	4
Robustness Score	0.9619

⚠️ 4 of 8400 queries (0.0%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

---

Evaluation completed by RouterArena automated workflow

[George-Avila]

The current automated result excludes the routing-decision call to gpt-4o-mini. Recorded usage totals 15,408,927 input and 645,732 output tokens, adding $2.69877825 at $0.15/$0.60 per million tokens. Including it gives $8.34907555 total ($0.9939 per 1,000 queries). Usage is unavailable for two failed routing-decision attempts. Which field or sidecar convention should we use to charge this call at its own model price?

[yl231]

Thanks, @George-Avila, and sorry for the slow answer. There isn't a field for this yet, and we're adding one. Routers that call an LLM to make the routing decision will record that call's model and its reasoning and response tokens in the prediction file, and a tool will compute its cost from the price table.

Once that's in, please add your gpt-4o-mini routing calls in that format; a short note is fine for the two failed attempts without usage. We'll re-price all routers together once the rebuilt price table is in place, and we'll post here when the format is ready.

[George-Avila]

Thanks @yl231. Yes, I still have the per-query routing logs, with query IDs and per-call token counts for all 8,400 rows. I've updated the PR to the new format:

• Merged current main. The merge conflict is resolved, and the pricing comes from main's table. The only addition is the bare gpt-5.6-terra alias, added to model_profiles.yaml and universal_model_names.py, with model_cost.json regenerated.
• All 8,400 rows now carry router_usage, taken from the per-query routing logs: one openai/gpt-4o-mini call per row. GeoBench_841 and GeoBench_1080 made no routing call. LiveCodeBench_375's only routing attempt timed out. The two attempts that failed without returning usage (LiveCodeBench_375, PubMedQA_311) are listed in the PR description, and no token counts were made up for them. Answers and the robustness file are unchanged.
• compute_cost.py now gives $8.617142 ($1.0259/1K). The $8.35 I quoted earlier was a historical figure based on the old Gemini 3.8 Flash price.

One mismatch to flag: compute_cost.py bills all rows, including the two cap-truncated QANTA rows. The official scorer leaves out failed-generation rows, so its total would be $8.599339. The difference is $0.017803, and $0.000606 of that is routing. Details are in the PR description. The usage records stay as logged.

Requesting another evaluation below.

[George-Avila]

/evaluate

[github-actions]

Router Evaluation Results

Router: iq-routing
Dataset Split: full
Evaluator: main @ 44cc1ff

RouterArena Metrics

Metric	Value
RouterArena Score	0.7567
Accuracy	79.98%
Total Cost	$8.599339
Avg Cost per Query	$0.001024
Avg Cost per 1K Queries	$1.0237
Number of Queries	8400
Abnormal Entries	4
Robustness Score	0.9619

⚠️ 4 of 8400 queries (0.0%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

---

Evaluation completed by RouterArena automated workflow