[ehanmire]

Add f-router

Files (the three submission files only):

• router_inference/config/f-router.json
• router_inference/predictions/f-router.json (full, generated_result populated)
• router_inference/predictions/f-router-robustness.json (routing only)
• model_cost/model_profiles.yaml + regenerated model_cost/model_cost.json: one new entry for the router's own classifier (see "Router cost")

How it routes

1. A local classifier, Qwen3-4B-Instruct-2507 (4-bit, MLX, greedy decoding), reads each query and outputs a few compact requirement scores (reasoning depth, code, math, factual recall, format and similar). We also use the probabilities of those score tokens.
2. A small L2 logistic regression per candidate model predicts the chance that the model answers correctly. Its inputs are the classifier scores plus six generic structural features (length, code block, math notation, answer choices, JSON request, code request).
3. The router picks argmax_m p_hat(m) - lambda * expected_cost(m), with lambda = 0.269.

Candidates: gpt-oss-20b, z-ai/glm-5.3-flash, openai/gpt-6-luna, deepseek/deepseek-v4.1-flash@high. On full the router picked GLM for 5,500 queries and Luna for 2,900. It picked gpt-oss and DeepSeek for none.

Training data (no RouterArena data)

The predictor, lambda and the choice among predictor variants come only from an independently authored development set of 1,500 questions. They are procedurally generated or hand-written, with quotas that are not derived from any benchmark distribution. The set was split 924 train / 289 validation / 287 holdout, and the holdout was scored once before any RouterArena routing. On that holdout the router scored 0.9543 against 0.9535 for the best single model (Luna). Development results do not transfer directly to RouterArena. No RouterArena questions, labels, cached results, other submissions or leaderboard scores were used to train, tune or select anything. The structural features are generic regexes written for our development set; they were not written against RouterArena's prompt templates or config/eval_config/.

Answers

Every answer comes from our own API calls, one user message per query with no system prompt:

• GLM and Luna go through OpenRouter with reasoning effort low, max_tokens 8192, and one retry at 16384 if the output hits the length limit.
• The execution policy is identical to the one used on the development set.
• model_used records the model OpenRouter actually served. No row was answered by a different model.
• 4 GLM rows have no usable answer (1 hit the length limit after the retry, 3 came back empty from the provider). They are submitted with success: false and were not retried, matching the development-set policy.

Router cost

Every row lists the classifier call in router_usage: on average 778 input and 37 output tokens. The classifier runs locally and has no published API price, so we added a price profile priced conservatively at the nearest larger listed Qwen model (qwen/qwen3.5-9b, 0.17 / 0.25 per M tokens). We are happy to use a different price if the maintainers prefer.

Disclosures

• Before our project adopted a no-benchmark-data rule (2026-10-07), our setup downloaded the RouterArena splits and ran upstream format checks. One time, we counted aggregate category proportions of full to set quotas for an earlier synthetic set. That earlier set and those quotas are excluded from this router.
• An earlier version of this project routed RouterArena sub_10 / robustness / full with a different router. The classifier prompt was written and frozen before that run and has not changed since. Results from those runs were not used for this router.
• The DeepSeek candidate is served through the DeepSeek direct API (revision DeepSeek-V4.1-Flash-1017) under the registered id deepseek/deepseek-v4.1-flash@high. It was never selected in this submission.
• gpt-oss-20b is submitted under the id gpt-oss-20b, which is present in both the price table and universal_model_names.py. It was never selected in this submission.
• Some factual-recall questions in our development set may be memorized by the candidate models, which can make development accuracy optimistic.

🤖 Generated with Claude Code

[ehanmire]

/evaluate

[github-actions]

Router Evaluation Results

Router: f-router
Dataset Split: full
Evaluator: main @ 44cc1ff

RouterArena Metrics

Metric	Value
RouterArena Score	0.7082
Accuracy	71.59%
Total Cost	$1.782481
Avg Cost per Query	$0.000212
Avg Cost per 1K Queries	$0.2122
Number of Queries	8400
Abnormal Entries	4
Robustness Score	0.8714

⚠️ 4 of 8400 queries (0.0%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

---

Evaluation completed by RouterArena automated workflow