[whyKirbyy]

Corealis Router: charge Jev's routing calls, and redo the 78 empty answers

Follow-up to #235 (merged via #256), as asked there.

Router calls. Every row now has router_usage with the calls the router made to typesafe/jev-1.14: one to pick the model on every row, and a second one to pick the reasoning effort on the 961 rows where the chosen model has more than one effort. That's 9,361 calls in total. We hadn't saved the token counts per row in the October 22 run. To get them, we sent Jev the exact same requests again (same prompt, instructions and option order) and recorded the usage it billed: on average 785 input tokens per call, and 32 to 50 output tokens. The picks in the file are the original ones, unchanged. The re-sent requests gave the same pick in 98.5% of calls.

The 420 robustness rows (corealis-router-robustness.json) get router_usage the same way: 470 calls, re-sent as in the run, picks unchanged.

How Jev is served. Not on our hardware. Jev is TypeSafe's model, hosted by TypeSafe and called through OpenRouter's Decisions API (/api/alpha/decisions). Its published price is $0.042 per 1M input tokens, and output is free: https://openrouter.ai/typesafe/jev-1.14. This PR adds that price to model_cost/model_profiles.yaml (with model_cost.json rebuilt), with sync: false, because Jev doesn't appear in OpenRouter's /api/v1/models list, only at /api/v1/models/typesafe/jev-1.14/endpoints.

Empty answers. The bot asked us to regenerate the 78 queries that came back empty, so we sent each one again to the same model at the same settings and host. 39 now have answers. The other 39 hit the 32,000-token thinking limit again, mostly DeepSeek V4.1 Flash at high effort, and stay as failed generations (success: false). Nothing else in the file changed.

Local check (current main): check_config_prediction_files.py corealis-router full --check-generated-result passes. compute_cost.py corealis-router: router calls $0.31 out of the $9.65 total. llm_evaluation/run.py: accuracy 75.88%, $1.07 per 1K queries, score 72.23, 39 abnormal entries.

🤖 Generated with Claude Code

[whyKirbyy]

/evaluate

[whyKirbyy]

/evaluate

[github-actions]

Router Evaluation Results

Router: corealis-router
Dataset Split: full
Evaluator: main @ 44cc1ff

RouterArena Metrics

Metric	Value
RouterArena Score	0.7223
Accuracy	75.88%
Total Cost	$9.026017
Avg Cost per Query	$0.001075
Avg Cost per 1K Queries	$1.0745
Number of Queries	8400
Abnormal Entries	39
Robustness Score	0.9500

⚠️ 39 of 8400 queries (0.5%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

---

Evaluation completed by RouterArena automated workflow