Repository navigation
Add OpenGateway og/auto submission (jev-omni judge + task table) [sionic-gajae] - #269
Open
bellman-sionic wants to merge 5 commits into
Open
bellman-sionic wants to merge 5 commits into
bellman-sionic wants to merge 5 commits into
Conversation
…nic-gajae] - router_inference/config/opengateway-og-auto.json, predictions (full 8,400 with generations and per-row router_usage) and robustness predictions (420 routing-only rows) - price table: sionic-ai/jev-omni (OpenGateway list price, input $0.126/MTok, output $0), router_usage only Local eval (llm_evaluation/run.py): Arena 75.08, Accuracy 78.84, Cost/1k $0.86, Robustness 96.67. [sionic-gajae]
…ast deepseek-v4.1-flash) [sionic-gajae] - second full 8,400 run; both v12 and v13 policies were pre-registered before their full results were seen - table: coding -> deepseek-v4.1-flash (UltraFast), every other task/default -> gemini-3.8-flash (effort none); jev-omni judge on live OpenGateway, per-row router_usage = actual judge tokens Local eval (llm_evaluation/run.py): Arena 76.54, Accuracy 81.57, Cost/1k $1.25, Robustness 99.76. [sionic-gajae]
- table: multiple_choice/factual_recall/other -> gemini-3.7-flash (effort none); math/passage_qa/fallback -> gemini-3.8-flash (effort none); translation -> glm-5.3-flash (UltraFast); coding -> deepseek-v4.1-flash (effort low, UltraFast) - jev-omni judge priced at OpenGateway current price (input $0.0063/1M, output $0) Local eval (llm_evaluation/run.py): Arena 77.12, Accuracy 81.61, Cost/1k $0.96, Robustness 96.67. [sionic-gajae]
Author
|
/evaluate |
… empty LiveCodeBench_419 answer as unsuccessful [sionic-gajae] [sionic-gajae]
Author
|
/evaluate |
Router Evaluation ResultsRouter: RouterArena Metrics
Evaluation completed by RouterArena automated workflow |
LiveCodeBench_419 hit the 32,000-token reasoning limit with deepseek-v4.1-flash@low and came back empty. Resent the same prompt to the same model/effort: 17,677 output tokens, finish_reason stop. Local re-score: 0 abnormal, accuracy 81.62%, $0.9607 per 1K queries, Arena 77.14. [sionic-gajae]
Author
|
/evaluate |
Router Evaluation ResultsRouter: RouterArena Metrics
Evaluation completed by RouterArena automated workflow |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[Submission] OpenGateway og/auto (jev-omni judge + task table) [sionic-gajae]
What
og/auto. A judge model (sionic-ai/jev-omni, OpenGateway decisions API) labels each query's task. A fixed task table then picks the serving model:google/gemini-3.7-flash(reasoning effort none)google/gemini-3.8-flash(reasoning effort none)z-ai/glm-5.3-flash(served on OpenGateway UltraFast)deepseek/deepseek-v4.1-flash(reasoning effort low, served on OpenGateway UltraFast)router_usagewith the judge's actual token counts. The judge is priced at the OpenGateway current price, input $0.0063/1M tokens, output $0 (model_cost/model_profiles.yaml,model_cost/model_cost.json).router_inference/config/opengateway-og-auto.json,router_inference/predictions/opengateway-og-auto.json(8,400 rows with generations),router_inference/predictions/opengateway-og-auto-robustness.json(420 routing-only rows), plus the jev-omni price entries (model_cost/model_cost.json,model_cost/model_profiles.yaml,universal_model_names.py).Local evaluation (
llm_evaluation/run.py)llm_evaluation/run.pytotal $8.070033, $0.9607 per 1K)Notes
LiveCodeBench_419first came back empty:deepseek-v4.1-flash@lowspent all 32,000 output tokens on reasoning and hit the cap. The same prompt was resent to the same model and effort and finished normally (17,677 output tokens,finish_reasonstop). With that row, the local re-score reports 0 abnormal entries.[sionic-gajae]