Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .github/workflows/spider2-benchmark.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
name: Spider2 benchmark checks
on:
pull_request:
workflow_dispatch:
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python: ['3.10', '3.13']
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v6
with:
python-version: ${{ matrix.python }}
- run: uv sync --extra benchmark --locked
- run: uv run --extra benchmark pytest -q
- run: uv run --with black black --check bench/spider2 tests/test_spider2.py
- run: uv run --with ruff ruff check --isolated --select E4,E7,E9,F bench/spider2 tests/test_spider2.py
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,9 @@ uv sync # create .venv and install deps
Shows the federation money-shot (one term, two team definitions, no conflict) and
the safety gate (DROP/INSERT blocked, SELECT passes). See [`bench/README.md`](bench/README.md).

For a reproducible public benchmark across API models, see the
[Spider 2.0-Lite SQLite baseline](bench/spider2/README.md).

### 2. Run the CLI (developer driver)

```bash
Expand Down
4 changes: 4 additions & 0 deletions bench/spider2/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
runs/
*.sqlite
*.zip
log.txt
21 changes: 21 additions & 0 deletions bench/spider2/LICENSE-Spider2.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2024 bird_sql

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
73 changes: 73 additions & 0 deletions bench/spider2/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Spider 2.0-Lite SQLite baseline

Evaluate the existing Lang2SQL agent on a pinned public benchmark across API providers. This is evaluation infrastructure, not a Codex migration. Product source under `src/` is unchanged from `e0a7fb1`. The measured system is **Lang2SQL master + a selected API model**, not a claim about Lang2SQL's lift over a plain model.

## Fixed protocol

- Upstream: `xlang-ai/Spider2`, commit `cafb867313aab4e674652054198f383cf4018943`.
- Full population: **135 SQLite (`local*`) tasks**, 30 referenced databases. All 135 have official evaluation configuration and gold CSVs at this revision.
- `manifest-sqlite135.json` fixes the questions, order, DB hashes, document hashes, gold result/configuration hashes and evaluator hash. `manifest-pilot20.json` is its first 20 tasks after seeded shuffle (seed 20261114); selection does not use difficulty or scores.
- Each question gets a new context, in-memory store and session. No cross-question learning. Only the official question, its named external document, and the normal Lang2SQL schema/tools reach the model. No gold SQL, gold table names or gold results are provided.
- The existing `ContextConcierge → agent_loop → tools → SafetyPipeline → SqlAlchemyExplorer` path runs unchanged. The benchmark supplies a measured LLMPort and read-only SQLite connections with per-query deadlines. Existing tool row limits remain; final SQL execution is separate and does not silently truncate output.
- Default budgets: 8 **total** model calls (including auxiliary tool inference), 4,096 output tokens per call, 120-second API deadline, 480-second case deadline, 30-second SQL deadline, 100,000 final rows, concurrency 2. No automatic provider fallback or API retries. Fix these settings before a comparison.
- All profiles use the same final-answer instruction: one fenced SQL block. Missing/multiple blocks are failures. Never infer a final answer from the last exploratory SQL.
- Official `evaluate_single_exec_result_instance` is loaded directly from the hash-pinned upstream evaluator. CSV column/order/tolerance handling is the official implementation. Our wrapper includes **all manifest IDs** in the denominator, including API errors, no SQL and SQL failures. It does not use the upstream CLI's intersection-only denominator or its hardcoded 547 denominator.
- Missing keys are `blocked_missing_key`, not 0% model accuracy. Incomplete runs have no accuracy. Partial task failures within a completed run count as incorrect.
- Token usage is taken from provider responses. Missing usage is counted, not treated as known zero. Dollar cost remains null: no invented rates or claims that token counts equal billed cost. `seconds` is the sum of case latencies, `wall_seconds` is elapsed time per profile when available.
- Snapshot IDs are preferred. Provider aliases can change; returned response model names are recorded. This is pass@1 with one sample per task, not a statistical confidence claim.

## Setup

```bash
uv sync --extra benchmark
# Place benchmark data outside this source repository.
git clone https://github.com/xlang-ai/Spider2.git /absolute/path/Spider2
git -C /absolute/path/Spider2 checkout cafb867313aab4e674652054198f383cf4018943
uv run --with gdown gdown 1coEVsCZq-Xvj9p2TnhBFoFTsY-UoYGmG -O /absolute/path/sqlite.zip
```

Extract `.sqlite` files into one DB directory, keeping their original basenames. Do not commit DBs, archives, credentials or provider debug logs. The upstream archive also contains a DB not referenced by this SQLite subset. The question manifests retain upstream MIT attribution in `LICENSE-Spider2.txt`; follow upstream data terms for the downloaded databases.

```bash
uv run --extra benchmark python -m bench.spider2.evaluate prepare \
--root /absolute/path/Spider2 --db-dir /absolute/path/dbs \
--output /tmp/manifest.json --limit 20
```

Compare the generated manifest with the checked-in one. `--limit 0` selects all 135. The `run` command verifies hashes and refuses product changes relative to the pinned baseline.

## API/model matrix

`models.example.json` describes independent profiles: provider/model identifier, name, and API key **environment variable name**. LiteLLM translates requests to the actual provider, while Lang2SQL's existing message/tool/response conversion is reused. Provider-specific tool-call metadata is retained for follow-up turns (including Gemini thought signatures).

Run with `OKF_BUNDLE_DIR` unset so an unrelated local knowledge bundle cannot enter the baseline prompt.

Set the keys through your shell or a local `.env` file and use `--env-file /path/to/file`. Only variables referenced by profiles are used. Existing project configurations using `OPEN_AI_KEY` can reference that name in `api_key_env`; no key needs to be copied into the JSON. Unknown/missing keys never trigger another provider. Model availability must be checked against the relevant account; example model names are not a claim of live verification for every provider.

```bash
uv run --extra benchmark python -m bench.spider2.evaluate run \
--root /absolute/path/Spider2 --db-dir /absolute/path/dbs \
--manifest bench/spider2/manifest-pilot20.json \
--profiles bench/spider2/models.example.json \
--env-file /absolute/path/private.env \
--output /absolute/path/runs/pilot20-unique-id

uv run --extra benchmark python -m bench.spider2.evaluate score \
--root /absolute/path/Spider2 --output /absolute/path/runs/pilot20-unique-id
```

Use a new output directory for every run; overwriting or mixing outputs is rejected. Record files are written atomically per completed task, so interrupted runs retain evidence. A full run replaces the manifest with `manifest-sqlite135.json`; keep the same budgets and model profiles. No production application deployment changes are needed; the extra installs evaluation dependencies only.

## Results and limits

[Measured 135-task baseline results](RESULTS.md): GPT-4.1 mini 21/135 (15.56%), GPT-4.1 17/135 (12.59%). Anthropic/Gemini remain blocked on credentials.

Runs contain a protocol snapshot, per-model final SQL/CSV, per-task answer/transcript/API usage/timing/status, official per-task scores and a summary. API exception messages are not persisted because providers may echo credentials. Public result exports can omit bulky transcripts but should retain SQL, scores and the exact manifest/settings.

Report scores as **Spider 2.0-Lite SQLite subset**, with the exact denominator (20 pilot vs 135 full). Do not compare them directly to full 547-task Lite scores. This evaluates task completion, not federation isolation or 100-turn memory. Additional controlled ablations are needed to claim benefits from Lang2SQL context features.

## Sources

- https://github.com/xlang-ai/Spider2
- https://github.com/xlang-ai/Spider2/tree/main/spider2-lite/evaluation_suite
- https://docs.litellm.ai/docs/python_sdk
47 changes: 47 additions & 0 deletions bench/spider2/RESULTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Spider 2.0-Lite SQLite baseline — 2026-11-14

현재 Lang2SQL master (`e0a7fb1`)를 고정하고 API 모델만 교체한 첫 기준 실험이다. 제품 소스 변경은 없으며, 일반 모델 대비 Lang2SQL의 개선 효과를 측정한 ablation은 아니다.

## 전체 135문항 결과

| 모델 프로필 | 정답 | 공식 실행 결과 정확도 | 최종 예측 생성/실행 실패 | API 호출 | 입력 / 출력 토큰 |
|---|---:|---:|---:|---:|---:|
| openai-gpt41-mini | 21/135 | 15.56% | 50 | 519 | 1,351,879 / 130,702 |
| openai-gpt41 | 17/135 | 12.59% | 89 | 269 | 454,301 / 65,848 |
| anthropic-sonnet45 | 미실행 | — | — | — | API 키 없음 |
| google-gemini25-flash | 미실행 | — | — | — | API 키 없음 |

OpenAI 2개 모델은 실제 API로 실행했다. Anthropic/Gemini는 실행기와 프로필을 준비했지만 로컬에 해당 키가 없어 미실행이다. 다른 모델로 대체하거나 미실행을 0점으로 표시하지 않았다. 금액은 공급자 청구 자료로 확인하지 않아 계산하지 않았다.

## 재현 조건

- Spider2 upstream `cafb867313aab4e674652054198f383cf4018943`의 SQLite `local*` 135문항, DB 30개. 공식 Lite 전체 547문항 점수와 직접 비교하면 안 된다.
- seed 20261114로 고정한 문항 순서. 문항, 문서, DB, 채점 코드, gold CSV/config 해시를 manifest에 저장했다. Gold는 채점 과정에서만 사용했다.
- 실행 환경의 `OKF_BUNDLE_DIR`는 설정되지 않았으며 별도 로컬 knowledge bundle을 주입하지 않았다.
- 모델별 snapshot 이름은 [settings.json](results/sqlite135-v1/settings.json)에 고정했다. 각 문항은 독립 세션/메모리로 시작한다.
- 모든 모델에 동일한 최종 SQL 지시문, 총 8 API 호출 제한(보조 추론 포함), 호출당 출력 4096 토큰, 호출 120초 / 문항 480초 / SQL 30초 제한, 최종 결과 최대 100,000행. 자동 재시도나 모델 fallback 없음.
- 전체 실행의 모델별 concurrency는 4이다. 모델 프로필은 순차 실행했다. latency는 토큰 길이, API 상태, 로컬 경합의 영향을 받으므로 순수 모델 속도로 해석하지 않는다.
- 각 문항 1회 생성(pass@1). temperature/seed는 공급자 기본값이며 결정론적 재현이나 통계적 우열을 주장하지 않는다.
- upstream의 공식 `evaluate_single_exec_result_instance`를 직접 호출한다. 오류/출력 누락을 포함해 135개 모두 분모에 넣는다. 예측 실패와 SQL은 실행됐지만 정답 결과와 다른 경우를 구분한다.
- 실행 시작 시 checkout HEAD는 제품 baseline인 `e0a7fb1`이었다. 평가 코드가 아직 커밋 전이었으므로 settings의 runner/provider SHA256으로 실제 실행 코드를 식별한다. 해당 해시는 이 PR의 코드와 일치함을 확인했다.

## 사전 점검과 증거

20문항 예비 실행은 GPT-4.1 mini 2/20, GPT-4.1 3/20이었다. 이를 보고 프롬프트나 제품 코드를 튜닝하지 않았다. 첫 예비 실행은 SQLite 연결 정리 시 스레드 오류로 중단했고, benchmark 연결의 NullPool/check_same_thread 설정을 고친 다음 새 디렉터리에서 재실행했다. 중단된 실행은 점수에 포함하지 않았다. 예비 실행은 순차 실행이며 전체 실행과 latency를 비교하지 않는다.

[원본 설정/manifest/summary](results/sqlite135-v1/)와 각 모델의 `scores.json`, `records.json`, 생성 SQL을 포함했다. `records.json`에는 각 호출의 사용량/응답 모델/소요 시간, 문항 상태와 결과 CSV 해시가 있다. 큰 transcript와 CSV 본문은 로컬 실행 디렉터리에 보존하고 Git에는 포함하지 않았다. SQL을 동일 DB에 재실행하고 공식 evaluator로 다시 채점할 수 있다. 외부 API 오류 원문은 키가 노출될 수 있어 클래스만 기록한다.

예비 실행 오류를 재확인한 예: mini/local004의 `customer_id`, mini/local008의 `playerID`, GPT-4.1/local009의 `a1.longitude`는 실제 DB에 없는 컬럼을 참조했다. 후속 실험에서 스키마 탐색/최종 SQL 검증을 개선할 근거이며, 이번 baseline 점수를 올리려고 수정하지 않았다.

## 검증

- Python 3.10 및 3.13 전체 테스트: 각각 170 passed.
- Black/Ruff(신규 benchmark 코드), git diff whitespace 검사 통과.
- 실제 SQLite 읽기 전용 실행, ATTACH/쓰기 차단, 행 초과 실패, SQL 시간 제한, 실패를 포함한 채점 분모, provider metadata 보존, 호출 예산, 병렬 상한/키 누락 회귀 테스트.
- 실제 upstream evaluator에 정답 CSV 하나와 실패 하나를 넣어 1/2로 채점되는 통합 점검 완료. 이 1/2는 모델 성능이 아니라 채점기 점검이다.
- 기존 제품 mypy는 4개 파일의 19개 오류로 실패한다. `src/` 차이가 없어 이 PR에서 제품 타입 오류를 수정하지 않았다.
- API 키/DB 원본/압축파일은 커밋하지 않는다. LiteLLM benchmark extra 때문에 공유 의존성 lock 일부가 갱신되며 전체 기존 테스트도 실행했다.

## 해석 범위와 다음 실험

이 결과는 공개 SQLite 하위집합에서의 현행 Lang2SQL+각 모델 기준점이다. Oracle SQL 방언, Snowflake/BigQuery, federation 격리, 장기 기억 성능을 입증하지 않는다. 각 API의 키를 설정하면 같은 manifest와 예산으로 추가 공급자를 실행할 수 있다. Lang2SQL의 기여를 주장하려면 이후에 모델 단독/현재 Lang2SQL/문맥 기능을 켠 Lang2SQL을 같은 모델과 문항으로 통제 비교해야 한다.
1 change: 1 addition & 0 deletions bench/spider2/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"""Reproducible Spider 2.0-Lite SQLite evaluation (not a product backend change)."""
Loading
Loading