[Lanre Shittu (Shizoqua)]

Summary

Fixes #400.

The OfficeQA loader read the gold answer with str(row.get("ground_truth") or row.get("answer") or ""). In a JSON split, a numeric gold answer of 0 is falsy, so it was skipped: the item loaded with an empty ground_truth and no answers, and a model answering 0 scored EM 0 and F1 0. Integral floats had a related problem: 1776.0 loaded as "1776.0" and never matched the answer 1776. The item id used the same or chain, so "uid": 0 became an empty id.

User impact: OfficeQA answers are mostly numbers from Treasury tables, and a JSON writer such as pandas to_json stores a float column as 1776.0. Those items gave zero reward for the correct answer and counted as misses in reported scores, and items with id 0 were hard to trace.

Proof: the regression tests fail on upstream 45f65ca and pass with this change.

Root cause

_normalize_row() in skillopt/envs/officeqa/dataloader.py chained or across ground_truth and answer, and across uid and id. or treats 0 and 0.0 as missing. str() of an integral float keeps the .0, and the evaluator compares "1776" with "1776.0" as text. CSV splits are not affected because CSV values are strings.

Changes

• New _first_text(row, *keys) helper returns the first value that is not None and not blank, as stripped text. Integral floats are written as integers first (1776.0 -> "1776", 0.0 -> "0").
• _normalize_row() uses it for the gold answer and for the item id.
• Non-integral floats (12.5) and string values (including "1776.0") are unchanged. The answer fallback and items with no gold answer behave as before; a whitespace-only ground_truth now also falls back to answer.
• New tests/test_officeqa_dataloader.py, loading real JSON split files through OfficeQADataLoader.load_split_items().
• CHANGELOG entry under Unreleased / Fixed.

Updated after review feedback (thanks Muhammad Abdullah Butt (@1AbdullahButt)) to handle integral floats and numeric ids.

Reproduction

import json, pathlib, tempfile
from skillopt.envs.officeqa.dataloader import OfficeQADataLoader
from skillopt.envs.officeqa.evaluator import evaluate

split = pathlib.Path(tempfile.mkdtemp()) / "val"
split.mkdir()
rows = [
    {"uid": 0, "question": "How many bonds were issued?", "ground_truth": 0},
    {"uid": "q2", "question": "Year?", "ground_truth": 1776.0},
]
(split / "items.json").write_text(json.dumps(rows))

items = OfficeQADataLoader().load_split_items(str(split))
for item, answer in zip(items, ["0", "1776"]):
    print(repr(item["id"]), repr(item["ground_truth"]), evaluate(answer, item["ground_truth"])["em"])

Before:

'' '' 0.0
'q2' '1776.0' 0.0

After:

'0' '0' 1.0
'q2' '1776' 1.0

Testing

• python -m pytest tests/test_officeqa_dataloader.py -q: on upstream 45f65ca, 6 failed and 4 passed; with the fix, 10 passed.
• All OfficeQA-related tests (tests/test_officeqa_*.py and tests/test_benchmark_task_output_paths.py): 117 passed.
• python -m pytest -q: 2226 passed, 12 skipped, 4 xfailed.
• python -m ruff check on the changed files: one F401 for an unused import os that is unchanged from main; the new test file is clean.
• python -m mkdocs build --strict: passed.

Risk and compatibility

Only JSON items with a numeric zero, an integral float, or a whitespace-only value in these fields load differently. No backend, config key, file format or CLI change. Numeric equivalence of strings such as "12" and "12.00" is left to the evaluator.

[Muhammad Abdullah Butt (1AbdullahButt)]

_first_text is right for the 0 and blank cases. None and whitespace-only values are skipped, 0 is kept, and ground_truth: "" correctly falls through to answer. Two things in the same function, though.

1. A JSON 0.0 is kept, but as "0.0", so an answer of 0 still scores 0. str(0.0) is "0.0". normalize_answer (officeqa/evaluator.py:11) keeps . and digits, so exact_match("0", "0.0") compares "0" with "0.0" and returns 0.0, and token_f1 does too. The new test only checks evaluate("0.0", "0.0"). The same happens for any integral float in a JSON split: a ground_truth of 1776.0 won't match 1776. A JSON writer such as pandas to_json on a float column produces exactly that. One option is to format integral floats as ints when loading:

if isinstance(value, float) and value.is_integer():
    value = int(value)

The other is to compare numerically in exact_match when both sides parse as numbers. The first is local to this PR. The second also covers "12" vs "12.00" in CSV.

2. The same or pattern drops a numeric id of 0. item_id = str(row.get("uid") or row.get("id") or "").strip() (dataloader.py:46) turns "uid": 0 into "", or into the id field if there is one. _first_text(row, "uid", "id") fixes it the same way. A JSON split with integer uids starting at 0 is a plausible case, and an item with an empty id is hard to trace in reports.

git merge-tree against main is clean. #401, #402 and #403 all add their CHANGELOG.md entries at the same spot, so they conflict with each other there. I read the loader and evaluator at the PR head. I didn't run the tests.

Reviewed with help from Claude Code.

[Lanre Shittu (Shizoqua)]

Thanks, both points were right. Pushed ac950fa: integral floats now load as integers (1776.0 -> "1776", 0.0 -> "0"), so an integer answer matches, and the item id goes through _first_text, so "uid": 0 or "id": 0 is kept. Non-integral floats and string values are unchanged. Added tests for both.