[jiangLLM]

Description

PrefEval can count an empty judge answer as a personalized response. It also misses lowercase answers and some responses that are both unhelpful and against the user's preference.

This PR puts failed or invalid judge answers in a separate Judge Failure category. It accepts case and punctuation differences, and gives Unhelpful Response priority. An answer must start with a whole Yes or No word, so Not sure and Yesterday remain judge failures. All 16 standard Yes/No combinations match the reference benchmark.

The Excel report includes Judge Failure in its totals. Each header has its own text-wrap format, so number formats cannot remove the line breaks. The script also loads its local utils package before any unrelated package with the same name.

Fixes #2396.

Type of change

• Bug fix

How Has This Been Tested?

• python -m pytest tests/evaluation -q: 49 passed with pandas 2.3.1 and 49 passed with pandas 3.1.0.
• The Excel tests cover mixed results, no judge failures, only judge failures, and no samples. They check values, percentage formats, time columns, and text wrap on every header. All four cases fail without the header fix and pass with it.
• CLI tests cover an unrelated utils module or package on the Python path.
• make format on the four PR Python files and pre-commit checks passed.

Checklist

• Reviewed the diff and added regression tests.
• Linked the related issue.
• No separate documentation update is needed for this bug fix.

[github-actions]

This PR has been automatically marked as stale due to inactivity.

[Memtensor-AI]

🤖 Open Code Review

Target: PR #2397
Task: 2a4a6eaa9951e70c
Base: main
Head: fix/prefeval-error-type-classifier

🔍 OpenCodeReview found 2 issue(s) in this PR.

⚠️ 1 warning(s) occurred during review.

---

1. evaluation/scripts/PrefEval/pref_eval.py (L258)

The refactor centralizes the bucket labels in utils/pref_classify.py, but only JUDGE_FAILURE is imported here; the other five labels above are still raw literals ("Preference-Unaware Violation", "Preference Hallucination Violation", "Inconsistency Violation", "Unhelpful Response", "Personalized Response"). If a constant in pref_classify is ever renamed, the corresponding get_pct lookup silently returns 0 and the summary column reports a wrong value with no error. Consider importing PREFERENCE_UNAWARE_VIOLATION, PREFERENCE_HALLUCINATION_VIOLATION, INCONSISTENCY_VIOLATION, UNHELPFUL_RESPONSE, PERSONALIZED_RESPONSE and using them as keys here (e.g. get_pct(PERSONALIZED_RESPONSE)), keeping the Chinese text only in the Excel column names.

---

2. evaluation/scripts/PrefEval/pref_eval.py (L18-L19)

This module-level import now comes after an executable statement (sys.path.insert), which triggers Ruff's E402 (module-import-not-at-top-of-file), included in Ruff's default E4 selection used by this repo's pre-commit config. No other script under evaluation/scripts/ does this (they only mutate sys.path after all imports). Either silence it explicitly with # noqa: E402 so the lint intent is documented, or move the path bootstrap so the import can stay at the top.

Generated by cloud-assistant via Open Code Review.

[Memtensor-AI]

✅ Automated Test Results: PASSED

All tests passed (43/43 executed). memos_python_core/changed-repo-python: 43/43. Duration: 1s [advisory, non-gating] AI-generated tests on branch test/auto-gen-bf846b37cf479ccd-20261025023031: 128/128 passed — these do NOT affect the PR verdict; review the branch manually.

Branch: fix/prefeval-error-type-classifier

[Memtensor-AI]

✅ Automated Test Results: PASSED

All tests passed (43/43 executed, 6 skipped). memos_python_core/changed-repo-python: 43 passed, 6 skipped. Duration: 1s

Branch: fix/prefeval-error-type-classifier

[Memtensor-AI]

✅ Automated Test Results: PASSED

All tests passed (43/43 executed, 6 skipped). memos_python_core/changed-repo-python: 43 passed, 6 skipped. Duration: 2s

Branch: fix/prefeval-error-type-classifier