fix(skillsbench): reject incomplete court forms

.claude/notes/lint-rules.md

@@ -961,3 +961,12 @@ the module as a parameter instead of a third copy.
 Taking the class name from the model follows CE056 importing `IN_CONTAINER_ENV` and CE057
 deriving its target set from `SIDECAR_MODULES`. Alias resolution removes the *local
 binding* spelling, not every rename; taking the name from the model covers the rest.
+
+## CE071 — patched binary verifiers must not skip
+
+The court-form verifier skipped checks when checkbox data or blank-field keys
+were absent. Its shell runner treated pytest's successful exit as reward 1.
+The guard rejects skip and xfail calls or marks in the two locally patched
+binary-reward SkillsBench verifiers. It also rejects bare `except:` and
+`except BaseException:` handlers. Real PDF regression tests
+also anchor the court-form field names and extraction to the shipped input.

pyproject.toml

@@ -67,6 +67,7 @@ Changelog = "https://github.com/UiPath/coder_eval/blob/main/CHANGELOG.md"
 
 [project.optional-dependencies]
 dev = [
+    "pypdf[crypto]==6.21.4",
     "pytest-asyncio>=1.3.0",
     "pytest>=9.0.2",
     "pytest-mock>=3.15.1",
@@ -241,8 +242,8 @@ constraint-dependencies = [
 [tool.ruff]
 line-length = 120
 target-version = "py313"
-# Vendored third-party sample tasks (benchflow-ai/skillsbench) are kept verbatim;
-# they are not framework code and are not linted/formatted.
+# Vendored third-party sample tasks (benchflow-ai/skillsbench) include local
+# verifier patches documented in their README. Keep them outside framework formatting.
 extend-exclude = ["tasks/samples"]
 
 [tool.ruff.lint]
@@ -310,6 +311,7 @@ external = [
     "CE065",
     "CE066",
     "CE068",
+    "CE071",
 ]  # custom architectural lint rules (tests/lint/)
 
 [tool.ruff.lint.pylint]

tasks/samples/skillsbench/README.md

@@ -7,18 +7,26 @@ evaluation (not just a smoke test) with one command.
 They are vendored from [**benchflow-ai/skillsbench**](https://github.com/benchflow-ai/skillsbench)
 (Apache-2.0), pinned at commit
 [`95b3e60`](https://github.com/benchflow-ai/skillsbench/tree/95b3e60de7a075090e2f9137f32e0d24da1a289b).
-Each sample keeps the upstream prompt, inputs, and verifier; only the container
-base and the coder_eval task wiring were added. The one local change is to the
-`3d-scan-calc` verifier, which is patched to grade against fixed expected
-values instead of re-deriving them from the agent-writable `scan_data.stl`
-(issue #233). The other verifiers are unchanged and can still read
-agent-writable inputs: `dialogue-parser` takes part of its ground truth from
-`/app/script.txt`.
-
-That patch closes only the `scan_data.stl` vector. Grading still runs in the
-agent's container, so the `pytest` on `PATH`, auto-loaded `pytest11` plugins,
-site-packages, and the `$TASK_DIR` verifier copy remain tamperable by the
-agent. See the follow-up issue
+Each sample keeps the upstream prompt and inputs. The container base and
+coder_eval task wiring were added locally, along with two verifier patches:
+
+- `3d-scan-calc` grades against fixed expected values instead of re-deriving
+  them from the agent-writable `scan_data.stl` (issue #233).
+- `court-form-filling` fails when required checkbox data or fields that must
+  remain blank are missing (issue #234). Its blank-field checks also replace
+  a nonexistent county field with the omitted third trial court address field.
+  AcroForm read and field dereference errors propagate to the test runner.
+  Its environment pins pypdf 6.21.4 (upstream 5.1.0) to remove known advisories.
+
+The `dialogue-parser` verifier is unchanged and takes part of its ground truth
+from the agent-writable `/app/script.txt`. The `court-form-filling` verifier
+also reads the agent-writable `/root/sc100-blank.pdf` for its input-versus-output
+identity check.
+
+The `3d-scan-calc` patch closes only the `scan_data.stl` vector. Grading still
+runs in the agent's container, so the `pytest` on `PATH`, auto-loaded `pytest11`
+plugins, site-packages, and the `$TASK_DIR` verifier copy remain tamperable by
+the agent. See the follow-up issue
 [#238](https://github.com/UiPath/coder_eval/issues/238).
 
 ## What they do
@@ -56,8 +64,8 @@ coder-eval run tasks/samples/skillsbench/*/*.yaml
 
 Each task builds its container, runs the agent (Claude Sonnet 4.6 by default) in
 an isolated sandbox, then scores the result with the upstream verifier (for
-`3d-scan-calc`, the locally patched one described above). Budget roughly 1–3
-minutes per task.
+`3d-scan-calc` and `court-form-filling`, the local patches described above).
+Budget roughly 1–3 minutes per task.
 
 ## Interpreting the results
 
@@ -90,6 +98,10 @@ is non-deterministic, so exact numbers vary from run to run:
 | `court-form-filling` | 1.00 |
 | `dialogue-parser` | ~0.83 |
 
+The court-form score above predates the verifier fix for issue #234 and the
+pypdf update. Those historical scores are not comparable with scores from
+the updated sample.
+
 ## How it works (brief)
 
 - **`environment/Dockerfile`** rebases the upstream task onto `coder-eval-agent` and bakes the inputs at the paths the prompt expects.
@@ -101,6 +113,6 @@ is non-deterministic, so exact numbers vary from run to run:
 
 These tasks are derived from benchflow-ai/skillsbench, licensed under Apache-2.0.
 The prompts, input data, and verifiers are the work of their original authors
-(credited in each upstream task's `task.md`). The `3d-scan-calc` verifier is
-patched locally to grade against fixed expected values. See the pinned commit
-above for the upstream source of truth.
+(credited in each upstream task's `task.md`). The `3d-scan-calc` and
+`court-form-filling` verifiers have the local patches described above. See the
+pinned commit above for the upstream source of truth.

tasks/samples/skillsbench/court-form-filling/environment/Dockerfile

@@ -11,7 +11,7 @@ RUN apt-get update \
     && apt-get install -y --no-install-recommends poppler-utils \
     && rm -rf /var/lib/apt/lists/* \
     && pip install --no-cache-dir \
-        pypdf==5.1.0 \
+        pypdf==6.21.4 \
         fillpdf==0.7.3 \
         pdfrw==0.4 \
         PyPDF2==3.0.1 \

tasks/samples/skillsbench/court-form-filling/verifier/test_outputs.py

@@ -1,3 +1,4 @@
+# Modified from benchflow-ai/skillsbench@95b3e60: require form fields and preserve AcroForm read and field dereference errors (issue #234).
 """
 Tests for court-form-filling task (SC-100 Small Claims Court form).
 
@@ -63,26 +64,23 @@ def extract_checkbox_values(pdf_path: Path) -> dict:
         checkbox_values.update(parse_xfa_checkboxes(xfa_xml))
 
     # Method 2: Try pypdf get_fields() for AcroForms
-    try:
-        from pypdf import PdfReader
-        reader = PdfReader(str(pdf_path))
-        fields = reader.get_fields() or {}
-
-        for name, field in fields.items():
-            # Check if it's a checkbox/radio field
-            if "Checkbox" in name or field.get("/FT") in ["/Btn"]:
-                value = field.get("/V", "")
-                if value:
-                    if hasattr(value, "get_object"):
-                        value = str(value.get_object())
-                    else:
-                        value = str(value)
-                    # Normalize value
-                    value = value.strip("/")
-                    if value and value not in ["Off", "None", ""]:
-                        checkbox_values[name] = value
-    except Exception:
-        pass
+    from pypdf import PdfReader
+    reader = PdfReader(str(pdf_path))
+    fields = reader.get_fields() or {}
+
+    for name, field in fields.items():
+        # Check if it's a checkbox/radio field
+        if "Checkbox" in name or field.get("/FT") in ["/Btn"]:
+            value = field.get("/V", "")
+            if value:
+                if hasattr(value, "get_object"):
+                    value = str(value.get_object())
+                else:
+                    value = str(value)
+                # Normalize value
+                value = value.strip("/")
+                if value and value not in ["Off", "None", ""]:
+                    checkbox_values[name] = value
 
     return checkbox_values
 
@@ -268,7 +266,6 @@ def normalize_text(text: str) -> str:
     ("SC-100[0].Page1[0].CaptionRight[0].CN[0].CaseName[0]", "page1_case_name"),
     ("SC-100[0].Page1[0].CaptionRight[0].CN[0].CaseNumber[0]", "page1_case_number"),
     ("SC-100[0].Page1[0].CaptionRight[0].County[0].CourtInfo[0]", "page1_court_info"),
-    ("SC-100[0].Page1[0].CaptionRight[0].County[0].County[0]", "page1_county"),
     # Trial date section 1
     ("SC-100[0].Page1[0].Order[0].List1[0].LI1[0].TrialDate1[0]", "page1_trial_date1"),
     ("SC-100[0].Page1[0].Order[0].List1[0].LI1[0].TrialTime1[0]", "page1_trial_time1"),
@@ -283,6 +280,7 @@ def normalize_text(text: str) -> str:
     ("SC-100[0].Page1[0].Order[0].List1[0].LI3[0].TrialDate3[0]", "page1_trial_date3"),
     ("SC-100[0].Page1[0].Order[0].List1[0].LI3[0].TrialTIme3[0]", "page1_trial_time3"),
     ("SC-100[0].Page1[0].Order[0].List1[0].LI3[0].TrialDepartment3[0]", "page1_trial_dept3"),
+    ("SC-100[0].Page1[0].Order[0].List1[0].LI3[0].TrialDateCourtAdd3[0]", "page1_court_addr3"),
     ("SC-100[0].Page1[0].Order[0].List1[0].LI3[0].TrialDateClerkSign[0]", "page1_clerk_sign"),
     ("SC-100[0].Page1[0].Order[0].List1[0].LI3[0].TrialDateClerkSignDate[0]", "page1_clerk_date"),
     # Second plaintiff/defendant (only one of each in this case)
@@ -347,10 +345,7 @@ def pdf_field_values():
         value = field.get("/V", "")
         if value:
             if hasattr(value, "get_object"):
-                try:
-                    value = str(value.get_object())
-                except:
-                    value = str(value)
+                value = str(value.get_object())
             else:
                 value = str(value)
         field_values[name] = value if value else ""
@@ -421,9 +416,6 @@ def test_checkbox_value(self, checkbox_data, field_pattern, description, expecte
         }
 
         if not matching_fields:
-            # If no checkboxes found at all, it might be extraction failure
-            if not checkbox_data:
-                pytest.skip("No checkbox data could be extracted from PDF (XFA extraction may not be supported)")
             pytest.fail(
                 f"Checkbox '{description}' (pattern: {field_pattern}) not found. "
                 f"Available checkboxes: {list(checkbox_data.keys())}"
@@ -459,9 +451,6 @@ def test_checkbox_unchecked(self, checkbox_data, field_pattern, description):
         }
 
         if not matching_fields:
-            # If no checkboxes found at all, it might be extraction failure
-            if not checkbox_data:
-                pytest.skip("No checkbox data could be extracted from PDF")
             # Field not found means it's likely unchecked (not in filled data)
             return
 
@@ -479,6 +468,7 @@ class TestEmptyFields:
                              ids=[f[1] for f in EMPTY_FIELDS])
     def test_field_empty(self, pdf_field_values, field_name, description):
         """Verify field is empty or has minimal placeholder."""
-        value = pdf_field_values.get(field_name, "")
+        assert field_name in pdf_field_values, f"Field '{description}' not found in PDF form"
+        value = pdf_field_values[field_name]
         is_empty = not value or value in ["", "None"] or len(value) < 3
         assert is_empty, f"Field '{description}' should be empty, got: '{value}'"

tests/test_custom_lint.py

@@ -5399,3 +5399,72 @@ def test_shares_manifest_skill_dirs_with_the_runtime_allowlist(self):
         from coder_eval.isolation import eval_material
 
         assert eval_material.manifest_skill_dirs is manifest_skill_dirs
+
+
+@pytest.mark.lint
+class TestCE071PatchedBinaryVerifiersDoNotSkip:
+    """CE071: patched binary verifiers must not skip, xfail, or catch BaseException.
+
+    Scope: the locally patched 3d-scan-calc and court-form-filling verifiers.
+    Blind spots: imported aliases and dynamically constructed calls are not resolved.
+    """
+
+    @staticmethod
+    def _offenders(source: str) -> list[int]:
+        import ast
+
+        return [
+            node.lineno
+            for node in ast.walk(ast.parse(source))
+            if (
+                isinstance(node, ast.Call)
+                and isinstance(node.func, ast.Attribute)
+                and isinstance(node.func.value, ast.Name)
+                and node.func.value.id == "pytest"
+                and node.func.attr in {"skip", "importorskip", "xfail"}
+            )
+            or (
+                isinstance(node, ast.Attribute)
+                and node.attr in {"skip", "skipif", "xfail"}
+                and isinstance(node.value, ast.Attribute)
+                and node.value.attr == "mark"
+                and isinstance(node.value.value, ast.Name)
+                and node.value.value.id == "pytest"
+            )
+            or (
+                isinstance(node, ast.ExceptHandler)
+                and (node.type is None or (isinstance(node.type, ast.Name) and node.type.id == "BaseException"))
+            )
+        ]
+
+    @pytest.mark.parametrize("slug", ["3d-scan-calc", "court-form-filling"])
+    def test_patched_verifier(self, slug: str) -> None:
+        path = (
+            Path(__file__).resolve().parents[1]
+            / "tasks"
+            / "samples"
+            / "skillsbench"
+            / slug
+            / "verifier/test_outputs.py"
+        )
+        offenders = self._offenders(path.read_text(encoding="utf-8"))
+        assert not offenders, f"CE071: silent outcome or broad catch in {path} at lines {offenders}"
+
+    @pytest.mark.parametrize(
+        "source",
+        [
+            "pytest.skip('missing')",
+            "pytest.importorskip('pypdf')",
+            "pytest.xfail('missing')",
+            "@pytest.mark.skip\ndef test(): pass",
+            "@pytest.mark.skipif(True, reason='missing')\ndef test(): pass",
+            "@pytest.mark.xfail\ndef test(): pass",
+            "try: read()\nexcept: pass",
+            "try: read()\nexcept BaseException: pass",
+        ],
+    )
+    def test_detects_silent_outcomes(self, source: str) -> None:
+        assert self._offenders(source)
+
+    def test_allows_failure_and_narrow_exception(self) -> None:
+        assert not self._offenders("try: read()\nexcept ValueError: pytest.fail('invalid')")