Skip to content

BUG: MAJORITY scorer explanation describes a tie as a strict false majority #3900

Open

Description

@romanlutz

Describe the bug

The MAJORITY true/false aggregator correctly returns False for a tie, but describes the tied result as a strict majority of constituent scorers not returning True. The wording overstates the evidence. Keep the existing verdict and change only the explanation.

Source: pyrit/score/true_false/true_false_score_aggregator.py:198-207,233-238 at the tested revision.

Steps/Code to Reproduce

from pyrit.models import ComponentIdentifier, Score
from pyrit.score.true_false.true_false_score_aggregator import TrueFalseScoreAggregator

identifier = ComponentIdentifier(class_name="ProbeScorer", class_module="probe")
result = TrueFalseScoreAggregator.MAJORITY([
    Score(score_type="true_false", score_value=value, scorer_class_identifier=identifier)
    for value in ("true", "false")
])
print(result.value)
print(result.description)

Expected Results

Return False and explain that a strict majority returning True was not reached, or distinguish a tie explicitly. Add description assertions for tied votes and actual true/false majorities.

Actual Results

False
A strict majority of constituent scorers did not return True in a MAJORITY composite scorer.

The same false explanation is used for [False, False, True], where a strict false majority actually exists. This is a wording issue only, not an incorrect aggregate verdict.

Screenshots

N/A.

Versions

Linux; Python 3.12.3; PyRIT 1.2.0.dev0 at 01ce6ca66a315770b36c7dca1f320668c0d92d7c. Actual models/aggregator in a standalone probe; no inference or persistence. Full package-version output was not collected.

Activity

  1. VISHNU0906 commented on Nov 4, 2026

    @VISHNU0906
    Contributor

    I would like to take this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions