Skip to content

BUG: Scorer evaluation accepts zero trials then fails after dataset extraction #3901

Open

Description

@romanlutz

Describe the bug

ObjectiveScorerEvaluator.evaluate_dataset_async accepts num_scorer_trials=0, extracts dataset messages through the memory interface, and then fails with an internal NumPy IndexError. Reject nonpositive trial counts at entry, before dataset extraction or scorer calls.

Source: pyrit/score/scorer_evaluation/scorer_evaluator.py:356-436,793-839 at the tested revision.

Steps/Code to Reproduce

import asyncio
from unittest.mock import AsyncMock, MagicMock
from pyrit.models import Message, MessagePiece, Score
from pyrit.score import (
    HumanLabeledDataset, MetricsType, ObjectiveHumanLabeledEntry,
    ObjectiveScorerEvaluator, TrueFalseScorer,
)

async def main():
    scorer = MagicMock(spec=TrueFalseScorer)
    scorer._memory = MagicMock()
    scorer._memory.add_message_to_memory_async = AsyncMock()
    scorer.get_chat_target.return_value = None
    scorer.score_async = AsyncMock(
        return_value=[Score(score_type="true_false", score_value="true")]
    )
    response = Message(message_pieces=[
        MessagePiece(role="assistant", original_value="response")
    ])
    dataset = HumanLabeledDataset(
        name="trial-boundary", version="1", metrics_type=MetricsType.OBJECTIVE,
        entries=[ObjectiveHumanLabeledEntry([response], [True], "objective")],
    )
    await ObjectiveScorerEvaluator(scorer).evaluate_dataset_async(
        dataset, num_scorer_trials=0
    )

asyncio.run(main())

Expected Results

A clear ValueError before memory/scorer calls. Cover zero and negative counts and preserve one-trial metrics. Apply the same contract consistently to higher-level evaluation entry points.

Actual Results

IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

The original probe records one awaited memory-add call and zero scorer calls at zero trials. A one-trial control succeeds with accuracy 1.0 and trial_scores == [[True]].

Screenshots

N/A.

Versions

Linux; Python 3.12.3; PyRIT 1.2.0.dev0 at 01ce6ca66a315770b36c7dca1f320668c0d92d7c. Actual evaluator/models/metrics, synthetic one-entry dataset, mocked memory and scorer. No real database, CLI, or external inference. Full package-version output was not collected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions