You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
BUG: Scorer evaluation accepts zero trials then fails after dataset extraction #3901
ObjectiveScorerEvaluator.evaluate_dataset_async accepts num_scorer_trials=0, extracts dataset messages through the memory interface, and then fails with an internal NumPy IndexError. Reject nonpositive trial counts at entry, before dataset extraction or scorer calls.
Source: pyrit/score/scorer_evaluation/scorer_evaluator.py:356-436,793-839 at the tested revision.
A clear ValueError before memory/scorer calls. Cover zero and negative counts and preserve one-trial metrics. Apply the same contract consistently to higher-level evaluation entry points.
Actual Results
IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed
The original probe records one awaited memory-add call and zero scorer calls at zero trials. A one-trial control succeeds with accuracy 1.0 and trial_scores == [[True]].
Screenshots
N/A.
Versions
Linux; Python 3.12.3; PyRIT 1.2.0.dev0 at 01ce6ca66a315770b36c7dca1f320668c0d92d7c. Actual evaluator/models/metrics, synthetic one-entry dataset, mocked memory and scorer. No real database, CLI, or external inference. Full package-version output was not collected.
Activity