Repository navigation
docs(datasets): use result attributes in run evaluator example - #2090
Open
shuaiyafei wants to merge 1 commit into
Open
shuaiyafei wants to merge 1 commit into
shuaiyafei wants to merge 1 commit into
Conversation
Assisted-by: OpenAI Codex (model: GPT-6, autonomous)
|
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
The
DatasetClient.run_experimentvocabulary-diversity example readsresult["output"], but run evaluators receiveExperimentItemResultobjects. Copying it into a run evaluator raisesTypeError: 'ExperimentItemResult' object is not subscriptable, so no run-level diversity score is produced. Useresult.output, matching the result type and the other run-evaluator examples.Found by checking the existing example against the SDK; no linked issue. This changes only the docstring, with no runtime API or generated-file changes.
Type of change
Verification
Local verification used Python 3.12.15.
content_diversityfunction and executed it through the SDK's_run_evaluatorwith realExperimentItemResultobjects. Before: the nonempty case raises theTypeErrorabove. After: two outputs (red blue,blue green) produce 0.75; an empty result list produces 0.0.uv run --frozen pytest tests/unit/test_datasets.py -q: 9 passed.uv run --frozen ruff format --check .: passes, 593 files already formatted.scripts/build_reference_docs.shinto a local output directory. The build succeeds with warnings about unresolved generated API re-exports. No live Langfuse or model-provider tests were run; validation uses local synthetic data.Checklist
code_review.mdusing Codex.AI assistance: OpenAI Codex (GPT-6, autonomous) prepared and validated this change.


Confidence Score: 5/5
This documentation fix appears safe to merge.Summary
Fixes the
content_diversityexample inDatasetClient.run_experiment.Reviews (1) · Last reviewed commit: "docs(datasets): use result attributes in..." · Reviewed by Greptile