Skip to content

docs(datasets): use result attributes in run evaluator example - #2090

Open
shuaiyafei wants to merge 1 commit into
langfuse:mainfrom
shuaiyafei:docs/dataset-run-evaluator-results
Open

shuaiyafei wants to merge 1 commit into
langfuse:mainfrom
shuaiyafei:docs/dataset-run-evaluator-results

Conversation

@shuaiyafei

@shuaiyafei shuaiyafei commented Nov 3, 2026 •

Copy link
Copy Markdown

What does this PR do?

The DatasetClient.run_experiment vocabulary-diversity example reads result["output"], but run evaluators receive ExperimentItemResult objects. Copying it into a run evaluator raises TypeError: 'ExperimentItemResult' object is not subscriptable, so no run-level diversity score is produced. Use result.output, matching the result type and the other run-evaluator examples.

Found by checking the existing example against the SDK; no linked issue. This changes only the docstring, with no runtime API or generated-file changes.

Type of change

  • Documentation update

Verification

Local verification used Python 3.12.15.

  • Extracted the documented content_diversity function and executed it through the SDK's _run_evaluator with real ExperimentItemResult objects. Before: the nonempty case raises the TypeError above. After: two outputs (red blue, blue green) produce 0.75; an empty result list produces 0.0.
  • uv run --frozen pytest tests/unit/test_datasets.py -q: 9 passed.
  • uv run --frozen ruff format --check .: passes, 593 files already formatted.
  • Configured pre-commit hooks on the changed file: Ruff check/format and full SDK mypy pass.
  • Built the API reference through scripts/build_reference_docs.sh into a local output directory. The build succeeds with warnings about unresolved generated API re-exports. No live Langfuse or model-provider tests were run; validation uses local synthetic data.

Checklist

  • Self-reviewed the one-line diff with code_review.md using Codex.
  • Updated the affected example and executed it against the actual SDK result type.
  • Did not hand-edit generated files or commit secrets.

AI assistance: OpenAI Codex (GPT-6, autonomous) prepared and validated this change.

RetriggerView in GreptileConfidence Score: 5/5

This documentation fix appears safe to merge.

Summary

Fixes the content_diversity example in DatasetClient.run_experiment.

  • The diversity example reads output from each result object.

Reviews (1) · Last reviewed commit: "docs(datasets): use result attributes in..." · Reviewed by Greptile

Assisted-by: OpenAI Codex (model: GPT-6, autonomous)

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants