Skip to content

Show demo 1's query plans: DataFusion's joins against einfold's fused pipeline #297

Open

Description

@alxmrs

🌀 Claude: benchmarks/ml/01_attention.py says einfold, a query optimizer that rewrites SQL plans into array algorithms, runs causal attention as FlashAttention does: tiled scores, a running softmax, and no stored N×N table. Today the demo shows this only as an operator count and an assertion that a fused pipeline ran. A picture of the plan would make the claim visible, and teach how a SQL engine runs the query.

Proposal: for each spelling of attention, and for the Nadaraya–Watson control, show:

  • DataFusion's plan: the joins and aggregates the SQL describes, where the scores are a table of N² rows;
  • einfold's plan for the same query: the fused pipeline that replaces them.

DataFusion can draw a plan as Graphviz: LogicalPlan.display_graphviz() in datafusion-python, and EXPLAIN for the physical plan. einfold would need to describe its pipelines in the same form. The output could be:

  • a --plans DIR option that writes the images;
  • or a figure in benchmarks/ml/README.md, or on the docs site.

Raised in review of #269.

Activity

  1. added
    enhancementNew feature or request
    mlGradients in SQL, the ml extra and its demos
    on Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestmlGradients in SQL, the ml extra and its demos

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions