Skip to content

Context.prepare can't prepare a query that calls grad, vjp or jvp #294

Open

Description

@alxmrs

🌀 Claude: Context.sql() computes a query's grad, vjp and jvp calls through ddx (the ml extra, on the ml-demos branch of #269), but Context.prepare() plans the query with DataFusion alone, so it fails on the call:

import numpy as np, xarray as xr, xarray_sql as xql

ctx = xql.Context()
ctx.from_dataset("w", xr.Dataset({"val": ("feature", np.array([1.0, 2.0]))}, coords={"feature": [0, 1]}))
q = ("WITH loss AS (SELECT SUM(val * val) AS l FROM w) "
     "SELECT feature, grad(loss, val) AS g FROM w ORDER BY feature")
ctx.sql(q).to_pandas()   # g = [2.0, 4.0]
ctx.prepare(q)           # ValueError: Error during planning: Invalid function 'grad'. Did you mean 'rpad'?

(ddxdb 0.4.6, DataFusion 55.)

A training loop is where prepare matters most: it runs the same gradient step on every iteration, and planning ddx's programs costs far more than running them at small sizes. ddxdb has its own prepare, which plans the forward and backward programs once.

Proposal:

  • Until this is done, Add an ml extra for gradients in SQL, and two ML demos #269 makes prepare() raise a NotImplementedError that names sql() and xarray_sql.ad.sql_all, rather than DataFusion's planning error.
  • Then route a query that calls grad, vjp or jvp through ddxdb's prepare, keeping PreparedQuery's contract: writes into registered Datasets before a call, and the check that a table hasn't been registered again since the query was prepared.
  • sql_all's several statements over one backward pass would want the same, as a prepare_all or a prepare that takes several statements.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    mlGradients in SQL, the ml extra and its demos

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions