You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Context.prepare can't prepare a query that calls grad, vjp or jvp #294
🌀 Claude: Context.sql() computes a query's grad, vjp and jvp calls through ddx (the ml extra, on the ml-demos branch of #269), but Context.prepare() plans the query with DataFusion alone, so it fails on the call:
importnumpyasnp, xarrayasxr, xarray_sqlasxqlctx=xql.Context()
ctx.from_dataset("w", xr.Dataset({"val": ("feature", np.array([1.0, 2.0]))}, coords={"feature": [0, 1]}))
q= ("WITH loss AS (SELECT SUM(val * val) AS l FROM w) ""SELECT feature, grad(loss, val) AS g FROM w ORDER BY feature")
ctx.sql(q).to_pandas() # g = [2.0, 4.0]ctx.prepare(q) # ValueError: Error during planning: Invalid function 'grad'. Did you mean 'rpad'?
(ddxdb 0.4.6, DataFusion 55.)
A training loop is where prepare matters most: it runs the same gradient step on every iteration, and planning ddx's programs costs far more than running them at small sizes. ddxdb has its own prepare, which plans the forward and backward programs once.
Then route a query that calls grad, vjp or jvp through ddxdb's prepare, keeping PreparedQuery's contract: writes into registered Datasets before a call, and the check that a table hasn't been registered again since the query was prepared.
sql_all's several statements over one backward pass would want the same, as a prepare_all or a prepare that takes several statements.
Activity
Address review: route quoted identifiers past ddx, and comment demo 1…