Skip to content

Python api - #31

Merged
Transurgeon merged 17 commits into
mainfrom
python-api
Nov 10, 2026
Merged

Transurgeon merged 17 commits into
mainfrom
python-api

Conversation

@Transurgeon

Copy link
Copy Markdown
Member

Python expression API (supersedes #12)

Reworks the interface from #12 from @dance858 .

scope = sp.Scope()
x = scope.Variable(3, 1)
fn = sp.compile(sp.exp(x) + 2.0 * x)
x.value = [1.0, 2.0, 3.0]
fn.forward(); fn.jacobian(); fn.hessian(w)    # csr_matrix results

What's included
- Scope/Variable/Parameter, operators (+ - * @ ** .T [], broadcasting), and the sp.* atoms.
- sp.compile(f) for a single expression; sp.Problem(objective, constraints) for solver callbacks.
- README with a quick start, a parameters example, and a scipy.optimize example.

Differences from #12
- No new C code. compile(f) makes f the only constraint of an engine problem with a zero objective. _bindings/ is unchanged, so DNLP's use of _sparsediffengine is unaffected. #12's expr_* bindings no longer compiled against the engine's matrix* API.
- Much less code. A single generic Node(make, args, shape) replaces about 30 node classes plus the registry and dispatch modules. The new package is about 670 lines, down from about 2,100.
- Fixes the build setup. Restores the SparseDiffEngine submodule (#12 had replaced it with a symlink to a local path) and declares scipy as a dependency.
- Input checks. Atoms the engine only supports on plain Variables (log tan atanh entr atan2 rel_entr quad_over_lin prod) now raise a clear error for other arguments instead of failing inside the engine.
- Correct Hessians. Problem.hessian() refreshes the cached Jacobians the engine's Hessian relies on, so it is correct after forward-only calls.

Tests
- Every atom's value is checked against NumPy, and its Jacobian and Hessian against finite differences.
- API, validation and Problem tests, plus the README examples run as tests.
- The suite runs in the cibuildwheel step.

🤖 Generated with Claude Code

Transurgeon and others added 6 commits November 7, 2026 14:36
Reworks the interface from #12 against current main:

- Generic Node(make, args, shape) replaces ~30 node classes, the registry
  and the dispatch module; sp.* atoms are mostly one-liners.
- compile(f) is built on the existing problem bindings (f as the only
  constraint, zero objective), so no new C code; _bindings/ is unchanged.
- Parameters sync to the engine automatically when their values change.
- Restricted-domain atoms validate that their arguments are plain
  Variables, matching the engine's requirements.
- Tests check every atom's value, Jacobian and Hessian against NumPy and
  finite differences, and run the README examples. They run in the
  cibuildwheel step.
- scipy becomes a runtime dependency (results are csr_matrix).

Co-authored-by: dance858 <danielcederberg1@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The engine's Hessian reuses Jacobian values cached by eval_jacobian, so
Problem.hessian() returned wrong values unless gradient() and jacobian()
had been called at the same point (7 of 39 atom cases differed). It now
refreshes them itself; a regression test covers both Problem and compile.

Also drop the unused Scope._parameters list and fold redundant branches in
_mul and _index.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A NumPy Q now goes to the engine's dense (BLAS) quad_form instead of being
converted to CSR; scipy.sparse Q keeps the sparse path, matching matmul.

The engine assumes Q is symmetric (gradient 2Qx), so a non-symmetric
constant Q gave a correct value but wrong derivatives. Constant Q is now
replaced by (Q + Q')/2, which leaves x'Qx unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem._u now validates that an explicit u has exactly n_vars elements
(the C bindings read the buffer unchecked, so a short array read past its
end) and slices the scope's vector to n_vars so variables declared after
compile do not change what the problem sees.

Constants in expressions must be scalars or 2-D arrays. A 1-D array used to
become a column silently, which turned row + np.ones(n) into an (n, n)
outer sum. Every expression is now (n, 1) or (1, n), never (n,).

CompiledExpression.forward() returns an array shaped like the expression
(column-major reshape of the stacked values). Problem.constraint_forward()
still returns the flat stacked vector solvers expect.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Problem remembers the point at which each stage (objective forward,
constraint forward, gradient, Jacobian) was last computed and skips the
engine call when asked again for the same point. A point is the scope's
values version plus the parameter version for u=None, or the last explicit
u (one copy kept, compared with array_equal) so scipy's separate fun/jac
callbacks at the same x cost one forward.

gradient()/jacobian() raise if no forward has run, and hessian() requires
both forwards at the same point instead of silently mixing Jacobian values
from two points. forward(); jacobian(); hessian(w) now runs one forward and
one Jacobian in the engine instead of three and two.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
X[i] on a matrix used to gather by column-major flat position, a silent
departure from NumPy's row selection. It now raises; vectors still take a
single index and matrices take X[i, j], X[i, :] or X[:, j]. Boolean and
multi-dimensional index arrays raise instead of being cast to integers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@Transurgeon Transurgeon left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good! let me know if you want me to review also.

dance858 and others added 11 commits November 8, 2026 05:42
Value inputs used to be raveled column-major and only checked for element
count, so a (3, 2) or flat (6,) array was silently accepted for a (2, 3)
variable. Now:

- Variable.value, Parameter.value and CompiledExpression.hessian(weights)
  take the exact shape of the variable, parameter or f. A 1-D array is
  also accepted for vector shapes and a scalar for (1, 1).
- scope.set_values, the Problem point u and Problem.hessian lagrange must
  be 1-D with exactly n entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bool subclasses int, so x[True] and x[False] were treated as x[1] and
x[0] instead of raising. Boolean index arrays were already rejected; now
scalar bools (Python and NumPy) are rejected too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_Builder.__call__ recursed once per tree level, so a sum built term by
term in a Python loop raised RecursionError at compile time from about
1,000 terms. Raising the recursion limit does not help: Python 3.14 then
reports a stack overflow around 7,000 levels.

The builder now walks the tree with an explicit stack, like _find_scope.
Nodes are built in the same left-to-right post-order as before, so the
capsules and the parameter registration order are unchanged.

Tests cover a 5,000-term sum with a parameter (values, Jacobian, Hessian,
and a parameter update) and a 3,000-level nonlinear chain through both
compile() and Problem, checked against closed forms.

Remaining limits are in the engine, not the Python layer: its C recursion
overflows the stack somewhere between 1e5 and 1e6 levels,
problem_init_derivatives is roughly quadratic in depth for long add chains,
memory grows with nodes x n_vars, and a shared subexpression is
re-evaluated once per use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Run pytest in the cibuildwheel test step. The [tool.cibuildwheel]
  test-command in pyproject.toml was overridden by the workflow's
  CIBW_TEST_COMMAND env var, so the suite never ran on the wheels.
- quad_form with a Parameter Q: the engine assumes Q symmetric, so check
  symmetry when the atom is built and on every later Q.value update.
- Expression.__iter__ raises TypeError instead of silently iterating via
  __getitem__ (a bare expression passed as `constraints` gave 0 constraints).
- Reject None in Constant and in value setters instead of storing NaN.
- index_flat wraps its argument like the other atoms.
- __pow__ accepts any numbers.Real, including NumPy scalars.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Also name the deep-sum test's loop count n_adds rather than n_terms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A node shared by the objective and a constraint became one engine node, so a
constraint_forward at another point overwrote values that gradient() then
read. Build the objective and the constraints with separate builders.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each case is evaluated on Variables (the expression under test) and on their
values with NumPy stand-ins (the expected result), instead of a registered
function plus a separate reference per case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
x / c for a number or array c is x * (1 / c). Dividing by an expression stays
unsupported (the error points to x * d ** -1), and dividing by zero raises.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Transurgeon
Transurgeon merged commit d3129a3 into main Nov 10, 2026
19 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants