Skip to content

Fix INT8 weight-gradient quantization and CUDA operand layout - #2115

Open
devinhuang2001 wants to merge 1 commit into
bitsandbytes-foundation:mainfrom
devinhuang2001:fix/issue-2112
Open

devinhuang2001 wants to merge 1 commit into
bitsandbytes-foundation:mainfrom
devinhuang2001:fix/issue-2112

Conversation

@devinhuang2001

Copy link
Copy Markdown

Fixes #2112

MatMul8bitLt.backward scales row-quantized grad_output with column statistics. Use the column-quantized tensor that matches those statistics.

The CUDA path also needs CAt.t() materialized contiguously: its int8 GEMM passes raw pointers and shape-derived leading dimensions without honoring transpose strides. On a T4, correcting only the quantization pair still gives a 1.47 relative weight-gradient error; making the second operand contiguous reduces the single-operator error to 0.0090.

Adds 16 regression cases covering 2D/3D inputs, thresholds 0/6, input gradients enabled/disabled, and both bnb.matmul and Linear8bitLt. The oracle computes grad_output.T @ input in fp32 with deliberately different gradient-column scales.

Validation on Tesla T4, PyTorch 2.11.0+cu130, source baseline a616886 (using the packaged CUDA 13.0 native library):

  • New regression cases: 16 failed on baseline; 16 passed with both corrections.
  • 32 selected existing CUDA test_matmullt cases: passed (selected across the collected case list).
  • Full pre-commit run --all-files: all 10 hooks passed.
  • git diff --check: passed.

The full test suite and other accelerator backends were not run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MatMul8bitLt backward pairs row-quantized grad_output with column stats, so weight gradients are wrong

1 participant