[grinder81]

Problem

LlamaModel.detokenize allocates 32 bytes for each token piece. llama_token_to_piece returns a negative required size when the buffer is too small. The existing assert n <= size accepts that negative result, then slicing with it silently omits the token bytes. We encountered this with Bangla tokens while requesting echoed prompt probabilities: truncated text offsets broke the evaluator's answer-position lookup.

Change

Allocate the returned required size and retry. Reject a negative or oversized retry result instead of slicing invalid data. Reuse the enlarged buffer for subsequent tokens. No model, scoring, native-library or vendor changes.

Validation

• Current upstream base: 1652066e0af45f2313b339670ef9555e8a54e545.
• New regression suite on unmodified upstream: 6 failed, 4 passed. With fix: 10 passed.
• Cases cover 0/1/31/32/33/34/128 bytes, multi-token Bangla text and Bengali digits, buffer reuse, special-token flag forwarding, and invalid retry results.
• make lint: passed across package and tests.
• Full suite after fetching the required model/vocabulary fixtures: 36 passed in 524.75 seconds. CPU-only run (CUDA_VISIBLE_DEVICES empty); JUnit evidence saved locally. Earlier offline fixture failures are resolved.
• Linux/WSL Ubuntu, Python 3.13.15. The focused regression tests mock token-to-piece calls and require no model computation; the full suite uses real upstream model fixtures. Native library is built from the pinned vendor revision with our separately tracked Tiny Aya tokenizer patch; that patch does not alter token-to-piece decoding and is not part of this PR. Earlier local integration evidence: Fix local Python decoding buffer for Bangla scoring grinder81/tiny-aya-edge#20.

Searched existing issues/PRs for detokenize-buffer and token_to_piece fixes; no matching fix found. Prepared with AI assistance; validation scope and limitations are stated above.