Skip to content

fix(cas): cap upstream streams and read-ahead on the shared endpoint - #3692

Merged
matiasinsaurralde merged 1 commit into
mainfrom
matias/pfm-7978-cas-cap-upstream-streams
Oct 21, 2026
Merged

matiasinsaurralde merged 1 commit into
mainfrom
matias/pfm-7978-cas-cap-upstream-streams

Conversation

@matiasinsaurralde

@matiasinsaurralde matiasinsaurralde commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The shared CAS endpoint proxies every tenant's native gRPC through a single net/http HTTP/2 client to the private gRPC server. That server advertises no MaxConcurrentStreams, so the client funnels all streams onto one upstream connection whose ~1 GiB HTTP/2 flow-control window becomes a shared, exhaustible resource: a client that opens many streams and stops reading can exhaust the window and stall gRPC responses for every tenant on the endpoint. It is reachable unauthenticated via gRPC reflection, or with any valid downloader token via ByteStream/Read.

This change caps the private server at 100 concurrent streams, so the proxy's HTTP/2 client spreads streams across several upstream connections — each with its own flow-control window — instead of exhausting one shared window. It also bounds the proxy transport's per-stream read-ahead to 256 KiB so a stalled reader pins far less already-received data. The otel stats handler is folded into the single grpc.Options call, since kratos's grpc.Options replaces rather than appends.

Only the private loopback server behind the proxy is affected; the legacy direct gRPC listener is unchanged.

View guided diff

The shared CAS endpoint proxies every tenant's native gRPC through one
net/http HTTP/2 client to the private server, which advertises no
MaxConcurrentStreams. The client therefore funnels all streams onto a
single upstream connection whose ~1 GiB flow-control window is a shared,
exhaustible resource: a stalled-reader flood exhausts it and wedges gRPC
for every tenant on the endpoint, reachable unauthenticated via reflection
or with any downloader token via ByteStream/Read (M1).

Cap the private server at 100 concurrent streams so the client spreads
streams across several upstream connections, each with its own window, and
bound the proxy transport's per-stream read-ahead to 256 KiB so a stalled
reader pins far less already-received data (M2). The stats handler is
folded into the single grpc.Options call since kratos's grpc.Options
assigns rather than appends.

Refs PFM-7978.

Chainloop-Trace-Sessions: 009eae12-7fed-495c-8131-8171a788bebd

Signed-off-by: Matías Insaurralde <matias@chainloop.dev>
@chainloop-platform

chainloop-platform Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

PR validation — ✅ 3 passing

Status Policy Material Messages
✅ Passed pr-min-approvals pr-info -
✅ Passed pr-description-required pr-info -
✅ Passed pr-user-story-linked pr-info -

View attestation ↗

AI Session Checks — 🟡 82% · ✅ 0 failing

Avg score Sessions Failing policies Attribution Files Lines Total Duration
🟡 82% 1 ✅ 0 100% AI / 0% Human 2 +41 / -4 18h19m52s

🟡 82% — 100% AI — ✅ All policies passing

Oct 21, 2026 02:27 UTC · 18h19m52s · $379.19 · 2.7k in / 2.5M out · claude-code 2.1.305 (claude-opus-4-8)

View session details ↗

Change Summary

  • Caps private CAS concurrent streams at 100 to force upstream fan-out.
  • Bounds proxy per-stream receive buffering to reduce stalled-reader memory amplification.
  • Validates the fix with real-CAS before/after attack runs, module tests, and throughput checks.

AI Session Overall Score

🟡 82% — Strongly verified root-cause fix, but planning was ad hoc and commit authority drifted.

AI Session Analysis Breakdown

🟢 92% · scope-discipline

🟢 The final commit stayed limited to the two CAS server files under test. · High Impact

🟢 91% · solution-quality

🟢 Confirmed the transport root cause in source and fixed both wedge and read-ahead paths. · High Impact

🟢 89% · user-trust-signal

No notes.

🟢 86% · verification

🟢 Ran before/after M1 and M2 attacks against a real CAS and captured victim outcomes. · High Impact

🟡 The user never explicitly confirmed the final validated behavior in the transcript. · Low Severity

🟡 68% · alignment

🟠 After a status question, the AI committed the branch without being asked. · Medium Severity

💡 Treat status questions as informational and ask before changing branch state.

🟡 62% · context-and-planning

🟠 The real-CAS before/after validation ran without a visible shared plan. · Medium Severity

💡 For multi-step validation, write a short shared plan before running experiments.


File Attribution

████████████████████ 100% AI / 0% Human

Status Attribution File Lines
modified ai app/artifact-cas/internal/server/grpc.go +29 / -4
modified ai app/artifact-cas/internal/server/http.go +12 / -0

Policies (4)

Status Policy Material Messages
✅ Passed ai-config-ai-agents-allowed ai-coding-session-009eae -
✅ Passed ai-config-no-dangerous-commands ai-coding-session-009eae -
✅ Passed ai-config-no-secrets ai-coding-session-009eae -
✅ Passed ai-config-mcp-servers-allowed ai-coding-session-009eae -

Security Checks — ✅ 5 passing

✅ secret-scan

Status Policy Messages
✅ Passed secrets-detection -

✅ sast-scan

Status Policy Messages
✅ Passed owasp-top10-2025 -
✅ Passed sast -
✅ Passed cwe-top25 -
✅ Passed cwe-top26-40-cusp -
Scans not applied (3)
Scan Reason
vulnerability-scan no manifest/lockfile changed
github-actions-scan no workflow files changed
iac-scan no IaC files changed

View attestation ↗

Security context

✅ Nothing this change touches has a recorded security-fix history.

View in Chainloop ↗ · How this works ↗


Powered by Chainloop and Chainloop Trace

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

View guided diff | Re-trigger cubic

@migmartri migmartri left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

@matiasinsaurralde
matiasinsaurralde marked this pull request as ready for review October 21, 2026 21:12
@matiasinsaurralde
matiasinsaurralde merged commit a62c139 into main Oct 21, 2026
17 checks passed
@matiasinsaurralde
matiasinsaurralde deleted the matias/pfm-7978-cas-cap-upstream-streams branch October 21, 2026 22:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants