Repository navigation
fix(cas): cap upstream streams and read-ahead on the shared endpoint - #3692
Conversation
The shared CAS endpoint proxies every tenant's native gRPC through one net/http HTTP/2 client to the private server, which advertises no MaxConcurrentStreams. The client therefore funnels all streams onto a single upstream connection whose ~1 GiB flow-control window is a shared, exhaustible resource: a stalled-reader flood exhausts it and wedges gRPC for every tenant on the endpoint, reachable unauthenticated via reflection or with any downloader token via ByteStream/Read (M1). Cap the private server at 100 concurrent streams so the client spreads streams across several upstream connections, each with its own window, and bound the proxy transport's per-stream read-ahead to 256 KiB so a stalled reader pins far less already-received data (M2). The stats handler is folded into the single grpc.Options call since kratos's grpc.Options assigns rather than appends. Refs PFM-7978. Chainloop-Trace-Sessions: 009eae12-7fed-495c-8131-8171a788bebd Signed-off-by: Matías Insaurralde <matias@chainloop.dev>
PR validation — ✅ 3 passing
AI Session Checks — 🟡 82% · ✅ 0 failing
|
| Status | Attribution | File | Lines |
|---|---|---|---|
| modified | ai | app/artifact-cas/internal/server/grpc.go |
+29 / -4 |
| modified | ai | app/artifact-cas/internal/server/http.go |
+12 / -0 |
Policies (4)
| Status | Policy | Material | Messages |
|---|---|---|---|
| ✅ Passed | ai-config-ai-agents-allowed |
ai-coding-session-009eae |
- |
| ✅ Passed | ai-config-no-dangerous-commands |
ai-coding-session-009eae |
- |
| ✅ Passed | ai-config-no-secrets |
ai-coding-session-009eae |
- |
| ✅ Passed | ai-config-mcp-servers-allowed |
ai-coding-session-009eae |
- |
Security Checks — ✅ 5 passing
✅ secret-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | secrets-detection |
- |
✅ sast-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | owasp-top10-2025 |
- |
| ✅ Passed | sast |
- |
| ✅ Passed | cwe-top25 |
- |
| ✅ Passed | cwe-top26-40-cusp |
- |
Scans not applied (3)
| Scan | Reason |
|---|---|
vulnerability-scan |
no manifest/lockfile changed |
github-actions-scan |
no workflow files changed |
iac-scan |
no IaC files changed |
Security context
✅ Nothing this change touches has a recorded security-fix history.
View in Chainloop ↗ · How this works ↗
Powered by Chainloop and Chainloop Trace
Summary
The shared CAS endpoint proxies every tenant's native gRPC through a single
net/httpHTTP/2 client to the private gRPC server. That server advertises noMaxConcurrentStreams, so the client funnels all streams onto one upstream connection whose ~1 GiB HTTP/2 flow-control window becomes a shared, exhaustible resource: a client that opens many streams and stops reading can exhaust the window and stall gRPC responses for every tenant on the endpoint. It is reachable unauthenticated via gRPC reflection, or with any valid downloader token viaByteStream/Read.This change caps the private server at 100 concurrent streams, so the proxy's HTTP/2 client spreads streams across several upstream connections — each with its own flow-control window — instead of exhausting one shared window. It also bounds the proxy transport's per-stream read-ahead to 256 KiB so a stalled reader pins far less already-received data. The otel stats handler is folded into the single
grpc.Optionscall, since kratos'sgrpc.Optionsreplaces rather than appends.Only the private loopback server behind the proxy is affected; the legacy direct gRPC listener is unchanged.