Skip to content

fix(mcp): drop X-Glean-Internal-Service header from gateway requests (fixes 403 Forbidden on setup) - #40

Closed
zh-ffmc wants to merge 1 commit into
gleanwork:mainfrom
zh-ffmc:fix/403-internal-service-header
Closed

zh-ffmc wants to merge 1 commit into
gleanwork:mainfrom
zh-ffmc:fix/403-internal-service-header

Conversation

@zh-ffmc

@zh-ffmc zh-ffmc commented Oct 28, 2026 •

Copy link
Copy Markdown

Problem

The Glean plugin fails setup with HTTP 403 Forbidden on every instance, before any sign-in prompt. Users cannot connect at all on 3.4.2 / 3.5.1 (reported on a production instance; plugin logs show connect.backend-error ... 403 Forbidden immediately after setup.configured).

Cause

buildTransport() in shared/glean/mcp/src/remote-client.ts attaches X-Glean-Internal-Service: "true" to every request it sends to the public MCP gateway (https://<instance>.glean.com/mcp/gateway/proxy). Glean's public edge (GCP LB) refuses any external request carrying that internal-service marker with a bare 403 Forbidden HTML page — no X-Glean-Trace-Id, i.e. the request never reaches the Glean app, and the OAuth flow never starts.

Live reproduction against a real instance (identical MCP initialize POST):

request status
without the header 401 + Www-Authenticate: Bearer resource_metadata=... (correct auth challenge)
with X-Glean-Internal-Service: true 403 Forbidden (edge HTML page)
browser User-Agent + the header 403 (User-Agent is irrelevant — the header is the trigger)
X-Glean-Gateway-Request-Metadata only 401 (attribution header is fine)

Fix

Remove the header from buildTransport() and leave a comment so it isn't re-added. Attribution via X-Glean-Gateway-Request-Metadata is untouched. Verified end-to-end against a live instance with the patched bundle: 401 → PRM/ASM discovery 200 → /oauth/register 201 → /oauth/token 200 → gateway initialize 200, setup completes and all remote tools list.

Tests

New regression test captures the transport's outgoing request headers (same fetch-bridge pattern as the existing elicitation test) and asserts the internal-service marker is absent while gateway metadata remains. Proven failing before the source change, passing after. Full bundle suite: 435/435 passed, typecheck:bundle clean.

If this header is required for some internal (non-public-ingress) path, the correct home for it is a host-internal build flag — sending it from the public client is what's broken today.

buildTransport() sends X-Glean-Internal-Service: true on every request to
the public MCP gateway (/mcp/gateway/proxy). Glean's public edge refuses
any external request carrying that internal-service marker with a bare
HTTP 403 before auth, so plugin setup fails before the OAuth sign-in
prompt is ever reached - on every instance.

Reproduced live (plugin 3.4.2 and 3.5.1, affected since the header's
introduction):

  POST /mcp/gateway/proxy  (MCP initialize, NO header)
    -> 401 + Www-Authenticate: Bearer resource_metadata=...  (correct)
  same POST with X-Glean-Internal-Service: true
    -> 403 Forbidden (GCP LB HTML page; no X-Glean-Trace-Id, request
       never reaches the Glean app)
  same POST with browser User-Agent + the header
    -> 403  (User-Agent is irrelevant; the header is the trigger)

Removing only the header restores the full flow against a live instance:
401 -> OAuth discovery (PRM/ASM 200) -> dynamic registration 201 ->
token 200 -> gateway initialize 200 -> tools enabled.

The X-Glean-Gateway-Request-Metadata attribution header is untouched.
Adds a regression test that captures the transport's request headers and
asserts the internal-service marker is absent while gateway metadata
remains (fails before this change, passes after). Full bundle suite:
435/435 green; typecheck clean.
@zh-ffmc
zh-ffmc requested a review from a team as a code owner October 28, 2026 09:57
@steve-calvert-glean

Copy link
Copy Markdown
Contributor

Thank you for tracking this down. The live reproduction table made the cause clear right away.

We're closing this in favor of #41, which fixes the same 403. It also removes X-Glean-Request-ScParams, which Glean's edge rejects the same way, so a server URL with ?sc= would still fail with only this change. #41 is merged and goes out in the next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants