Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/content/docs-core/en/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ A change in Core reaches every product that uses it, so each change follows the
Clients such as Claude Code and Codex send their requests to the gateway, and Core provides the following:

- **Rule-based routing.** Rules match on the model, the gateway key, the input size, the presence of tools or images and other properties of a request, and send it to an upstream or a group, rewrite its parameters or refuse it. When the client and the upstream use different API formats, the request is converted between Anthropic Messages, OpenAI Chat Completions, OpenAI Responses and Gemini.
- **Failover before the first byte.** Until the first byte reaches the client, a failing upstream is replaced by the next one without the client noticing. After that point, the failure is reported. A circuit breaker keeps requests away from an upstream that keeps failing.
- **Failover before the first byte.** Until the first byte reaches the client, a failing upstream is replaced by the next one without the client noticing, and when every upstream fails only for the moment, the request is sent again after a short pause, a single upstream included. After that point, the failure is reported. A circuit breaker keeps requests away from an upstream that keeps failing.
- **Cost accounting.** Token usage and cache hits are priced from a public price table, which the control plane refreshes daily, or from a price sheet in the configuration. Each request records its cost and where the price came from. Estimated amounts are marked as such, and usage that cannot be priced is labelled *unknown* rather than given an invented figure.
- **Outbound redaction** (Secrets & privacy in ThinkWatch Lite). Credentials and personal information in a request are replaced with placeholders before the request leaves, and restored exactly as written when the model echoes them back, tool-call arguments included; a rule can instead only record what it finds. Field names are not searched, and custom rules also skip the ids, types, roles and names a protocol needs as they are.
- **Tool-call inspection** (Dangerous commands in ThinkWatch Lite). Tool calls returned by an upstream are checked against a rule set, and a dangerous call can be cut off mid-stream or only recorded.
Expand Down
2 changes: 1 addition & 1 deletion src/content/docs-core/zh-CN/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Core 中的改动会影响所有使用它的产品,因此每个改动都须遵
Claude Code、Codex 等客户端把请求发往网关后,Core 提供以下功能:

- **按规则路由。** 规则按模型、网关密钥、输入规模、是否携带工具或图片等请求属性匹配,将请求发往某个上游或策略组、改写其参数,或拒绝请求。客户端与上游的接口格式不同时,请求在 Anthropic Messages、OpenAI Chat Completions、OpenAI Responses 与 Gemini 之间转换。
- **首字节前的故障转移。** 首字节到达客户端之前,出错的上游由下一个上游替换,客户端无从察觉;此后发生的故障如实报告。熔断器使持续出错的上游暂不接收请求。
- **首字节前的故障转移。** 首字节到达客户端之前,出错的上游由下一个上游替换,客户端无从察觉;所有上游都只是暂时出错时,请求稍候重发,只有一个上游时也是如此。此后发生的故障如实报告。熔断器使持续出错的上游暂不接收请求。
- **费用核算。** token 用量与缓存命中按公开价目表计价(控制面每天刷新一次),或按配置中的价目表计价;每个请求都记录费用及价格来源。估算的金额另行标注,无法计价的用量标记为「未知」,不会填入虚构的数值。
- **出站脱敏**(ThinkWatch Lite 中称为「密钥与隐私」)。请求发出之前,其中的凭据和个人信息替换为占位符;模型回显时按原值的写法如实恢复,工具调用参数中的值也是如此。规则也可以只记录、不替换。字段名不在查找之列,自定义规则还会跳过协议要求原样保留的 id、类型、角色和名称。
- **工具调用审查**(ThinkWatch Lite 中称为「危险命令」)。上游返回的工具调用按规则集审查,高危调用可在流式传输中途截断,或只作记录。
Expand Down
1 change: 1 addition & 0 deletions src/content/docs-lite/en/claude-code-other-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ ThinkWatch Lite connects Claude Code to a gateway on the same computer, which fo
- **Balance.** A DeepSeek upstream, and a Kimi upstream on `api.moonshot.cn` or `api.moonshot.ai`, shows the account balance in the Quota / billing column.
- **The /model list** shows gateway models only when their names contain `claude` or `anthropic`. Other models are typed by name, or added as one entry with `ANTHROPIC_CUSTOM_MODEL_OPTION`.
- **Cost.** A rewritten request is priced as the model actually sent.
- **Reasoning effort.** From ThinkWatch Lite 2026.10.29, a rule can set **Reasoning** under **Parameter rewrites**, for example to max, so that the requests Claude Code sends to GLM always use the highest level. Requests with reasoning on are sent with that level, and those without it, such as title generation, are left as they are; when the upstream does not accept the level, the gateway resends once with one it accepts and remembers it, and the request details say what changed and why.
- **Restore…** puts back only the fields the app wrote; model variables added by hand stay in `settings.json`.

Related: [Features](/docs/lite/features/), [Use Codex with Claude, Gemini or a Chat Completions-only relay](/docs/lite/codex-other-models/), [Use Claude Desktop with third-party models](/docs/lite/claude-desktop-third-party-models/).
9 changes: 5 additions & 4 deletions src/content/docs-lite/en/failover-and-load-balancing.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Fail over and balance load across relays and API keys

ThinkWatch Lite turns each relay key into an upstream and puts several upstreams behind a group. A routing rule sends requests to the group, whose strategy sets the order of its members; when one fails before the answer starts, the gateway passes the request to the next without the client seeing the error. A conversation stays with the upstream that answered it while its prompt cache is worth keeping.
ThinkWatch Lite turns each relay key into an upstream and puts several upstreams behind a group. A routing rule sends requests to the group, whose strategy sets the order of its members; when one fails before the answer starts, the gateway passes the request to the next without the client seeing the error, and when every upstream fails only for the moment, it tries again shortly. A conversation stays with the upstream that answered it while its prompt cache is worth keeping.

## Before you start

- ThinkWatch Lite, [installed](/lite/#install), with a client connected on the Clients page. This guide follows version 2026.10.26.
- ThinkWatch Lite, [installed](/lite/#install), with a client connected on the Clients page. This guide follows version 2026.10.29.
- The base URL and API keys of each relay.

## Steps
Expand All @@ -26,13 +26,14 @@ ThinkWatch Lite turns each relay key into an upstream and puts several upstreams

A strategy only sets the order; every member remains available for failover.

- **When the next upstream is tried.** Before anything has reached the client, the gateway moves on when the upstream cannot be reached or its credential cannot be read; answers 5xx, 401, 403, 402 or 404; answers 400 or 422 with an error about an insufficient balance, a used-up quota or an unavailable model; in a streamed answer, reports an error before the first content, such as an overload; or sends no content within the no-response timeout. A 429 that reports a used-up quota counts as **Quota used up**; any other 429 is a rate limit, which pauses the upstream instead of counting as a failure, and the request moves on or waits (see **Rate limits**). Other 4xx responses, and the last member's 4xx, go back to the client unchanged.
- **When the next upstream is tried.** Before anything has reached the client, the gateway moves on when the upstream cannot be reached or its credential cannot be read; answers 5xx, 401, 403, 402 or 404; answers 400 or 422 with an error about an insufficient balance, a used-up quota or an unavailable model; in a streamed answer, reports an error before the first content, such as an overload; or sends no content within the no-response timeout. A 429 that reports a used-up quota counts as **Quota used up**; any other 429 is a rate limit, which pauses the upstream instead of counting as a failure, and the request moves on or waits (see **Rate limits**). A 400 about the reasoning effort is first sent once more to the same upstream with a level it accepts (see [Features](/docs/lite/features/#routing-and-failover)). Other 4xx responses, and the last member's 4xx, go back to the client unchanged.
- **Retrying after a temporary failure.** When every member has been tried and the last attempt failed temporarily before the answer began, the request is sent again after a wait instead of the error going to the client. A temporary failure is an overload, a 500, 502, 503, 504 or 529, an overload or server error reported in a streamed answer before its content, an upstream that cannot be reached, or a connection or stream that breaks or ends before any content. The waits are 2, 4 and 8 seconds, then 16 seconds each time, each varied by up to a fifth; a longer wait asked for in `retry-after-ms`, `Retry-After` or the error body is used instead. Each retry goes to the members that failed temporarily in the previous round, a rule with a single upstream retries it as well, and a conversation that stays on an upstream retries only that one. Rate limits, used-up quotas, an insufficient balance, rejected credentials, missing models, errors in the request itself and no-response timeouts are not retried, and an answer that has already started is never sent again. Retrying stops when the next wait would end after **Wait at most**, and the client then receives the last error. On the Traffic page a request waiting to be sent again is **Retrying**, and its **Routing** tab lists each retry as an attempt, such as **Retry 1** with **Waited 2.1 s**.
- **Weights and distribution.** In a **Round robin** group, a member's weight, from 1 to 100, is its long-run share of requests: weights 7 and 3 send seven requests in ten to the first, interleaved with the other three. **By ratio** uses the weights alone; **By speed** multiplies each by how much faster than the members' median the upstream starts answering (time to first token, squared, kept between 0.1 and 10), **By reliability** by its success rate over its last 50 outcomes within 30 minutes (squared, at least 0.05), and **By speed and reliability** by both. An upstream with too few samples counts as average, requests that stay with an upstream for their conversation count toward its share, and paused or full members sit out.
- **No response.** An upstream that sends no content for the **No-response timeout** in Settings › Failover, 300 seconds by default (30 to 3,600), is given up on. Only content counts: text, reasoning or a tool call, not keep-alive pings. The clock starts when the request is sent to that upstream, so time spent waiting before it is sent does not count, and starts again with every piece of content. Before anything has reached the client, the request moves to the next upstream and the silent one counts as a failure toward its pause; when none is left, the client receives a timeout error in its own format. Once the answer has started, it ends with an error event the client understands. A whole, non-streamed answer has the same time to arrive. A streamed answer is held back for at most 15 seconds; after that the client receives the response headers and keep-alive comments (Gemini clients get the headers only), so it does not give up while the gateway can still move to the next upstream. The silent upstream's time to first token is recorded as the full wait, which ranks it lower for **Lowest latency** and **By speed**. The request's **Routing** tab marks the attempt **No response · 300 s** with the input the upstream may have billed for it. A model that thinks for a long time without sending anything may need a longer timeout.
- **Concurrency limits.** An upstream with a **Concurrency limit** takes at most that many requests at once. When it is full, a conversation that stays on it waits for it, and other requests go straight to the next member; how long a request waits is described under **Waiting**.
- **Rate limits.** An upstream that answers 429 is paused for the time it asks for, given in a `retry-after-ms` or `Retry-After` header or in the error body. When it gives none, the pause lasts 5, 10, 20 and 40 seconds, then 60 seconds for each further 429 in a row; any other answer starts the count over. **Rate limit** in Settings › Failover caps the pause, 60 minutes by default. A rate limit is not a failure: it does not count toward **Consecutive failures** or toward the success rate used by **By reliability**. The request that received the 429 goes to the next member when one can take it at once, and otherwise waits for the pause to end; the client sees nothing of the 429. When the pause ends, one request goes first, and the others follow once it gets a normal answer. A rule with a single upstream waits for it as well. The Upstreams page tags a paused upstream **Rate limited · back in 42 s**, and the request's **Routing** tab marks the attempt **429 · Paused 30 s**.
- **Usage limits.** **Usage limits** under **Advanced settings** in the upstream dialog (for a ChatGPT account, below its **Concurrency limit**) keep an upstream within the quota of the relay or plan behind it. Each caps the requests, tokens or cost in USD per minute, hour, day, week or month; a minute or hour limit can also span several, from 1 to 60 minutes or 1 to 24 hours, such as 10,000,000 tokens every 5 minutes. When a minute or hour limit is full, the upstream is passed over like a full one: a conversation that stays on it waits for room, and other requests go to the next member. When a day, week or month limit is used up, requests go to the next member until it resets, at local midnight, on Monday or on the 1st; none waits for it. Every request sent to the upstream counts, except one it answered with 429. Tokens and cost come from the answer: tokens are uncached input, cache writes and output, and **Count cache reads** adds cache reads to a token limit; a free upstream, and a model without a price, count as $0 toward a cost limit. Being at a limit is not a failure. The Upstreams page tags such an upstream **Usage limit reached · back in 1 min 10 s**, or **Daily usage limit reached** (weekly, monthly) for a used-up period, and hovering over the tag or the **Quota / billing** cell lists each limit with its use. After a restart, day, week and month limits are added up again from the request records, which must therefore be kept for at least 1, 7 or 31 days; minute and hour limits start empty.
- **Waiting.** A request waits for upstreams that cannot take it yet, because they are full, rate limited or at a minute or hour usage limit, for at most **Wait at most** in Settings › Failover: 120 seconds by default, from 0 to 600, with 0 for no wait. The same time also covers a key's per-minute and per-hour usage limits. A conversation that stays on an upstream for its prompt cache waits for it when it can recover within that time, and otherwise moves on, where its cache starts over; any other request goes to the next member at once. When no member is left, the request waits for the first one that becomes available. A streamed request receives the response headers and keep-alive comments meanwhile (Gemini clients get the headers only), so the client sees only a slower first token. If none becomes available in time, the client receives a 429 with `Retry-After` set to when the first one will, or the error of an earlier attempt when one was sent. When every member has used up a day, week or month limit, `Retry-After` gives the earliest reset and the response tells the client not to retry before then. On the Traffic page, a waiting request is **Queued**, with the upstreams it waits for in place of the upstream and, on hover, how long it waits at most; its **Routing** tab gives each wait, such as **Queued 12 s**.
- **Waiting.** A request waits for upstreams that cannot take it yet, because they are full, rate limited or at a minute or hour usage limit, for at most **Wait at most** in Settings › Failover: 120 seconds by default, from 0 to 600, with 0 for neither waiting nor retrying. The same time also covers a key's per-minute and per-hour usage limits and the waits before retrying a temporary failure. A conversation that stays on an upstream for its prompt cache waits for it when it can recover within that time, and otherwise moves on, where its cache starts over; any other request goes to the next member at once. When no member is left, the request waits for the first one that becomes available. A streamed request receives the response headers and keep-alive comments meanwhile (Gemini clients get the headers only), so the client sees only a slower first token. If none becomes available in time, the client receives a 429 with `Retry-After` set to when the first one will, or the error of an earlier attempt when one was sent. When every member has used up a day, week or month limit, `Retry-After` gives the earliest reset and the response tells the client not to retry before then. On the Traffic page, a waiting request is **Queued**, with the upstreams it waits for in place of the upstream and, on hover, how long it waits at most; its **Routing** tab gives each wait, such as **Queued 12 s**.
- **Pauses.** A failing upstream is paused for as long as Settings › Failover sets: by default 60 seconds after 3 **Consecutive failures**, doubling up to 600; 30 minutes for **Insufficient balance**; until the reset time, or 60 minutes, for **Quota used up**. A rule with a single upstream is never held back by these pauses, and when every member is paused they are tried anyway. A request aborted on the Traffic page does not count against its upstream.
- **Sessions and prompt cache.** Within a turn, while the client sends tool results back, requests keep the rule chosen at the start of the turn and the upstream that answered. Across turns, a conversation stays with the upstream that answered last if that answer read or wrote at least 1,024 cached tokens within the last five minutes; moving would rebuild the cache at full price. Otherwise, or when that upstream is paused after failures, is rate limited past the wait limit, or has used up a day, week or month limit, the strategy orders the members again, which is when **Round robin** moves on. A conversation carrying history that an OpenAI Responses upstream encrypted, such as a ChatGPT account's, goes back to that upstream whatever the rule now picks, since no other upstream can read it; when only reasoning is encrypted, that upstream is tried first and the others remain as fallbacks. The **Conversation** line on a request's **Routing** tab shows when a request stayed.
- **Same model name.** Every member is asked for the model the client sent, or the name a rule rewrote it to. A member whose model list lacks it is skipped; one without a list is tried, and its 404 moves the request on. A model that a member serves but leaves out of its list can be added with **Add models…** in the member's model list on the Upstreams page, or in the **Models** section of its edit dialog.
Expand Down
Loading
Loading