diff --git a/src/content/docs-core/en/overview.md b/src/content/docs-core/en/overview.md index 72f535d..e4709da 100644 --- a/src/content/docs-core/en/overview.md +++ b/src/content/docs-core/en/overview.md @@ -13,7 +13,7 @@ A change in Core reaches every product that uses it, so each change follows the Clients such as Claude Code and Codex send their requests to the gateway, and Core provides the following: - **Rule-based routing.** Rules match on the model, the gateway key, the input size, the presence of tools or images and other properties of a request, and send it to an upstream or a group, rewrite its parameters or refuse it. When the client and the upstream use different API formats, the request is converted between Anthropic Messages, OpenAI Chat Completions, OpenAI Responses and Gemini. -- **Failover before the first byte.** Until the first byte reaches the client, a failing upstream is replaced by the next one without the client noticing. After that point, the failure is reported. A circuit breaker keeps requests away from an upstream that keeps failing. +- **Failover before the first byte.** Until the first byte reaches the client, a failing upstream is replaced by the next one without the client noticing, and when every upstream fails only for the moment, the request is sent again after a short pause, a single upstream included. After that point, the failure is reported. A circuit breaker keeps requests away from an upstream that keeps failing. - **Cost accounting.** Token usage and cache hits are priced from a public price table, which the control plane refreshes daily, or from a price sheet in the configuration. Each request records its cost and where the price came from. Estimated amounts are marked as such, and usage that cannot be priced is labelled *unknown* rather than given an invented figure. - **Outbound redaction** (Secrets & privacy in ThinkWatch Lite). Credentials and personal information in a request are replaced with placeholders before the request leaves, and restored exactly as written when the model echoes them back, tool-call arguments included; a rule can instead only record what it finds. Field names are not searched, and custom rules also skip the ids, types, roles and names a protocol needs as they are. - **Tool-call inspection** (Dangerous commands in ThinkWatch Lite). Tool calls returned by an upstream are checked against a rule set, and a dangerous call can be cut off mid-stream or only recorded. diff --git a/src/content/docs-core/zh-CN/overview.md b/src/content/docs-core/zh-CN/overview.md index be978a3..d871f50 100644 --- a/src/content/docs-core/zh-CN/overview.md +++ b/src/content/docs-core/zh-CN/overview.md @@ -13,7 +13,7 @@ Core 中的改动会影响所有使用它的产品，因此每个改动都须遵 Claude Code、Codex 等客户端把请求发往网关后，Core 提供以下功能： - **按规则路由。** 规则按模型、网关密钥、输入规模、是否携带工具或图片等请求属性匹配，将请求发往某个上游或策略组、改写其参数，或拒绝请求。客户端与上游的接口格式不同时，请求在 Anthropic Messages、OpenAI Chat Completions、OpenAI Responses 与 Gemini 之间转换。 -- **首字节前的故障转移。** 首字节到达客户端之前，出错的上游由下一个上游替换，客户端无从察觉；此后发生的故障如实报告。熔断器使持续出错的上游暂不接收请求。 +- **首字节前的故障转移。** 首字节到达客户端之前，出错的上游由下一个上游替换，客户端无从察觉；所有上游都只是暂时出错时，请求稍候重发，只有一个上游时也是如此。此后发生的故障如实报告。熔断器使持续出错的上游暂不接收请求。 - **费用核算。** token 用量与缓存命中按公开价目表计价（控制面每天刷新一次），或按配置中的价目表计价；每个请求都记录费用及价格来源。估算的金额另行标注，无法计价的用量标记为「未知」，不会填入虚构的数值。 - **出站脱敏**（ThinkWatch Lite 中称为「密钥与隐私」）。请求发出之前，其中的凭据和个人信息替换为占位符；模型回显时按原值的写法如实恢复，工具调用参数中的值也是如此。规则也可以只记录、不替换。字段名不在查找之列，自定义规则还会跳过协议要求原样保留的 id、类型、角色和名称。 - **工具调用审查**（ThinkWatch Lite 中称为「危险命令」）。上游返回的工具调用按规则集审查，高危调用可在流式传输中途截断，或只作记录。 diff --git a/src/content/docs-lite/en/claude-code-other-models.md b/src/content/docs-lite/en/claude-code-other-models.md index 7bb459a..351db9e 100644 --- a/src/content/docs-lite/en/claude-code-other-models.md +++ b/src/content/docs-lite/en/claude-code-other-models.md @@ -38,6 +38,7 @@ ThinkWatch Lite connects Claude Code to a gateway on the same computer, which fo - **Balance.** A DeepSeek upstream, and a Kimi upstream on `api.moonshot.cn` or `api.moonshot.ai`, shows the account balance in the Quota / billing column. - **The /model list** shows gateway models only when their names contain `claude` or `anthropic`. Other models are typed by name, or added as one entry with `ANTHROPIC_CUSTOM_MODEL_OPTION`. - **Cost.** A rewritten request is priced as the model actually sent. +- **Reasoning effort.** From ThinkWatch Lite 2026.10.29, a rule can set **Reasoning** under **Parameter rewrites**, for example to max, so that the requests Claude Code sends to GLM always use the highest level. Requests with reasoning on are sent with that level, and those without it, such as title generation, are left as they are; when the upstream does not accept the level, the gateway resends once with one it accepts and remembers it, and the request details say what changed and why. - **Restore…** puts back only the fields the app wrote; model variables added by hand stay in `settings.json`. Related: [Features](/docs/lite/features/), [Use Codex with Claude, Gemini or a Chat Completions-only relay](/docs/lite/codex-other-models/), [Use Claude Desktop with third-party models](/docs/lite/claude-desktop-third-party-models/). diff --git a/src/content/docs-lite/en/failover-and-load-balancing.md b/src/content/docs-lite/en/failover-and-load-balancing.md index 94322d2..9a209e2 100644 --- a/src/content/docs-lite/en/failover-and-load-balancing.md +++ b/src/content/docs-lite/en/failover-and-load-balancing.md @@ -1,10 +1,10 @@ # Fail over and balance load across relays and API keys -ThinkWatch Lite turns each relay key into an upstream and puts several upstreams behind a group. A routing rule sends requests to the group, whose strategy sets the order of its members; when one fails before the answer starts, the gateway passes the request to the next without the client seeing the error. A conversation stays with the upstream that answered it while its prompt cache is worth keeping. +ThinkWatch Lite turns each relay key into an upstream and puts several upstreams behind a group. A routing rule sends requests to the group, whose strategy sets the order of its members; when one fails before the answer starts, the gateway passes the request to the next without the client seeing the error, and when every upstream fails only for the moment, it tries again shortly. A conversation stays with the upstream that answered it while its prompt cache is worth keeping. ## Before you start -- ThinkWatch Lite, [installed](/lite/#install), with a client connected on the Clients page. This guide follows version 2026.10.26. +- ThinkWatch Lite, [installed](/lite/#install), with a client connected on the Clients page. This guide follows version 2026.10.29. - The base URL and API keys of each relay. ## Steps @@ -26,13 +26,14 @@ ThinkWatch Lite turns each relay key into an upstream and puts several upstreams A strategy only sets the order; every member remains available for failover. -- **When the next upstream is tried.** Before anything has reached the client, the gateway moves on when the upstream cannot be reached or its credential cannot be read; answers 5xx, 401, 403, 402 or 404; answers 400 or 422 with an error about an insufficient balance, a used-up quota or an unavailable model; in a streamed answer, reports an error before the first content, such as an overload; or sends no content within the no-response timeout. A 429 that reports a used-up quota counts as **Quota used up**; any other 429 is a rate limit, which pauses the upstream instead of counting as a failure, and the request moves on or waits (see **Rate limits**). Other 4xx responses, and the last member's 4xx, go back to the client unchanged. +- **When the next upstream is tried.** Before anything has reached the client, the gateway moves on when the upstream cannot be reached or its credential cannot be read; answers 5xx, 401, 403, 402 or 404; answers 400 or 422 with an error about an insufficient balance, a used-up quota or an unavailable model; in a streamed answer, reports an error before the first content, such as an overload; or sends no content within the no-response timeout. A 429 that reports a used-up quota counts as **Quota used up**; any other 429 is a rate limit, which pauses the upstream instead of counting as a failure, and the request moves on or waits (see **Rate limits**). A 400 about the reasoning effort is first sent once more to the same upstream with a level it accepts (see [Features](/docs/lite/features/#routing-and-failover)). Other 4xx responses, and the last member's 4xx, go back to the client unchanged. +- **Retrying after a temporary failure.** When every member has been tried and the last attempt failed temporarily before the answer began, the request is sent again after a wait instead of the error going to the client. A temporary failure is an overload, a 500, 502, 503, 504 or 529, an overload or server error reported in a streamed answer before its content, an upstream that cannot be reached, or a connection or stream that breaks or ends before any content. The waits are 2, 4 and 8 seconds, then 16 seconds each time, each varied by up to a fifth; a longer wait asked for in `retry-after-ms`, `Retry-After` or the error body is used instead. Each retry goes to the members that failed temporarily in the previous round, a rule with a single upstream retries it as well, and a conversation that stays on an upstream retries only that one. Rate limits, used-up quotas, an insufficient balance, rejected credentials, missing models, errors in the request itself and no-response timeouts are not retried, and an answer that has already started is never sent again. Retrying stops when the next wait would end after **Wait at most**, and the client then receives the last error. On the Traffic page a request waiting to be sent again is **Retrying**, and its **Routing** tab lists each retry as an attempt, such as **Retry 1** with **Waited 2.1 s**. - **Weights and distribution.** In a **Round robin** group, a member's weight, from 1 to 100, is its long-run share of requests: weights 7 and 3 send seven requests in ten to the first, interleaved with the other three. **By ratio** uses the weights alone; **By speed** multiplies each by how much faster than the members' median the upstream starts answering (time to first token, squared, kept between 0.1 and 10), **By reliability** by its success rate over its last 50 outcomes within 30 minutes (squared, at least 0.05), and **By speed and reliability** by both. An upstream with too few samples counts as average, requests that stay with an upstream for their conversation count toward its share, and paused or full members sit out. - **No response.** An upstream that sends no content for the **No-response timeout** in Settings › Failover, 300 seconds by default (30 to 3,600), is given up on. Only content counts: text, reasoning or a tool call, not keep-alive pings. The clock starts when the request is sent to that upstream, so time spent waiting before it is sent does not count, and starts again with every piece of content. Before anything has reached the client, the request moves to the next upstream and the silent one counts as a failure toward its pause; when none is left, the client receives a timeout error in its own format. Once the answer has started, it ends with an error event the client understands. A whole, non-streamed answer has the same time to arrive. A streamed answer is held back for at most 15 seconds; after that the client receives the response headers and keep-alive comments (Gemini clients get the headers only), so it does not give up while the gateway can still move to the next upstream. The silent upstream's time to first token is recorded as the full wait, which ranks it lower for **Lowest latency** and **By speed**. The request's **Routing** tab marks the attempt **No response · 300 s** with the input the upstream may have billed for it. A model that thinks for a long time without sending anything may need a longer timeout. - **Concurrency limits.** An upstream with a **Concurrency limit** takes at most that many requests at once. When it is full, a conversation that stays on it waits for it, and other requests go straight to the next member; how long a request waits is described under **Waiting**. - **Rate limits.** An upstream that answers 429 is paused for the time it asks for, given in a `retry-after-ms` or `Retry-After` header or in the error body. When it gives none, the pause lasts 5, 10, 20 and 40 seconds, then 60 seconds for each further 429 in a row; any other answer starts the count over. **Rate limit** in Settings › Failover caps the pause, 60 minutes by default. A rate limit is not a failure: it does not count toward **Consecutive failures** or toward the success rate used by **By reliability**. The request that received the 429 goes to the next member when one can take it at once, and otherwise waits for the pause to end; the client sees nothing of the 429. When the pause ends, one request goes first, and the others follow once it gets a normal answer. A rule with a single upstream waits for it as well. The Upstreams page tags a paused upstream **Rate limited · back in 42 s**, and the request's **Routing** tab marks the attempt **429 · Paused 30 s**. - **Usage limits.** **Usage limits** under **Advanced settings** in the upstream dialog (for a ChatGPT account, below its **Concurrency limit**) keep an upstream within the quota of the relay or plan behind it. Each caps the requests, tokens or cost in USD per minute, hour, day, week or month; a minute or hour limit can also span several, from 1 to 60 minutes or 1 to 24 hours, such as 10,000,000 tokens every 5 minutes. When a minute or hour limit is full, the upstream is passed over like a full one: a conversation that stays on it waits for room, and other requests go to the next member. When a day, week or month limit is used up, requests go to the next member until it resets, at local midnight, on Monday or on the 1st; none waits for it. Every request sent to the upstream counts, except one it answered with 429. Tokens and cost come from the answer: tokens are uncached input, cache writes and output, and **Count cache reads** adds cache reads to a token limit; a free upstream, and a model without a price, count as $0 toward a cost limit. Being at a limit is not a failure. The Upstreams page tags such an upstream **Usage limit reached · back in 1 min 10 s**, or **Daily usage limit reached** (weekly, monthly) for a used-up period, and hovering over the tag or the **Quota / billing** cell lists each limit with its use. After a restart, day, week and month limits are added up again from the request records, which must therefore be kept for at least 1, 7 or 31 days; minute and hour limits start empty. -- **Waiting.** A request waits for upstreams that cannot take it yet, because they are full, rate limited or at a minute or hour usage limit, for at most **Wait at most** in Settings › Failover: 120 seconds by default, from 0 to 600, with 0 for no wait. The same time also covers a key's per-minute and per-hour usage limits. A conversation that stays on an upstream for its prompt cache waits for it when it can recover within that time, and otherwise moves on, where its cache starts over; any other request goes to the next member at once. When no member is left, the request waits for the first one that becomes available. A streamed request receives the response headers and keep-alive comments meanwhile (Gemini clients get the headers only), so the client sees only a slower first token. If none becomes available in time, the client receives a 429 with `Retry-After` set to when the first one will, or the error of an earlier attempt when one was sent. When every member has used up a day, week or month limit, `Retry-After` gives the earliest reset and the response tells the client not to retry before then. On the Traffic page, a waiting request is **Queued**, with the upstreams it waits for in place of the upstream and, on hover, how long it waits at most; its **Routing** tab gives each wait, such as **Queued 12 s**. +- **Waiting.** A request waits for upstreams that cannot take it yet, because they are full, rate limited or at a minute or hour usage limit, for at most **Wait at most** in Settings › Failover: 120 seconds by default, from 0 to 600, with 0 for neither waiting nor retrying. The same time also covers a key's per-minute and per-hour usage limits and the waits before retrying a temporary failure. A conversation that stays on an upstream for its prompt cache waits for it when it can recover within that time, and otherwise moves on, where its cache starts over; any other request goes to the next member at once. When no member is left, the request waits for the first one that becomes available. A streamed request receives the response headers and keep-alive comments meanwhile (Gemini clients get the headers only), so the client sees only a slower first token. If none becomes available in time, the client receives a 429 with `Retry-After` set to when the first one will, or the error of an earlier attempt when one was sent. When every member has used up a day, week or month limit, `Retry-After` gives the earliest reset and the response tells the client not to retry before then. On the Traffic page, a waiting request is **Queued**, with the upstreams it waits for in place of the upstream and, on hover, how long it waits at most; its **Routing** tab gives each wait, such as **Queued 12 s**. - **Pauses.** A failing upstream is paused for as long as Settings › Failover sets: by default 60 seconds after 3 **Consecutive failures**, doubling up to 600; 30 minutes for **Insufficient balance**; until the reset time, or 60 minutes, for **Quota used up**. A rule with a single upstream is never held back by these pauses, and when every member is paused they are tried anyway. A request aborted on the Traffic page does not count against its upstream. - **Sessions and prompt cache.** Within a turn, while the client sends tool results back, requests keep the rule chosen at the start of the turn and the upstream that answered. Across turns, a conversation stays with the upstream that answered last if that answer read or wrote at least 1,024 cached tokens within the last five minutes; moving would rebuild the cache at full price. Otherwise, or when that upstream is paused after failures, is rate limited past the wait limit, or has used up a day, week or month limit, the strategy orders the members again, which is when **Round robin** moves on. A conversation carrying history that an OpenAI Responses upstream encrypted, such as a ChatGPT account's, goes back to that upstream whatever the rule now picks, since no other upstream can read it; when only reasoning is encrypted, that upstream is tried first and the others remain as fallbacks. The **Conversation** line on a request's **Routing** tab shows when a request stayed. - **Same model name.** Every member is asked for the model the client sent, or the name a rule rewrote it to. A member whose model list lacks it is skipped; one without a list is tried, and its 404 moves the request on. A model that a member serves but leaves out of its list can be added with **Add models…** in the member's model list on the Upstreams page, or in the **Models** section of its edit dialog. diff --git a/src/content/docs-lite/en/faq.md b/src/content/docs-lite/en/faq.md index de4778f..339d4bd 100644 --- a/src/content/docs-lite/en/faq.md +++ b/src/content/docs-lite/en/faq.md @@ -72,7 +72,7 @@ Its gateway can. ThinkWatch Core runs as a systemd service on Linux on x86_64 or ## How is ThinkWatch Lite updated and uninstalled? -ThinkWatch Lite checks for a new version two minutes after it starts and once a day. Installed from the releases page, it updates itself with one press: it verifies the download and waits for the requests in progress to finish before restarting. A Homebrew installation updates through Homebrew instead, with `brew update && brew upgrade --cask thinkwatch-lite`. +ThinkWatch Lite checks for a new version two minutes after it starts and once a day. Installed from the releases page, it updates itself with one press: it verifies the download and restarts. A Homebrew installation updates through Homebrew instead, with `brew update && brew upgrade --cask thinkwatch-lite`. From 2026.10.29, neither interrupts the gateway: the running gateway hands its port over to the new version, finishes the requests in progress and exits. To uninstall, run **Settings › Full uninstall** first, which restores connected clients, turns off launch at login and can delete the data directory; then move the app to the Trash on macOS, or delete the AppImage on Linux or the portable folder on Windows. The Windows installer can also be removed through the system, which does the same restoring first. diff --git a/src/content/docs-lite/en/features.md b/src/content/docs-lite/en/features.md index 2e104f3..68bf11b 100644 --- a/src/content/docs-lite/en/features.md +++ b/src/content/docs-lite/en/features.md @@ -20,9 +20,9 @@ The Traffic page lists requests as they arrive: status, key, model, upstream, ti A request opens into its timeline, its routing (the rule it matched, the group it went through and each attempt with its status and duration), its content, and its usage and cost. The timeline gives the proxy the request went out through (**Egress**, or **Direct**) and how much it sent to its upstreams and received from them (**Upload** and **Download**), measured as it crossed the network: after the gateway's own compression, before decoding, and summed over failover attempts. A request from DeepSeek Harness also shows the size of the session log it carried, the whole conversation the client attaches to every request; the gateway removes it before a request goes to an upstream other than DeepSeek. A finished request can be sent again, unchanged, to another upstream after an estimate of its cost, and the two responses are shown side by side. When the gateway's own code fails during a request, the request's error says what failed and where. -The **Content** tab has two views, **Parsed** and **Raw**, and both are live while the request runs. **Parsed** shows the conversation, with the answer appearing as it streams, thinking and tool-call arguments included. **Raw** shows the request and response lines, headers and bodies, with each streamed event appended as it arrives; **Follow latest** keeps the newest in view. In Raw, **Client side** and **Upstream side** show what the client and the upstream each exchanged with the gateway, which differ when the API format is converted or a plugin rewrites the request. Keys are replaced in both views, as in stored requests. The places a security rule matched are marked in both views, in the colors of the security log, and a folded part that holds one carries a dot. +The **Content** tab has two views, **Parsed** and **Raw**, and both are live while the request runs. **Parsed** shows the conversation, with the answer appearing as it streams, thinking and tool-call arguments included. Answers, thinking and system prompts render their Markdown, so headings, bold, lists, tables and code blocks appear formatted, here and in a session's Conversation tab; HTML in an answer is shown as text, links are not clickable and images are not loaded. **Raw** shows the request and response lines, headers and bodies, with each streamed event appended as it arrives; **Follow latest** keeps the newest in view. In Raw, **Client side** and **Upstream side** show what the client and the upstream each exchanged with the gateway, which differ when the API format is converted or a plugin rewrites the request. Keys are replaced in both views, as in stored requests. The places a security rule matched are marked in both views, in the colors of the security log, and a folded part that holds one carries a dot. -The attempts also show an upstream passed over at its concurrency limit (**At its concurrency limit**), while rate limited (**Rate limited**) or at a usage limit (**Usage limit reached**, or **Usage limit reached for this period** for a day, week or month), an attempt answered with 429 and the pause it caused (**429 · Paused 30 s**), the time spent queued for any of these, an upstream given up because it sent no content within the no-response timeout (**No response · 300 s** at the default), with the input tokens that upstream may have billed for it, and the attempt under way when a request was aborted (**Aborted**). A request that is waiting shows **Queued**, with the upstreams it waits for in place of the upstream. A request refused by its key's usage limit, or not taken because every upstream stayed at its concurrency limit, stayed rate limited, was at a usage limit, or none became available in time, shows **Usage limit**, **At capacity**, **Rate limited**, **Upstream limit** or **Wait timed out** in place of an upstream, and a connection in the Responses API's WebSocket mode is listed one request per turn, each with its own usage and cost. +The attempts also show an upstream passed over at its concurrency limit (**At its concurrency limit**), while rate limited (**Rate limited**) or at a usage limit (**Usage limit reached**, or **Usage limit reached for this period** for a day, week or month), an attempt answered with 429 and the pause it caused (**429 · Paused 30 s**), the time spent queued for any of these, an upstream given up because it sent no content within the no-response timeout (**No response · 300 s** at the default), with the input tokens that upstream may have billed for it, and the attempt under way when a request was aborted (**Aborted**). A retry after an upstream failed temporarily is an attempt of its own, marked **Retry 1** with the wait before it (**Waited 2.1 s**), and an attempt whose reasoning effort a rule changed, or that was resent with another level after the upstream refused one, says which level was asked for, which was sent and why, such as **Reasoning effort max was rejected by the upstream; resent with high**. A request that is waiting shows **Queued**, or **Retrying** while it waits to be sent again, with the upstreams it waits for in place of the upstream. A request refused by its key's usage limit, or not taken because every upstream stayed at its concurrency limit, stayed rate limited, was at a usage limit, or none became available in time, shows **Usage limit**, **At capacity**, **Rate limited**, **Upstream limit** or **Wait timed out** in place of an upstream, and a connection in the Responses API's WebSocket mode is listed one request per turn, each with its own usage and cost. A request in progress can be stopped with **Abort request** in its menu. **Abort session**, in a session's menu or at the top of its panel, stops every request in progress in that session after a confirmation. The upstream call is dropped at once, the client receives an error, and the request is marked **Aborted**, in grey; its upstream is not paused for it. Turns on a Responses API WebSocket connection cannot be aborted. @@ -65,7 +65,7 @@ A relay, a vendor or a ThinkWatch Enterprise console can hand out an import link ## Routing and failover -Each key follows a route, and keys without one follow the default route. A route is a list of rules evaluated in order. A rule matches on the model, the key, the client's API format, input tokens, `max_tokens`, the number of tools, images, extended thinking, streaming, prompt caching, the kind of auxiliary request or the time of day; it then forwards the request to an upstream, a group or specified models, or refuses it, and can rewrite the model, `max_tokens` or extended thinking. Specified models name an upstream and one of its models, with backups tried in order, and send the model as written; to have one client use one model under another model's name, a rule on that client's key does it without changing what the name means for other clients. A model condition written for an upstream model also matches the aliases that list it. The time of day is judged by the local clock of the machine the gateway runs on: pick the weekdays and a start and end time, several windows per rule if needed, and an end earlier than the start is an overnight window. Setting `max_tokens` caps the length of answers: the upstream stops at that point by itself, without an error. A group puts several upstreams behind one name and decides the order in which they are tried: as listed, manually selected, in turn, lowest latency first or lowest cost first. In a round-robin group each member has a weight from 1 to 100, its long-run share of requests, and **Distribute by** can multiply the weights by each upstream's speed (time to first token), its reliability (recent success rate) or both; conversations in progress stay on their upstream. When an attempt fails, the request moves on to the next upstream, and by default a session stays on one upstream so that its prompt cache keeps hitting. An upstream that answers 429 is paused instead of failing, and its requests move on or wait for it. An upstream that sends no content for the **No-response timeout** in Settings › Failover, 300 seconds by default, is given up on; keep-alive pings do not count as content. Before anything has reached the client, the request moves to the next upstream, and when none is left the client receives a timeout error; once the answer has started, it ends with an error. A map at the top of the page traces every key through its route and groups to the upstreams, and requests in flight move along their path from the key to the upstream, each in its own colour. +Each key follows a route, and keys without one follow the default route. A route is a list of rules evaluated in order. A rule matches on the model, the key, the client's API format, input tokens, `max_tokens`, the number of tools, images, extended thinking, streaming, prompt caching, the kind of auxiliary request or the time of day; it then forwards the request to an upstream, a group or specified models, or refuses it, and can rewrite the model, `max_tokens` or the reasoning. **Reasoning** can be **Off** or one of the levels low, medium, high, xhigh and max: a level changes only requests that already have reasoning on, so requests without it, such as title generation, are left as they are and reasoning is never turned on. **Off** is written the way the upstream's format writes it, also when the request is converted to another format: OpenAI Chat Completions and Responses upstreams receive `none`, Gemini 2.5 Flash a thinking budget of 0, and Anthropic and Bedrock upstreams no thinking configuration, while Gemini models that cannot stop thinking (2.5 Pro, Gemini 3) are sent no thinking configuration. When an upstream refuses a reasoning level, the request is sent once more to the same upstream with a level it accepts: a refused `none` is replaced by the lowest level the upstream lists, and the effort is left out only when it lists none. The choice is remembered for that upstream and model for 7 days, so later requests send a working level straight away. A rule saved before 2026.10.29 that turned extended thinking on or off no longer loads, and the app opens in safe mode: **Repair** removes the setting, after which the rule's **Reasoning** can be set again, with **Off** matching the old off. Specified models name an upstream and one of its models, with backups tried in order, and send the model as written; to have one client use one model under another model's name, a rule on that client's key does it without changing what the name means for other clients. A model condition written for an upstream model also matches the aliases that list it. The time of day is judged by the local clock of the machine the gateway runs on: pick the weekdays and a start and end time, several windows per rule if needed, and an end earlier than the start is an overnight window. Setting `max_tokens` caps the length of answers: the upstream stops at that point by itself, without an error. A group puts several upstreams behind one name and decides the order in which they are tried: as listed, manually selected, in turn, lowest latency first or lowest cost first. In a round-robin group each member has a weight from 1 to 100, its long-run share of requests, and **Distribute by** can multiply the weights by each upstream's speed (time to first token), its reliability (recent success rate) or both; conversations in progress stay on their upstream. When an attempt fails, the request moves on to the next upstream, and by default a session stays on one upstream so that its prompt cache keeps hitting. When every upstream has failed temporarily before the answer began, by being overloaded, answering with a server error, being unreachable or dropping the connection, the gateway waits 2, 4 and 8 seconds, then 16 seconds each time, and sends the request again, also when there is only one upstream; a longer wait asked for by the upstream is respected. Retrying and queueing together stay within **Wait at most** in Settings › Failover. An answer that has already started is never sent again, and errors in the request itself, a used-up quota, rejected credentials and rate limits are not retried. An upstream that answers 429 is paused instead of failing, and its requests move on or wait for it. An upstream that sends no content for the **No-response timeout** in Settings › Failover, 300 seconds by default, is given up on; keep-alive pings do not count as content. Before anything has reached the client, the request moves to the next upstream, and when none is left the client receives a timeout error; once the answer has started, it ends with an error. A map at the top of the page traces every key through its route and groups to the upstreams, and requests in flight move along their path from the key to the upstream, each in its own colour. Auxiliary requests that clients send on their own (health checks, warm-ups, titles, topic detection and input suggestions) can be answered locally at no cost, or forwarded. Forwarded ones go through the routing rules like any other request, and a rule can send a given kind to a lower-cost upstream. @@ -105,7 +105,7 @@ Plugins are short JavaScript files that change requests before they go to an ups ## Settings -Settings has eight sections. Connection lists the local core and the saved remote cores, described in [Connecting to a remote core](/docs/lite/remote-core). General sets the language, the appearance, what the menu bar item shows on macOS (or that it is hidden), launch at login, whether notices arrive as system notifications, in the app only or not at all, and shows hidden guidance hints and ignored warnings again. Listening sets who can reach the gateway (this machine only, the local network of a chosen interface, or every interface), its port and the allowed address ranges. Failover sets how long a failing upstream is paused before requests go to the next one: after how many consecutive failures, for how long, and separate pauses for an insufficient balance, a used-up quota and rate limits. It also sets the no-response timeout, after which an upstream that has sent no content is given up on (300 seconds by default, from 30 to 3,600), and **Wait at most**, how long in total a request may wait while upstreams are full, rate limited or at a usage limit, or while a key's per-minute or per-hour limit is used up: 120 seconds by default, after which the request ends with an error. Log retention sets how long request payloads and request records are kept, and a size cap for payloads; the security log expires with the request records. Sync and backup exports and imports the configuration, turns on sync and lists the versions of the configuration that can be restored (see [Sync and backup](#sync-and-backup)). About shows the version, checks for updates and produces a diagnostics bundle with keys and addresses masked. Uninstall restores every connected client and removes the autostart entry, and is meant to be run before the app is deleted. +Settings has eight sections. Connection lists the local core and the saved remote cores, described in [Connecting to a remote core](/docs/lite/remote-core). General sets the language, the appearance, what the menu bar item shows on macOS (or that it is hidden), launch at login, whether notices arrive as system notifications, in the app only or not at all, and shows hidden guidance hints and ignored warnings again. Listening sets who can reach the gateway (this machine only, the local network of a chosen interface, or every interface), its port and the allowed address ranges. Failover sets how long a failing upstream is paused before requests go to the next one: after how many consecutive failures, for how long, and separate pauses for an insufficient balance, a used-up quota and rate limits. It also sets the no-response timeout, after which an upstream that has sent no content is given up on (300 seconds by default, from 30 to 3,600), and **Wait at most**, how long in total a request may spend queueing for a full, rate-limited or usage-limited upstream (or a key's per-minute or per-hour limit) and retrying upstreams that failed temporarily: 120 seconds by default, after which the request ends with an error, and 0 for no queueing or retrying. Log retention sets how long request payloads and request records are kept, and a size cap for payloads; the security log expires with the request records. Sync and backup exports and imports the configuration, turns on sync and lists the versions of the configuration that can be restored (see [Sync and backup](#sync-and-backup)). About shows the version, checks for updates and produces a diagnostics bundle with keys and addresses masked. Uninstall restores every connected client and removes the autostart entry, and is meant to be run before the app is deleted. ## Sync and backup diff --git a/src/content/docs-lite/en/install.md b/src/content/docs-lite/en/install.md index 4a6b90e..d441721 100644 --- a/src/content/docs-lite/en/install.md +++ b/src/content/docs-lite/en/install.md @@ -81,13 +81,13 @@ The app looks for a new version two minutes after it starts and once a day after When the automatic check finds a new version, the app lists it among the notices and posts a system notification. With **Notices** in Settings › General set to **In app only**, only the notice appears; with **Off**, neither does. The notice, the notification, the **Install Version** item in the menu bar or tray menu, and **Update to** in Settings › About open the update window; **Check for updates**, in the same menu or in Settings › About, opens it at once when there is a new version. What happens next depends on how the app was installed. -**Downloaded from the releases page on macOS:** one press on the install button does the rest. The app downloads the update, verifies it against a key compiled into itself, waits for the requests the gateway is serving to finish — up to three minutes — then replaces itself and restarts. A task in the middle of a response is not cut off to make room for the update. +**Downloaded from the releases page on macOS:** one press on the install button does the rest. The app downloads the update, verifies it against a key compiled into itself, then replaces itself and restarts. A task in the middle of a response is not cut off by the update (see below). -**With the Windows installer:** the same single press. The app downloads the new installer, verifies it against the key compiled into itself, waits for the requests in flight to finish in the same way, then runs the installer, and the new version starts once it is done. The app is installed for all users, so Windows asks for administrator permission at every update; declining leaves the current version running. +**With the Windows installer:** the same single press. The app downloads the new installer, verifies it against the key compiled into itself, then runs the installer, and the new version starts once it is done. The app is installed for all users, so Windows asks for administrator permission at every update; declining leaves the current version running. -**The Windows portable copy:** the same single press, and no administrator permission is needed. The app downloads the new zip, verifies it against the key compiled into itself, waits for the requests in flight to finish, then replaces the two program files in its folder and restarts. If replacing them fails, the old files are put back and the current version keeps running. +**The Windows portable copy:** the same single press, and no administrator permission is needed. The app downloads the new zip, verifies it against the key compiled into itself, then replaces the two program files in its folder and restarts. If replacing them fails, the old files are put back and the current version keeps running. -**On Linux:** the same single press, and no password is asked for. The app downloads the new AppImage, verifies it against the key compiled into itself, waits for the requests in flight to finish, then replaces its own file and restarts. The AppImage has to be in a folder the user can write to. +**On Linux:** the same single press, and no password is asked for. The app downloads the new AppImage, verifies it against the key compiled into itself, then replaces its own file and restarts. The AppImage has to be in a folder the user can write to. **Installed with Homebrew:** the window gives the command to copy, and the app never replaces itself. Homebrew records which version it put in `/Applications`; an app that overwrote it would be written back over by the next `brew upgrade`. A Homebrew installation is offered a new version only once the tap carries it, so the command always has something to install: @@ -97,6 +97,8 @@ brew update && brew upgrade --cask thinkwatch-lite `brew update` comes first because `brew upgrade` refreshes taps at most once a day on its own. +However the app was installed, an update does not interrupt the gateway: when the new version starts, the running gateway hands its port over to the new one, finishes the requests in flight and exits, so the port stays open and those requests complete. Meanwhile the requests in progress on the Overview page and in the menu bar include those the previous gateway is still finishing. A Homebrew upgrade is recognised and hands the gateway over in the same way as an in-app update, and so does **Restart Gateway** in the menu. After an update the window comes back as it was before. This applies from 2026.10.29: updating from an earlier version to 2026.10.29 still has the old version wait for the requests in flight to finish, up to three minutes, and then restart the gateway. + ## Next steps - [Overview](/docs/lite): what the app shows. diff --git a/src/content/docs-lite/en/overview.md b/src/content/docs-lite/en/overview.md index b772e5d..6beed84 100644 --- a/src/content/docs-lite/en/overview.md +++ b/src/content/docs-lite/en/overview.md @@ -11,7 +11,7 @@ ThinkWatch Lite is a local gateway for Claude Code, Codex and other AI clients, - **MCP servers, skills and hooks, scanned.** The MCP servers of thirteen clients side by side, and a scan of client configuration for hidden characters, prompt injection, dangerous commands and overly broad permissions. - **Plugins.** Short JavaScript plugins adjust requests and answers, such as asking for answers in a chosen language or converting file paths between WSL and Windows. They run in a sandbox, see placeholders instead of keys, and every change they make is recorded. - **Every request traceable.** The matched rule, each attempt, any format conversion and the cost, with replay against another upstream; the whole history can be searched, including the text of requests and answers. -- **Routing and failover.** Rules by model, tools, images and more; groups that fail over before the answer begins and keep each session on one upstream. +- **Routing and failover.** Rules by model, tools, images and more, which can also set the reasoning effort; groups that fail over before the answer begins, retry brief failures and keep each session on one upstream. - **Any upstream.** API keys, Amazon Bedrock, ChatGPT and Z.ai accounts, relays and local models, with conversion between the Anthropic, OpenAI and Gemini APIs. - **Sync and backup.** The configuration can be exported to one file and imported on a new computer or after a reinstall, or synced between devices through a cloud drive folder, WebDAV or S3-compatible storage, encrypted by default. Every change to it leaves a version that can be restored. - **Costs stated as they are.** Estimates marked, unpriced requests counted separately rather than as zero. diff --git a/src/content/docs-lite/en/plugins.md b/src/content/docs-lite/en/plugins.md index 54ab518..d14768e 100644 --- a/src/content/docs-lite/en/plugins.md +++ b/src/content/docs-lite/en/plugins.md @@ -28,7 +28,7 @@ A request is routed first, on what the client sent: the routing rules, a rule's 3. If a plugin changed the request, the content filter checks it again, hidden-character rules included, and reports only what the plugins added. A block refuses the whole request; it is not tried on another upstream. 4. The request is converted to the upstream's format if needed, Secrets & privacy applies, and it is sent. -When an attempt fails and the request moves to another upstream, the hooks run again from the request as the client sent it, so changes made for one upstream never reach another. A retry to the same upstream, such as the one after a sign-in token is refreshed, reuses what the hooks produced. +When an attempt fails and the request moves to another upstream, the hooks run again from the request as the client sent it, so changes made for one upstream never reach another. A retry after an upstream failed temporarily is a new attempt, and the hooks run again from the client's request as well. A resend to the same upstream, such as the one after a sign-in token is refreshed, reuses what the hooks produced. A plugin that changes the model renames only what is sent to the upstream of this attempt, like a routing rule's rename, and replaces any name a rule set. The request is not routed again and the upstream's model list is not checked again, but the models the key may use still apply: a model outside them refuses the request. diff --git a/src/content/docs-lite/zh-CN/claude-code-other-models.md b/src/content/docs-lite/zh-CN/claude-code-other-models.md index 4931704..b66e2e6 100644 --- a/src/content/docs-lite/zh-CN/claude-code-other-models.md +++ b/src/content/docs-lite/zh-CN/claude-code-other-models.md @@ -38,6 +38,7 @@ ThinkWatch Lite 把 Claude Code 接到本机的网关上，网关再把每个请 - **余额。**DeepSeek 上游，以及地址在 `api.moonshot.cn` 或 `api.moonshot.ai` 上的 Kimi 上游，在「额度 / 计费」列中显示账户余额。 - **`/model` 列表**只显示名称含 `claude` 或 `anthropic` 的网关模型；其他模型需要输入名称，或用 `ANTHROPIC_CUSTOM_MODEL_OPTION` 加入一项。 - **费用。**改写过模型名的请求按实际发出的模型计价。 +- **推理强度。**自 ThinkWatch Lite 2026.10.29 起，规则的「改写参数」中可以设置「推理」，例如设为 max，使 Claude Code 发往 GLM 的请求始终使用最高一档。已开启推理的请求改用这一档，生成标题等未开启推理的请求保持不变；上游不接受这一档时，网关改用它接受的档位重发一次并记住，请求详情中注明改动及原因。 - **「还原…」**只恢复应用写入的字段，手动加入的模型变量仍留在 `settings.json` 中。 相关文档：[功能详解](/zh-CN/docs/lite/features/)、[让 Codex 使用 Claude、Gemini 或只提供 Chat Completions 的中转](/zh-CN/docs/lite/codex-other-models/)、[让 Claude Desktop 使用第三方模型](/zh-CN/docs/lite/claude-desktop-third-party-models/)。 diff --git a/src/content/docs-lite/zh-CN/failover-and-load-balancing.md b/src/content/docs-lite/zh-CN/failover-and-load-balancing.md index 9e62e4d..5e63d1e 100644 --- a/src/content/docs-lite/zh-CN/failover-and-load-balancing.md +++ b/src/content/docs-lite/zh-CN/failover-and-load-balancing.md @@ -1,10 +1,10 @@ # 在多个中转站和多把密钥之间自动故障转移与负载均衡 -ThinkWatch Lite 把每个中转站的每把密钥建成一个上游，再把多个上游放进一个策略组。路由规则把请求转发至策略组，由策略决定成员的先后；某个上游在回答开始之前失败时，网关把请求交给下一个上游，客户端不会看到这次错误。提示缓存值得保留时，同一段对话继续使用上次回答它的上游。 +ThinkWatch Lite 把每个中转站的每把密钥建成一个上游，再把多个上游放进一个策略组。路由规则把请求转发至策略组，由策略决定成员的先后；某个上游在回答开始之前失败时，网关把请求交给下一个上游，客户端不会看到这次错误；所有上游都只是暂时出错时，网关稍候重试。提示缓存值得保留时，同一段对话继续使用上次回答它的上游。 ## 准备 -- 已[安装](/zh-CN/lite/#install) ThinkWatch Lite，并已在客户端页接管客户端。本文按 2026.10.26 版编写。 +- 已[安装](/zh-CN/lite/#install) ThinkWatch Lite，并已在客户端页接管客户端。本文按 2026.10.29 版编写。 - 各中转站的接口地址和 API 密钥。 ## 步骤 @@ -26,13 +26,14 @@ ThinkWatch Lite 把每个中转站的每把密钥建成一个上游，再把多 策略只决定顺序，所有成员都留作故障转移的后备。 -- **何时换到下一个上游。**在尚未向客户端发出任何内容时，上游出现以下情况，网关即改用下一个成员：无法连接或读不到它的凭据；返回 5xx、401、403、402 或 404；返回 400 或 422，且错误信息说明余额不足、额度用完或模型不可用；流式回答在第一段内容之前报错，例如过载；在无响应超时内没有发出内容。说明额度用完的 429 按「额度用完」处理；其他 429 属于限流，上游随之暂停，不计为失败，请求换到下一个成员或等待（见「限流」）。其他 4xx，以及最后一个成员返回的 4xx，原样返回客户端。 +- **何时换到下一个上游。**在尚未向客户端发出任何内容时，上游出现以下情况，网关即改用下一个成员：无法连接或读不到它的凭据；返回 5xx、401、403、402 或 404；返回 400 或 422，且错误信息说明余额不足、额度用完或模型不可用；流式回答在第一段内容之前报错，例如过载；在无响应超时内没有发出内容。说明额度用完的 429 按「额度用完」处理；其他 429 属于限流，上游随之暂停，不计为失败，请求换到下一个成员或等待（见「限流」）。因推理强度返回的 400 先在同一个上游改用它接受的档位重发一次（见[功能详解](/zh-CN/docs/lite/features/#路由与故障转移)）。其他 4xx，以及最后一个成员返回的 4xx，原样返回客户端。 +- **暂时出错后重试。**所有成员都试过、最后一次在回答开始之前暂时出错时，请求不把错误交给客户端，而是等待后再次发送。暂时出错指过载、返回 500、502、503、504 或 529、流式回答在内容之前报告过载或服务器错误、无法连接，以及连接或流在内容之前中断或提前结束。等待依次为 2、4、8 秒，此后每次 16 秒，各有最多两成的浮动；上游以 `retry-after-ms`、`Retry-After` 或错误信息要求等待更久的，按它的要求。每次重试只发给上一轮暂时出错的成员，只有一个上游的规则同样重试；留在某个上游的对话只重试这一个。限流、额度用完、余额不足、凭据被拒、模型不存在、请求本身的错误和无响应超时都不重试，已经开始的回答也不会重发。下一次等待会超出「最多等待」时不再重试，客户端收到最后一次的错误。流量页把等待重试的请求标为「重试中」，请求的「路由」标签中每次重试单列一跳，例如「第 1 次重试」「等待 2.1 秒」。 - **权重与分配依据。**「轮询」策略组中，成员的权重（1 到 100）就是它长期分到的请求比例：权重 7 与 3 时，每十个请求中七个交给前者，与另外三个交错分配。「按比例」只看权重；「按速度」再乘以该上游开始回答比成员中位数快多少（首 token 时间之比的平方，限定在 0.1 到 10 之间）；「按稳定性」再乘以它 30 分钟内最近 50 次结果的成功率（取平方，最低 0.05）；「按速度和稳定性」两者都乘。样本不足的上游按平均水平计算；因对话延续而留在某个上游的请求计入它的份额；暂停中或并发已满的成员本轮不参与。 - **无响应。**上游连续「设置 › 故障转移」中的「无响应超时」（默认 300 秒，可设 30 到 3,600 秒）没有发出内容时，网关放弃这个上游。只有内容才计入：文字、推理或工具调用，保活心跳不算。计时从请求发给这个上游时开始，发出之前的等待不计入，每收到一段内容重新计时。尚未向客户端发出内容时，请求转到下一个上游，没有响应的上游计一次失败，按暂停规则处理；没有可用的上游时，客户端收到其自身格式的超时错误。回答已经开始时，以客户端能识别的错误事件结束。非流式回答同样最多等待这么久。流式回答最多压住 15 秒，此后客户端先收到响应头和保活注释（Gemini 客户端只收到响应头），因此不会在网关仍可转到下一个上游时自行放弃。没有响应的上游，首 token 时间按等待的全部时长记录，「延迟最低」与「按速度」因此把它排到后面。请求的「路由」标签把这次尝试标为「无响应超时 · 300 秒」，并标出上游可能已计费的输入。长时间思考而不发出任何内容的模型，可能需要更长的超时。 - **并发上限。**设置了「并发上限」的上游同时最多接收这么多请求。上游已满时，留在它上面的对话等待它，其他请求直接交给下一个成员；最多等待多久见「等待」。 - **限流。**返回 429 的上游按它要求的时长暂停，时长取自 `retry-after-ms` 或 `Retry-After` 响应头，或错误信息中的等待时间。上游没有给出时，暂停 5、10、20、40 秒，此后每次连续返回 429 暂停 60 秒；收到其他答复即重新计数。暂停最长为「设置 › 故障转移」中的「限流」，默认 60 分钟。限流不算失败：不计入「连续失败」，也不计入「按稳定性」所用的成功率。收到 429 的请求，有成员可以立即接收时交给它，否则等待暂停结束；客户端看不到这次 429。暂停结束时先放行一个请求，它收到正常答复之后，其余请求随后发出。只有一个上游的规则同样等待它恢复。上游页把暂停中的上游标为「限流中 · 42 秒后恢复」，请求的「路由」标签把这次尝试标为「429 · 暂停 30 秒」。 - **用量上限。**上游对话框「高级设置」中的「用量上限」（ChatGPT 账号在「并发上限」下方）让上游不超出中转站或套餐本身的额度。每条上限限定每分钟、小时、天、周或月最多使用的请求次数、token 数或费用（美元）；分钟与小时上限还可以跨越多个单位，可设 1 到 60 分钟或 1 到 24 小时，例如每 5 分钟 10,000,000 token。分钟或小时上限已满时，上游按并发已满处理：留在它上面的对话等待空出用量，其他请求交给下一个成员。天、周或月上限用满后，请求交给下一个成员，直到本地时间零点、周一或每月 1 日重置，不等待它。发给上游的每个请求都计入，上游返回 429 的除外。token 与费用按回答计算：token 为未命中缓存的输入、缓存写入与输出，勾选「计入缓存读取」后 token 上限也计入缓存读取；不计费的上游和没有价格的模型按 0 计入费用上限。达到上限不算失败。上游页把这样的上游标为「用量已满 · 1 分 10 秒后恢复」，用满一整期的标为「今日用量已满」（或本周、本月），悬停在标签或「额度 / 计费」一格上可以查看每条上限的用量。重启之后，天、周、月上限的用量按请求记录重新累计，因此请求记录至少要保留 1、7 或 31 天；分钟与小时上限从零开始。 -- **等待。**上游并发已满、正在限流或达到分钟、小时用量上限而暂时无法接收时，请求最多等待「设置 › 故障转移」中的「最多等待」，默认 120 秒，可设 0 到 600 秒，0 表示不等待。密钥的分钟、小时用量上限共用这段时间。为保留提示缓存而留在某个上游的对话，在它能于这段时间内恢复时等待它，否则换到下一个成员，在那里重新建立缓存；其他请求直接交给下一个成员。没有可用的成员时，请求等待最先恢复的那一个。等待期间，流式请求先收到响应头和保活注释（Gemini 客户端只收到响应头），客户端只会感到首 token 慢了一些。到时仍没有成员可用，客户端收到 429，`Retry-After` 为最早恢复的时刻；此前已有尝试发出时，返回那次尝试的错误。所有成员都用满了天、周或月上限时，`Retry-After` 为最早的重置时刻，并告知客户端在此之前不要重试。流量页把等待中的请求标为「排队中」，在上游的位置列出它等待的上游，悬停可以查看最晚等到何时；请求的「路由」标签写明每次等待的时长，例如「排队 12 秒」。 +- **等待。**上游并发已满、正在限流或达到分钟、小时用量上限而暂时无法接收时，请求最多等待「设置 › 故障转移」中的「最多等待」，默认 120 秒，可设 0 到 600 秒，0 表示既不等待也不重试。密钥的分钟、小时用量上限，以及暂时出错后重试之前的等待，共用这段时间。为保留提示缓存而留在某个上游的对话，在它能于这段时间内恢复时等待它，否则换到下一个成员，在那里重新建立缓存；其他请求直接交给下一个成员。没有可用的成员时，请求等待最先恢复的那一个。等待期间，流式请求先收到响应头和保活注释（Gemini 客户端只收到响应头），客户端只会感到首 token 慢了一些。到时仍没有成员可用，客户端收到 429，`Retry-After` 为最早恢复的时刻；此前已有尝试发出时，返回那次尝试的错误。所有成员都用满了天、周或月上限时，`Retry-After` 为最早的重置时刻，并告知客户端在此之前不要重试。流量页把等待中的请求标为「排队中」，在上游的位置列出它等待的上游，悬停可以查看最晚等到何时；请求的「路由」标签写明每次等待的时长，例如「排队 12 秒」。 - **暂停。**失败的上游按「设置 › 故障转移」暂停使用，默认值为：「连续失败」3 次后暂停 60 秒，此后每次加倍，最长 600 秒；「余额不足」暂停 30 分钟；「额度用完」暂停到重置时刻，未给出时暂停 60 分钟。只有一个上游的规则不受这些暂停影响；所有成员都在暂停时，网关仍会逐个尝试。在流量页手动中止的请求不计入上游的失败。 - **会话与提示缓存。**同一轮之内（客户端回传工具结果期间），请求沿用这一轮开头确定的规则和回答它的上游。跨轮时，如果上次回答在五分钟以内、且读写了至少 1,024 个缓存 token，对话继续使用该上游，因为换到别处要按全价重建缓存；否则，或者该上游因失败而暂停、限流要到等待时限之后才结束、已用满天、周或月上限时，由策略重新排序，「轮询」正是在这时轮到下一个成员。对话带有 OpenAI Responses 类上游（如 ChatGPT 账号）加密的历史时，由于其他上游无法读取，无论规则现在选择哪个上游，对话都送回该上游；加密的只有推理内容时，先尝试该上游，其余成员仍作为备选。请求「路由」标签中的「对话延续」一行说明请求是否因此留在原上游。 - **模型名相同。**每个成员收到的都是客户端请求的模型名，或者规则改写之后的模型名。模型列表中没有该模型的成员会被跳过；没有模型列表的成员照常尝试，它返回 404 时请求换到下一个。成员能服务、却没有列进模型列表的模型，可以在上游页该成员的模型列表中选择「添加模型…」加上，也可以在它的编辑对话框的「模型」一节中添加。 diff --git a/src/content/docs-lite/zh-CN/faq.md b/src/content/docs-lite/zh-CN/faq.md index 9ce93a7..8b0fbc5 100644 --- a/src/content/docs-lite/zh-CN/faq.md +++ b/src/content/docs-lite/zh-CN/faq.md @@ -72,7 +72,7 @@ ThinkWatch Lite 未经 Apple 注册开发者签名，Windows 版本也未经代 ## ThinkWatch Lite 如何更新和卸载？ -ThinkWatch Lite 启动两分钟后检查一次新版本，此后每天检查一次。从 release 页面下载安装的，点击一次即可完成更新：应用校验下载的文件，等待进行中的请求结束后重新启动。通过 Homebrew 安装的，改用 Homebrew 更新：`brew update && brew upgrade --cask thinkwatch-lite`。 +ThinkWatch Lite 启动两分钟后检查一次新版本，此后每天检查一次。从 release 页面下载安装的，点击一次即可完成更新：应用校验下载的文件后重新启动。通过 Homebrew 安装的，改用 Homebrew 更新：`brew update && brew upgrade --cask thinkwatch-lite`。自 2026.10.29 起，两种方式都不中断网关：正在运行的网关把端口交给新版本，处理完进行中的请求后退出。 卸载时先在「设置 › 完全卸载」中卸载，它会还原已接管的客户端、取消开机启动，并可同时删除数据目录；之后在 macOS 上把应用移到废纸篓，在 Linux 上删除 AppImage，在 Windows 绿色版上删除整个文件夹。Windows 安装版也可以通过系统卸载，卸载程序会先完成同样的还原。 diff --git a/src/content/docs-lite/zh-CN/features.md b/src/content/docs-lite/zh-CN/features.md index bf5dd91..3e3a4ac 100644 --- a/src/content/docs-lite/zh-CN/features.md +++ b/src/content/docs-lite/zh-CN/features.md @@ -20,9 +20,9 @@ 打开一个请求可以查看时间线、路由（命中的规则、经过的策略组，以及每一次尝试的状态与耗时）、内容、用量与费用。时间线给出请求经由的代理（「出口」，未经代理时为「直连」），以及请求发往上游与从上游收到的数据量（「上传」与「下载」），按网络上实际传输的大小计算：在网关自身的压缩之后、解码之前，故障转移的各次尝试合计在内。DeepSeek Harness 发出的请求还会显示所带会话日志的大小：这是客户端随每个请求附带的整段对话记录，发往 DeepSeek 以外的上游之前由网关去除。已结束的请求可以在预估费用后原样发送到另一个上游，两次的响应并排对照。网关自身的代码在请求过程中出错时，请求的错误信息写明出错的内容与位置。 -「内容」标签页有「解析」与「原始报文」两种视图，请求进行中时两者都实时更新。「解析」以对话的形式显示，回答随流式输出逐步出现，思考与工具调用的参数同样实时显示。「原始报文」显示请求行与状态行、报文头和正文，流式返回的每个事件到达后依次追加；「跟随最新」使最新的内容保持在视野内。「原始报文」中的「客户端一侧」与「上游一侧」分别显示客户端和上游与网关之间收发的内容，转换 API 格式或插件改写请求时两者不同。两种视图中的密钥与存下的请求一样已被替换。安全规则命中之处在两种视图中都以安全日志的颜色标出，折叠的部分中含有命中时，折叠处带有一个圆点。 +「内容」标签页有「解析」与「原始报文」两种视图，请求进行中时两者都实时更新。「解析」以对话的形式显示，回答随流式输出逐步出现，思考与工具调用的参数同样实时显示。回答、思考与系统提示按 Markdown 排版，标题、粗体、列表、表格与代码块照常显示，会话的「对话」页也是如此；回答中的 HTML 按文本显示，链接不可点击，图片不加载。「原始报文」显示请求行与状态行、报文头和正文，流式返回的每个事件到达后依次追加；「跟随最新」使最新的内容保持在视野内。「原始报文」中的「客户端一侧」与「上游一侧」分别显示客户端和上游与网关之间收发的内容，转换 API 格式或插件改写请求时两者不同。两种视图中的密钥与存下的请求一样已被替换。安全规则命中之处在两种视图中都以安全日志的颜色标出，折叠的部分中含有命中时，折叠处带有一个圆点。 -尝试链还会标出因并发已满、限流或达到用量上限而跳过的上游（「并发已满」「限流中」「用量已满」，天、周、月上限用满时为「本期用量已满」）、返回 429 的尝试及其引起的暂停（「429 · 暂停 30 秒」）、为这些情况排队等待的时间、超过无响应超时仍未发出内容而被放弃的上游（默认值下显示为「无响应超时 · 300 秒」）及该上游可能已计费的输入 token，以及请求被手动中止时正在进行的尝试（「手动中止」）。正在等待的请求显示「排队中」，在上游的位置列出它等待的上游。因密钥的用量上限而被拒绝，或因上游均并发已满、均在限流、均达到用量上限、等待期间没有上游恢复而未被接下的请求，在上游的位置显示「用量上限」「并发已满」「上游限流中」「上游用量已满」或「等待超时」；以 Responses API 的 WebSocket 模式建立的连接按轮记为请求，每一轮单独统计用量与费用。 +尝试链还会标出因并发已满、限流或达到用量上限而跳过的上游（「并发已满」「限流中」「用量已满」，天、周、月上限用满时为「本期用量已满」）、返回 429 的尝试及其引起的暂停（「429 · 暂停 30 秒」）、为这些情况排队等待的时间、超过无响应超时仍未发出内容而被放弃的上游（默认值下显示为「无响应超时 · 300 秒」）及该上游可能已计费的输入 token，以及请求被手动中止时正在进行的尝试（「手动中止」）。上游暂时出错后的重试单列一跳，标为「第 1 次重试」，并注明之前等待的时长（「等待 2.1 秒」）；推理强度被规则改写、或被上游拒收后改用其他档位重发的一跳，注明原来与实际发出的档位及原因，例如「推理强度 max 被上游拒收，已改用 high 重发」。正在等待的请求显示「排队中」，等待重试的请求显示「重试中」，在上游的位置列出它等待的上游。因密钥的用量上限而被拒绝，或因上游均并发已满、均在限流、均达到用量上限、等待期间没有上游恢复而未被接下的请求，在上游的位置显示「用量上限」「并发已满」「上游限流中」「上游用量已满」或「等待超时」；以 Responses API 的 WebSocket 模式建立的连接按轮记为请求，每一轮单独统计用量与费用。 进行中的请求可以在它的操作菜单中选择「中止请求」停止。会话的操作菜单和会话详情顶部的「中止会话」经确认后停止该会话中所有进行中的请求。与上游的连接立即断开，客户端收到错误，请求以灰色标为「手动中止」，上游不会因此被暂停。Responses API 的 WebSocket 连接中的轮次不能中止。 @@ -65,7 +65,7 @@ API 密钥和请求头的值可以写成 `${变量名}`，读取系统环境变 ## 路由与故障转移 -每把密钥使用一条路由，未指定的使用默认路由。路由由按顺序匹配的规则组成。规则的条件包括模型、密钥、客户端的 API 格式、输入 token 数、`max_tokens`、工具数量、图片、扩展思考、流式、提示缓存、辅助请求的类型以及时段；命中后把请求交给某个上游、策略组或指定模型，或拒绝请求，也可以改写模型、`max_tokens` 或扩展思考。指定模型写明上游和它的某个模型，可按顺序列出备用，模型名原样发出；要让某个客户端把一个模型当另一个模型的名称使用，在这个客户端的密钥上写规则即可，不会改变这个名称对其他客户端的含义。条件写上游模型名时，也匹配列有它的别名。时段按网关所在机器的本地时间判断：选择星期与起止时刻即可，一条规则可以写多个时段，结束早于开始的为跨夜时段。设置 `max_tokens` 可以限制回答的长度：上游到达上限时自行停止，不会报错。策略组把多个上游放在同一个名字下，并决定尝试的先后：按顺序、手动选择、轮询、延迟最低优先或费用最低优先。轮询策略组的每个成员有 1 到 100 的权重，即长期分到的请求比例；「分配依据」可以在权重之上再乘以各上游的速度（首 token 时间）、稳定性（最近的成功率）或两者，进行中的对话仍留在原上游。一次尝试失败时，请求转到下一个上游；同一会话默认保持在同一个上游上，以便提示缓存持续命中。返回 429 的上游暂停而不算失败，发给它的请求转到下一个上游或等待它恢复。上游连续「设置 › 故障转移」中的「无响应超时」（默认 300 秒）没有发出内容时，网关放弃这个上游，保活心跳不算内容：尚未向客户端发出内容时，请求转到下一个上游，没有可用的上游时客户端收到超时错误；回答已经开始时，以错误结束。页面顶部的路由图显示每把密钥经过的路由、策略组和上游，进行中的请求各有一种颜色，沿各自的路径从密钥移向上游。 +每把密钥使用一条路由，未指定的使用默认路由。路由由按顺序匹配的规则组成。规则的条件包括模型、密钥、客户端的 API 格式、输入 token 数、`max_tokens`、工具数量、图片、扩展思考、流式、提示缓存、辅助请求的类型以及时段；命中后把请求交给某个上游、策略组或指定模型，或拒绝请求，也可以改写模型、`max_tokens` 或推理。「推理」可设为「关闭」，或 low、medium、high、xhigh、max 中的一档：指定档位时只改写已开启推理的请求，生成标题等未开启推理的请求保持不变，推理也不会因此开启。「关闭」按上游的格式写入，请求转换为其他格式时同样生效：发往 OpenAI Chat Completions 与 Responses 上游的写为 `none`，发往 Gemini 2.5 Flash 的思考预算为 0，发往 Anthropic 与 Bedrock 上游的不带思考设置；无法关闭思考的 Gemini 模型（2.5 Pro、Gemini 3）不带思考设置。上游拒收某一档推理强度时，请求改用上游接受的档位在同一上游重发一次：拒收 `none` 的，改用上游列出的最低一档，上游一档都没有列出时才去掉推理强度。这一选择按上游和模型记住 7 天，之后的请求直接发送可用的档位。2026.10.29 之前改写扩展思考的规则不再能加载，应用以安全模式打开：「一键修复」删去这一项后，在规则中重新设置「推理」即可，「关闭」对应原来的关闭扩展思考。指定模型写明上游和它的某个模型，可按顺序列出备用，模型名原样发出；要让某个客户端把一个模型当另一个模型的名称使用，在这个客户端的密钥上写规则即可，不会改变这个名称对其他客户端的含义。条件写上游模型名时，也匹配列有它的别名。时段按网关所在机器的本地时间判断：选择星期与起止时刻即可，一条规则可以写多个时段，结束早于开始的为跨夜时段。设置 `max_tokens` 可以限制回答的长度：上游到达上限时自行停止，不会报错。策略组把多个上游放在同一个名字下，并决定尝试的先后：按顺序、手动选择、轮询、延迟最低优先或费用最低优先。轮询策略组的每个成员有 1 到 100 的权重，即长期分到的请求比例；「分配依据」可以在权重之上再乘以各上游的速度（首 token 时间）、稳定性（最近的成功率）或两者，进行中的对话仍留在原上游。一次尝试失败时，请求转到下一个上游；同一会话默认保持在同一个上游上，以便提示缓存持续命中。所有上游都在回答开始之前暂时出错时，例如过载、返回服务器错误、无法连接或连接中断，网关依次等待 2、4、8 秒，此后每次 16 秒，再次发送，只有一个上游时也是如此；上游要求等待更久的，按它的要求等待。重试与排队合计不超过「设置 › 故障转移」中的「最多等待」；已经开始的回答不会重发，请求本身的错误、额度用完、凭据被拒和限流也不重试。返回 429 的上游暂停而不算失败，发给它的请求转到下一个上游或等待它恢复。上游连续「设置 › 故障转移」中的「无响应超时」（默认 300 秒）没有发出内容时，网关放弃这个上游，保活心跳不算内容：尚未向客户端发出内容时，请求转到下一个上游，没有可用的上游时客户端收到超时错误；回答已经开始时，以错误结束。页面顶部的路由图显示每把密钥经过的路由、策略组和上游，进行中的请求各有一种颜色，沿各自的路径从密钥移向上游。 客户端自行发出的辅助请求（连通性检查、预热、生成标题、话题识别、输入建议）可以由网关在本地应答而不产生费用，也可以转发。转发的辅助请求和普通请求一样经过路由规则，规则可以按辅助请求的类型把它们分流到费用更低的上游。 @@ -105,7 +105,7 @@ MCP 页管理客户端从自己的配置文件中加载的内容，这些内容 ## 设置 -设置页分为八节。「连接」列出本机 core 和已保存的远程 core，详见[连接远程 core](/zh-CN/docs/lite/remote-core)。「通用」设置语言、外观、菜单栏显示的内容（仅 macOS，也可设为不显示）、开机启动、提醒以系统通知发送、仅在应用内显示还是关闭，并可让设为不再显示的引导提示和忽略过的提醒重新显示。「网关监听」设置网关的访问范围（仅本机、所选网卡所在的局域网或所有网卡）、端口和放行网段。「故障转移」设置上游失败后暂停多久、请求交给下一个上游：连续失败几次后暂停、暂停多长，以及余额不足、额度用完和限流时各自的暂停时长；还设置无响应超时，即上游多久没有发出内容即被放弃（默认 300 秒，可设 30 到 3,600 秒），以及「最多等待」，即上游并发已满、正在限流或达到用量上限，以及密钥的分钟、小时上限用满时，请求合计最多等待的时长，默认 120 秒，超过后请求以错误结束。「日志保留」分别设置请求报文与请求记录的保留天数，以及报文的空间上限；安全日志随请求记录一同过期。「同步与备份」导出与导入配置、开启同步，并列出可以恢复的配置版本（见[同步与备份](#同步与备份)）。「关于」显示版本、检查更新，并可生成诊断包，其中的密钥与地址均已脱敏。「卸载」还原所有已接管的客户端并取消开机启动，应在删除应用之前执行。 +设置页分为八节。「连接」列出本机 core 和已保存的远程 core，详见[连接远程 core](/zh-CN/docs/lite/remote-core)。「通用」设置语言、外观、菜单栏显示的内容（仅 macOS，也可设为不显示）、开机启动、提醒以系统通知发送、仅在应用内显示还是关闭，并可让设为不再显示的引导提示和忽略过的提醒重新显示。「网关监听」设置网关的访问范围（仅本机、所选网卡所在的局域网或所有网卡）、端口和放行网段。「故障转移」设置上游失败后暂停多久、请求交给下一个上游：连续失败几次后暂停、暂停多长，以及余额不足、额度用完和限流时各自的暂停时长；还设置无响应超时，即上游多久没有发出内容即被放弃（默认 300 秒，可设 30 到 3,600 秒），以及「最多等待」，即请求排队等待上游空出位置、限流结束或用量上限恢复（含密钥的分钟、小时上限），以及上游暂时出错后重试，两者合计最多等待的时长，默认 120 秒，超过后请求以错误结束，设为 0 时既不排队也不重试。「日志保留」分别设置请求报文与请求记录的保留天数，以及报文的空间上限；安全日志随请求记录一同过期。「同步与备份」导出与导入配置、开启同步，并列出可以恢复的配置版本（见[同步与备份](#同步与备份)）。「关于」显示版本、检查更新，并可生成诊断包，其中的密钥与地址均已脱敏。「卸载」还原所有已接管的客户端并取消开机启动，应在删除应用之前执行。 ## 同步与备份 diff --git a/src/content/docs-lite/zh-CN/install.md b/src/content/docs-lite/zh-CN/install.md index 35818f5..bbc3d45 100644 --- a/src/content/docs-lite/zh-CN/install.md +++ b/src/content/docs-lite/zh-CN/install.md @@ -81,13 +81,13 @@ curl -fsSL https://github.com/ThinkWatchProject/ThinkWatch-Lite/releases/latest/ 自动检查发现新版本时，应用会将其列入提醒，并发送一条系统通知；「设置 › 通用」中的提醒设为「仅在应用内」时只列入提醒，设为「关闭」时两者都没有。点按这条提醒或系统通知、菜单栏或托盘菜单中的「安装新版本」，或「设置 › 关于」中的「更新到」，都会打开更新窗口；在同一菜单或「设置 › 关于」中选择「检查更新」，发现新版本时也会立即打开它。之后的处理方式取决于安装方式。 -**在 macOS 上从 release 页面下载安装的**：点击一次安装按钮，其余步骤自动完成——下载更新包，用编译进应用的公钥验签，等待网关正在处理的请求结束（最多三分钟），然后替换并重新启动。正在输出的任务不会因更新而中断。 +**在 macOS 上从 release 页面下载安装的**：点击一次安装按钮，其余步骤自动完成——下载更新包，用编译进应用的公钥验签，然后替换并重新启动。正在输出的任务不会因更新而中断（见下文）。 -**Windows 安装版**：同样点击一次即可。应用下载新版本的安装程序，用编译进应用的公钥验签，同样等待进行中的请求结束，然后运行安装程序，安装完成后新版本自动启动。应用为所有用户安装，因此每次更新 Windows 都会请求管理员权限；拒绝则继续运行当前版本。 +**Windows 安装版**：同样点击一次即可。应用下载新版本的安装程序，用编译进应用的公钥验签，然后运行安装程序，安装完成后新版本自动启动。应用为所有用户安装，因此每次更新 Windows 都会请求管理员权限；拒绝则继续运行当前版本。 -**Windows 绿色版**：同样点击一次即可，不需要管理员权限。应用下载新版本的压缩包，用编译进应用的公钥验签，等待进行中的请求结束，然后替换文件夹中的两个程序文件并重新启动；替换失败时换回原来的文件，继续运行当前版本。 +**Windows 绿色版**：同样点击一次即可，不需要管理员权限。应用下载新版本的压缩包，用编译进应用的公钥验签，然后替换文件夹中的两个程序文件并重新启动；替换失败时换回原来的文件，继续运行当前版本。 -**Linux 上**：同样点击一次即可，不需要输入密码。应用下载新版本的 AppImage，用编译进应用的公钥验签，等待进行中的请求结束，然后替换自身文件并重新启动。AppImage 须位于当前用户可写的目录中。 +**Linux 上**：同样点击一次即可，不需要输入密码。应用下载新版本的 AppImage，用编译进应用的公钥验签，然后替换自身文件并重新启动。AppImage 须位于当前用户可写的目录中。 **用 Homebrew 安装的**：窗口给出更新命令和复制按钮，应用不会替换自身。Homebrew 记录着它放入 `/Applications` 的版本，应用自行替换后，下一次 `brew upgrade` 会把旧版本写回。通过 Homebrew 安装的应用只在 tap 已包含新版本时才会提示更新，因此给出的命令一定有可安装的内容： @@ -97,6 +97,8 @@ brew update && brew upgrade --cask thinkwatch-lite 命令先执行 `brew update`，是因为 `brew upgrade` 自身最多每天刷新一次 tap。 +无论哪种安装方式，更新都不会中断网关：新版本启动时，正在运行的网关把端口交给新版本的网关，自己处理完进行中的请求后退出，网关端口始终可用，进行中的请求照常完成。在此期间，概览与菜单栏中进行中的请求包括上一版网关仍在处理的请求。用 Homebrew 升级时，应用识别出这次升级，与应用内更新一样交接网关；菜单中的「重新启动网关」也以同样的方式交接。更新完成后，窗口恢复为更新前的状态。这一方式自 2026.10.29 起生效：从更早的版本更新到 2026.10.29 时，仍由旧版本等待进行中的请求结束（最多三分钟）后重启网关。 + ## 下一步 - [概览](/zh-CN/docs/lite)：应用展示什么。 diff --git a/src/content/docs-lite/zh-CN/overview.md b/src/content/docs-lite/zh-CN/overview.md index 18ac253..6211c65 100644 --- a/src/content/docs-lite/zh-CN/overview.md +++ b/src/content/docs-lite/zh-CN/overview.md @@ -11,7 +11,7 @@ ThinkWatch Lite 是 Claude Code、Codex 等 AI 客户端的本地网关，支持 - **扫描 MCP、技能与钩子。** 十三款客户端的 MCP 服务器并列显示，并扫描客户端配置中的隐藏字符、提示注入、危险命令与过宽权限。 - **插件。** 用简短的 JavaScript 插件调整请求与回答，例如要求用指定的语言回答、在 WSL 与 Windows 之间转换路径。插件在沙箱中运行，只看到占位符而看不到密钥，每一处改动都有记录。 - **每个请求都可追溯。** 命中的规则、每一次尝试、格式转换与费用都有记录，也可以重放到另一个上游对比；全部请求记录都可以搜索，包括请求与回答的内容。 -- **路由与故障转移。** 按模型、工具、图片等条件分流；回答开始前上游出错时换用下一个，同一会话固定使用同一上游。 +- **路由与故障转移。** 按模型、工具、图片等条件分流，并可指定推理强度；回答开始前上游出错时换用下一个，暂时出错时稍候重试，同一会话固定使用同一上游。 - **多种上游。** API 密钥、Amazon Bedrock、ChatGPT 与 Z.ai 账号、中转服务与本机模型，Anthropic、OpenAI、Gemini 接口之间自动转换。 - **同步与备份。** 配置可以导出为一个文件，在新电脑上或重装之后导入，也可以经自选的云盘文件夹、WebDAV 或 S3 兼容存储在多台设备之间同步，缺省加密。每次配置变更都留下可以恢复的版本。 - **费用如实计算。** 估算的金额单独标注，无法计价的请求单独计数，不按零计入。 diff --git a/src/content/docs-lite/zh-CN/plugins.md b/src/content/docs-lite/zh-CN/plugins.md index 510a72b..68b6013 100644 --- a/src/content/docs-lite/zh-CN/plugins.md +++ b/src/content/docs-lite/zh-CN/plugins.md @@ -28,7 +28,7 @@ core 按客户端自己的格式写回改动，只动改过的条目，缓存标 3. 插件改动了请求时，内容过滤（包括隐藏字符规则）再检查一遍，只报告插件加入的内容；拦截时整个请求被拒绝，不会转到别的上游； 4. 需要时转换为上游的格式，经过「密钥与隐私」处理后发出。 -一次发送失败、请求转到另一个上游时，请求钩子从客户端的原始请求重新运行，为一个上游做的改动不会带到另一个上游。向同一个上游重发（例如登录令牌刷新后的重试）时，沿用请求钩子已有的结果。 +一次发送失败、请求转到另一个上游时，请求钩子从客户端的原始请求重新运行，为一个上游做的改动不会带到另一个上游。上游暂时出错后的重试是尝试链上新的一跳，请求钩子同样从原始请求重新运行。向同一个上游重发（例如登录令牌刷新后的重试）时，沿用请求钩子已有的结果。 插件修改模型时，只改变这一次发往上游的模型名，效果与路由规则改名相同，并取代规则改过的名字。请求不会重新路由，也不再核对上游的模型列表，但密钥的可用模型依然有效：改成密钥不可用的模型时，请求被拒绝。 diff --git a/src/content/docs/en/api-reference.md b/src/content/docs/en/api-reference.md index c7c8e1a..94aa3b0 100644 --- a/src/content/docs/en/api-reference.md +++ b/src/content/docs/en/api-reference.md @@ -45,6 +45,25 @@ Before a request is sent upstream, it is checked against the rate limits and bud [`GET /v1/usage`](#get-v1usage) tells a client what its key has used and how much room its limits leave. +### Temporary upstream failures + +When every route of a model has been tried and the last one failed for the moment before any of the answer reached the client, the request waits and is sent again instead of the error going straight back. A failure for the moment is a `500`, `502`, `503`, `504` or `529`; a connection that could not be made or broke before the answer's head; or a stream that broke off, or reported an overload or a server error (`overloaded_error`, `api_error`, `server_error`, `server_is_overloaded`, `slow_down`, Gemini `UNAVAILABLE`), before any content. The waits are 2, 4 and 8 seconds, then 16 seconds each time, each give or take 20%, or as long as the upstream said in `retry-after-ms` or `Retry-After` (a zero wait is ignored, and one shorter than 250 ms is taken as 250 ms). + +- A request goes again only to the routes that failed that way; a model with one route, and a stream, goes back to the same route. Each attempt is picked as before: the strategy, the circuit breaker, the route caps, session affinity and the provider holding a conversation's encrypted history all apply, and each failed attempt counts against its route's breaker. +- Retries stop when the next wait would end later than `gateway.retry_wait_secs` (120 seconds by default, `0` for no retries) after the request arrived; an attempt already under way is not cut short. Clients and load balancers with a shorter timeout may give up first. +- Rate limits and spent quota (`429`), refused credentials (`401`, `403`), any other 4xx, timeouts, the gateway's own limits, budgets and policies, a client that left, and a stream after part of its answer went out are never retried. +- While a stream can still be sent again, what the upstream sends before any content (Anthropic `message_start` and `ping`, Responses `response.created` and `response.in_progress`, a Chat chunk with only the role) is held back and goes out with the first content. While it is held, and while the gateway waits to send again, the client receives an SSE comment (`: keep-alive`) after every 15 seconds without data, except Gemini clients; the status is `200`. +- A non-streaming request's attempt is cut 2 seconds before the gateway's 120-second request timeout, counted from when the request reached the gateway port, and a retry starts only when its wait and another attempt as long as the last one fit before then. At the timeout the client gets the last upstream error, or `504` (`ProviderTimeout`) when there was none, with its `gateway_logs` row. +- The `gateway_logs` row of a request that was sent again records `"retries": n` in its detail, whether it then succeeded or failed. The metric `gateway_provider_retry_total{provider}` counts the attempts sent again, by the provider they went to. + +### Reasoning effort + +**Reasoning** in the model editor (`models.reasoning_effort`) sets the reasoning of the requests sent to that model: `off` takes reasoning out of requests that have it on, and a level (`low`, `medium`, `high`, `xhigh`, `max`) changes the effort of requests whose reasoning is on. Requests without reasoning, such as title generation, go out unchanged, and so does every request while the setting is empty. The setting is written the way each hop's upstream format writes it: with `off`, OpenAI Chat and Responses upstreams receive `none`, Gemini 2.5 Flash a thinking budget of 0, and Anthropic and Bedrock upstreams no thinking; Gemini models that cannot stop thinking (2.5 Pro, Gemini 3) get no thinking configuration. The response cache keys on the request together with the setting. + +When an upstream answers `400` about the reasoning level, the request goes once more to the same route with the level the error names (the lowest one it lists when `none` was refused), or without one when it names none. This is not counted as a retry, though its time falls within the retry window. When that is answered, the choice is remembered in Redis per provider, upstream model and request format for 7 days, so later requests send what worked straight away; a provider's memory is cleared when its URL or credentials change, a route relearns its protocol, or the provider is deleted. A hop whose reasoning level was changed records `"effort": {asked, sent, reason}` on its `gateway_logs` row. + +This applies to every model, whether or not it has a reasoning setting, and cannot be turned off: when an upstream refuses the reasoning level a request carries, the request goes out once more with a level the upstream accepts, or without one, and the change is remembered for that provider and model for 7 days. The level sent can therefore differ from the one the caller asked for, in either direction. + ### POST /v1/chat/completions OpenAI-compatible chat completions endpoint. Proxies requests to the configured upstream provider. diff --git a/src/content/docs/en/architecture.md b/src/content/docs/en/architecture.md index 8c3ae1e..952bebb 100644 --- a/src/content/docs/en/architecture.md +++ b/src/content/docs/en/architecture.md @@ -294,7 +294,7 @@ The AI API proxy engine. Contains: - **`token_counter.rs`** -- Token counting using `tiktoken-rs` for usage tracking and TPM enforcement. - **`cost_tracker.rs`** -- Real-time cost calculation based on model pricing from the database. - **`circuit_breaker.rs`** -- Three-state circuit breaker (Closed/Open/HalfOpen) per provider, with configurable `failure_threshold` and `recovery_secs`. -- **`retry.rs`** -- Retry with exponential backoff for transient failures (NetworkError, UpstreamRateLimited), with configurable `max_retries`, `initial_delay_ms`, `max_delay_ms`, and `jitter`. +- **`proxy/retry.rs`** -- Sends a request again when every route failed for the moment (an overload, a server error, a broken connection) before any of the answer reached the client, backing off 2, 4, 8, then 16 seconds within `gateway.retry_wait_secs`. Rate limits, refused credentials and other 4xx are not retried. ### mcp-gateway diff --git a/src/content/docs/en/configuration.md b/src/content/docs/en/configuration.md index 091c5fc..31f0d73 100644 --- a/src/content/docs/en/configuration.md +++ b/src/content/docs/en/configuration.md @@ -545,6 +545,14 @@ Policies saved before 3.8.0 keep their behaviour: redaction rules replace with p Turn it on where a request that escapes its limits is worse than a refused one. `GET /v1/usage` answers `503` while the limits cannot be read, whichever way the setting is. In the console, the setting is **Fail closed when limits can't be checked** under Settings › Security › Rate Limiter. +### Gateway + +| Setting | Default | Description | +| ---------------------------- | ------- | -------------------------------------------- | +| `gateway.retry_wait_secs` | `120` | How long after a request arrives it may still be sent again, in seconds, from 0 to 600; `0` turns retries off. See [Temporary upstream failures](/docs/api-reference#temporary-upstream-failures) | + +In the console, the setting is **Retry window (seconds)** under Settings › Gateway › Routing strategy. + ### Budget | Setting | Default | Description | diff --git a/src/content/docs/zh-CN/api-reference.md b/src/content/docs/zh-CN/api-reference.md index a118b35..6585b0c 100644 --- a/src/content/docs/zh-CN/api-reference.md +++ b/src/content/docs/zh-CN/api-reference.md @@ -45,6 +45,25 @@ [`GET /v1/usage`](#get-v1usage) 告诉客户端它的密钥已用多少、限额还剩多少。 +### 暂时失败的重试 + +模型的各条路由都已尝试、最后一次在回答的任何内容发给客户端之前暂时失败时，请求不立即返回错误，而是等待后重发。暂时失败包括：`500`、`502`、`503`、`504` 或 `529`；在回答头之前无法建立或已经中断的连接；在任何内容之前中断，或报告过载、服务端错误（`overloaded_error`、`api_error`、`server_error`、`server_is_overloaded`、`slow_down`、Gemini `UNAVAILABLE`）的流。等待依次为 2、4、8 秒，此后每次 16 秒，各上下浮动不超过 20%；上游以 `retry-after-ms` 或 `Retry-After` 给出等待时间的，按它给出的时间等待（为 0 的忽略，短于 250 毫秒的按 250 毫秒）。 + +- 重发只发往以这种方式失败的路由；只有一条路由的模型，以及流式请求，重发到同一条路由。每次尝试照常选路：路由策略、熔断器、路由上限、会话黏性，以及持有对话加密历史的 Provider 都照常生效，每次失败的尝试照旧计入所在路由的熔断。 +- 下一次等待会在请求到达后 `gateway.retry_wait_secs`（默认 120 秒，`0` 关闭重试）之后才结束时，不再重试；已经开始的尝试不会因此中断。客户端或负载均衡器的超时短于这段时间时，可能先行放弃。 +- 限流与额度用尽（`429`）、凭据被拒（`401`、`403`）、其他 4xx、超时、网关自身的限额、预算与策略、已断开的客户端，以及已有部分回答发出的流，都不重试。 +- 流仍可能重发期间，上游在内容之前发出的事件（Anthropic 的 `message_start` 与 `ping`，Responses 的 `response.created` 与 `response.in_progress`，只有角色的 Chat 分块）暂缓，与第一段内容一起发出。暂缓与等待重发期间，客户端每 15 秒没有数据时收到一行 SSE 注释（`: keep-alive`），Gemini 客户端不收到；状态码仍为 `200`。 +- 非流式请求的每次尝试在网关 120 秒的请求超时（从请求到达网关端口时算起）之前 2 秒截止；只有等待加上一次与上次同样长的尝试都能在此之前完成时，才开始重试。到时客户端收到最后一次的上游错误，没有上游错误时收到 `504`（`ProviderTimeout`），并留有 `gateway_logs` 记录。 +- 经过重发的请求，其 `gateway_logs` 行的详情记录 `"retries": n`，无论最终成功与否。指标 `gateway_provider_retry_total{provider}` 按发往的 Provider 统计重发的次数。 + +### 推理强度 + +模型编辑器中的「推理」（`models.reasoning_effort`）指定发往该模型的请求所带的推理：`off` 关闭已开启推理的请求中的推理；一档强度（`low`、`medium`、`high`、`xhigh`、`max`）改写已开启推理的请求的强度。未开启推理的请求（如生成标题）原样发出，留空时请求同样原样发出。这一设置按每一跳上游的格式写入：设为 `off` 时，OpenAI Chat 与 Responses 上游收到 `none`，Gemini 2.5 Flash 收到为 0 的思考预算，Anthropic 与 Bedrock 上游不带思考设置；无法停止思考的 Gemini 模型（2.5 Pro、Gemini 3）不带思考设置。响应缓存按请求连同这一设置计键。 + +上游因推理强度返回 `400` 时，请求在同一条路由上重发一次：改用错误中列出的档位（被拒的是 `none` 时，改用其中最低的一档），错误中一档都没有列出时去掉强度。这次重发不算作重试，但所用时间计入重试时限。重发得到答复后，这一选择按 Provider、上游模型与请求格式在 Redis 中保存 7 天，之后的请求直接发送可用的档位；Provider 的地址或凭据变更、路由重新识别协议或 Provider 被删除时，记录随之清除。推理强度被改动过的一跳，其 `gateway_logs` 行记录 `"effort": {asked, sent, reason}`。 + +这一处理适用于所有模型，与模型是否设置了「推理」无关，且不能关闭：请求所带的推理强度被上游拒收时，一律改用上游接受的档位或去掉强度后重发，并按 Provider 与模型记住 7 天。因此发出的档位可能与调用方要求的不同，可能更高，也可能更低。 + ### POST /v1/chat/completions OpenAI 兼容的聊天补全端点。将请求代理到已配置的上游提供商。 diff --git a/src/content/docs/zh-CN/architecture.md b/src/content/docs/zh-CN/architecture.md index afc987d..e4234e2 100644 --- a/src/content/docs/zh-CN/architecture.md +++ b/src/content/docs/zh-CN/architecture.md @@ -294,7 +294,7 @@ AI API 代理引擎。包含： - **`token_counter.rs`** —— 使用 `tiktoken-rs` 进行 Token 计数，用于用量跟踪和 TPM 限制。 - **`cost_tracker.rs`** —— 基于数据库中模型定价的实时成本计算。 - **`circuit_breaker.rs`** —— 按 Provider 的三态断路器（Closed/Open/HalfOpen），可配置 `failure_threshold` 和 `recovery_secs`。 -- **`retry.rs`** —— 针对瞬态故障（NetworkError、UpstreamRateLimited）的指数退避重试，可配置 `max_retries`、`initial_delay_ms`、`max_delay_ms` 和 `jitter`。 +- **`proxy/retry.rs`** —— 所有路由都在回答的任何内容发给客户端之前暂时失败（过载、服务端错误、连接中断）时重发请求，依次退避 2、4、8 秒，此后每次 16 秒，以 `gateway.retry_wait_secs` 为限。限流、凭据被拒和其他 4xx 不重试。 ### mcp-gateway diff --git a/src/content/docs/zh-CN/configuration.md b/src/content/docs/zh-CN/configuration.md index 033a063..651da5a 100644 --- a/src/content/docs/zh-CN/configuration.md +++ b/src/content/docs/zh-CN/configuration.md @@ -545,6 +545,14 @@ openssl rand -base64 32 漏算限额比拒绝请求更糟时，应打开这一设置。无论如何设置，限额无法读取时 `GET /v1/usage` 都返回 `503`。在控制台中，这一设置是「设置 › 安全 › 限速引擎」中的「限额无法检查时拒绝请求」。 +### 网关 + +| 设置 | 默认值 | 描述 | +| ---------------------------- | ------- | -------------------------------------------- | +| `gateway.retry_wait_secs` | `120` | 请求到达后仍可重发的秒数，取值 0 到 600；`0` 关闭重试。见[暂时失败的重试](/zh-CN/docs/api-reference#暂时失败的重试) | + +在控制台中，这一设置是「设置 › 网关 › 路由策略」中的「重试时限 (秒)」。 + ### 预算 | 设置 | 默认值 | 描述 | diff --git a/src/data/core-docs/config.md b/src/data/core-docs/config.md index 31f55de..a4c5f63 100644 --- a/src/data/core-docs/config.md +++ b/src/data/core-docs/config.md @@ -1134,6 +1134,23 @@ Before the first content of a streamed answer reaches the client, an error the upstream sends in the stream moves the request to the next candidate, the same as an error status would. +When every candidate has been tried and the last one failed temporarily +before any content reached the client — 500, 502, 503, 504 or 529, an +overload or server error reported in a streamed answer, unreachable, or its +stream broke or ended before any content — the request is retried after a +pause instead of passing the error on. The pause is 2, 4 and 8 seconds, then +16 seconds each time, varied by up to a fifth; when the upstream asks for a +longer one (`retry-after-ms`, `Retry-After`), it is used instead. Each retry +goes again to the candidates that failed temporarily, in order, skipping those +set aside unless all of them are; a conversation that stays on an upstream +retries only that one. Retrying stops when the next pause would end after +`wait_secs`, the time a request may wait in all; the client then gets the last +error. Rate limits, used-up quotas, an insufficient balance, rejected +credentials, missing models, errors in the request itself, other errors +reported in the stream, other 5xx statuses and idle timeouts are not retried; +they still move the request to the next candidate as before. Each retry appears among the request's attempts with its +number and the pause before it. `wait_secs: 0` turns retrying off. + An upstream can also go quiet: it accepts the request and then sends nothing, or stops partway through. After `idle_timeout_secs` (300 by default) without content, the gateway stops waiting for it. The time counts from the moment the @@ -1149,9 +1166,12 @@ to the complete answer. outcome `idle_timeout`, and the request moves to the next candidate. Until then an upstream's answer is held back from the client, so the next upstream starts it afresh. With no candidate left, the client gets a timeout error - (504) in its own format. The last candidate's stream is passed on as it - arrives, so once its response has started, a timeout there ends it with an - error event instead. + (504) in its own format. When the client receives a stream, the last + candidate's stream is held the same way while the request can still be + retried within `wait_secs`; otherwise it is passed on as it arrives, and + once its response has started, a timeout there ends it with an error event + instead. A non-streamed answer from the last candidate is always passed on + as it arrives. - A streamed answer is held for at most 15 seconds. If no content has arrived by then, the client receives `200` and the streaming headers, so that its own wait for headers does not run out, followed by an SSE comment @@ -1183,9 +1203,9 @@ pause is at most `rate_limit_max_pause_secs`, and the circuit breaker and `load-balance` do not count it as a failure. The request that got the 429 moves to the next candidate, or waits for the pause to end; the client sees nothing of it. A streamed answer that reports a rate limit before any content -counts as a 429 too; only on a request's last candidate that has not been rate -limited in the last five minutes does the client get that error (the upstream -is paused all the same). When the pause ends, one request goes first, and the +counts as a 429 too; only on a request's last candidate, once the request can +no longer wait (`wait_secs`), does the client get that error (the upstream is +paused all the same). When the pause ends, one request goes first, and the others follow once the upstream has accepted it (for a streamed answer, once its content starts), or at once if that request gets no answer, times out or finds the upstream's quota used up. A pause lasts until twcore restarts at @@ -1194,10 +1214,10 @@ most. A request waits for upstreams that cannot take it yet for at most `wait_secs` in all: an upstream at its `max_concurrent`, one paused after a 429, and one at its `minute` or `hour` usage limit. The same time also covers waiting for -a key's `minute` or `hour` limit, so a request never waits longer than -`wait_secs` for all of them together. An ongoing conversation waits for the -upstream it stays on and, if it does not become available in time, moves on -to the next one, where its cache starts over. A new conversation skips such an +a key's `minute` or `hour` limit and the pauses before retries, so a request +never waits longer than `wait_secs` for all of them together. An ongoing +conversation waits for the upstream it stays on and, if it does not become +available in time, moves on to the next one, where its cache starts over. A new conversation skips such an upstream at once. An upstream that has used up a day, week or month usage limit is skipped. When no candidate is left, the request waits for whichever becomes available first; streamed answers keep the client's connection open @@ -1219,7 +1239,7 @@ receive the request and failed, the client gets that failure instead. | `quota_pause_secs` | integer | `3600` | Seconds to set aside an upstream whose quota is used up when it does not say when the quota resets. When it does, the upstream is set aside until then. | | `rate_limit_max_pause_secs` | integer | `3600` | Longest pause, in seconds, after an upstream answers 429 (rate limited). The pause lasts as long as the upstream says (`retry-after-ms`, `Retry-After`, or a delay in the error body); when it says nothing, it backs off by the upstream's consecutive 429s: 5, 10, 20 and 40 seconds, then 60 seconds each time. A pause is not a failure: requests wait for the upstream within `wait_secs`, and the circuit breaker does not count it. | | `idle_timeout_secs` | integer | `300` | Seconds an upstream may go without sending content before the gateway stops waiting for it. Counted from the moment the request is sent and started again by every piece of content: text, reasoning and tool calls count, keep-alives do not. A whole (non-streamed) answer counts from sending to the complete answer. Before any content has reached the client, the upstream counts as failed and the request moves to the next one; with none left, the client gets a timeout error. After content has reached the client, the answer ends with an error. From 30 to 3600. | -| `wait_secs` | integer | `120` | Seconds a request waits in all, counted once the key's own `max_concurrent` lets it in: for a key's `minute` or `hour` limit to free up, and for an upstream that cannot take it yet — at its `max_concurrent`, paused after answering 429, or at one of its `minute` or `hour` usage limits. A key limit that does not free up in time refuses the request. An upstream that cannot take it is skipped for the next one, and when none is left the request waits for the first to become available; a conversation that stays on an upstream waits for it. When nothing became available in time, the client gets 429 with `Retry-After`. `0`: never wait. From 0 to 600. | +| `wait_secs` | integer | `120` | Seconds a request waits in all, counted once the key's own `max_concurrent` lets it in: for a key's `minute` or `hour` limit to free up, for an upstream that cannot take it yet — at its `max_concurrent`, paused after answering 429, or at one of its `minute` or `hour` usage limits — and before retrying upstreams that failed temporarily before any content (overloaded, a server error, unreachable). A key limit that does not free up in time refuses the request. An upstream that cannot take it is skipped for the next one, and when none is left the request waits for the first to become available; a conversation that stays on an upstream waits for it. When nothing became available in time, the client gets 429 with `Retry-After`; when retries did not succeed in time, it gets the last upstream error. `0`: never wait or retry. From 0 to 600. | ### `aliases` @@ -1320,6 +1340,21 @@ it to. A `load-balance` weight is therefore the long-run share of requests: conversations in progress stay where they are and count toward that upstream's share. +A conversation carrying content an upstream encrypted is a different case. +OpenAI Responses hands clients encrypted state to send back each turn +(`encrypted_content`: reasoning, compacted history, what Codex sub-agents +report to each other), and only the account that encrypted it can read it. +Such a conversation goes back to the upstream that last answered it whatever +the rule now picks, even after the configuration changed, the cache went cold +or a day has passed (it is remembered for 30 days). When it carries more than +encrypted reasoning, that upstream is the only candidate: there is no +failover, and if the upstream has been removed the request is refused with an +explanation. Encrypted reasoning alone can be left out, so other upstreams +remain as failover. After twcore restarts the gateway no longer knows which +upstream encrypted a conversation; if another upstream receives it and cannot +read it, the failure says so, and the conversation has to go back to the +original upstream or start over. + `balance_by` lets a `load-balance` group also look at how each upstream has been doing lately. Each member's weight is multiplied by a factor, and the group shares out requests by the result in the same way as above. @@ -1465,6 +1500,27 @@ read, naming the rule and the value. `set.model` may name an alias; each upstream then receives its own name for it. Answers carry the name the client asked for, as with aliases. +`set.reasoning` writes the effort where the client's format keeps it +(`output_config.effort`, `reasoning_effort`, `reasoning.effort`, Gemini's +`thinkingLevel`, or the thinking budget when the client gave a budget). A +request converted for an upstream of another format carries it the way that +format writes it: `off` sent to an OpenAI Chat or Responses upstream is +`none` there, and sent to Gemini 2.5 Flash it is a thinking budget of 0. + +When an upstream answers 400 because of the reasoning effort, whether the +client or a rule chose it, the request is sent once more to the same upstream +with a level the upstream named (the nearest one, or the ceiling it gave), or +without an effort when it named none. A refused `none` is replaced by the +lowest level the upstream named, so reasoning stays as low as it allows. The resend is a new attempt in the +request details and does not count as a retry. If it succeeds, the level is +remembered for that upstream and model for 7 days in `effort-memory.json` in +the data directory, so later requests send the working level directly. The +entries of an upstream are forgotten when its address, key or protocol +changes, or when it is removed. Requests that only give a thinking budget are +not resent. On WebSocket connections (Codex), a turn refused for its effort is +not resent and nothing is remembered: the refusal reaches the client as it +is. + @@ -1472,7 +1528,7 @@ it. Answers carry the name the client asked for, as with aliases. |---|---|---|---| | `model` | string | — | Send a different model. The prompt cache of the session is lost. | | `max_tokens` | integer | — | Replace `max_tokens`. | -| `thinking` | bool | — | Turn extended thinking on or off. | +| `reasoning` | `off` \| `low` \| `medium` \| `high` \| `xhigh` \| `max` | — | Reasoning. `off` turns it off: the reasoning settings and the effort are removed, and in the OpenAI formats the effort is set to `none`. A level changes the effort. Only requests that already have reasoning on are changed; requests without reasoning (such as title generation) are left as they are. | | `only_at_session_start` | bool | `false` | Apply only when a session starts. Recorded and shown; not in effect yet. | diff --git a/src/data/core-docs/config.zh-CN.md b/src/data/core-docs/config.zh-CN.md index 529f4b5..001b488 100644 --- a/src/data/core-docs/config.zh-CN.md +++ b/src/data/core-docs/config.zh-CN.md @@ -852,6 +852,16 @@ security: 流式回答在第一段内容交给客户端之前，上游在流里报的错误和错误状态码一样，会把 请求换到下一个候选。 +候选都试过之后，最后一个候选在内容交给客户端之前暂时出错的 —— 500、502、503、504、529， +流式回答里报的过载或服务端错误，连不上，流在内容之前断了或收尾 —— 请求不把错误交给客户端， +而是等待一段时间后重试。等待 2、4、8 秒，之后每次 16 秒，上下浮动不超过两成；上游要求等得 +更久的（`retry-after-ms`、`Retry-After`），按它要求的。每次重试再发给暂时出错的那几个候选，按原来的顺序，停用中的跳过， +都在停用中时照样重试；留在某一家的对话只重试这一家。下一次等待会超出 `wait_secs`（一个请求 +合计最多等的时间）时不再重试，客户端收到最后一次的错误。限流、额度用完、余额不足、凭据被拒、 +模型不存在、请求本身的错误、流里报的其他错误、其他 5xx 和无响应超时都不重试，照旧换到下一个 +候选。每次重试在请求的尝试链上单列一跳，带着它是 +第几次重试和之前等了多久。`wait_secs: 0` 关闭重试。 + 上游也可能不出声：收下请求之后一直不发内容，或者发到一半停住。连续 `idle_timeout_secs` 秒（默认 300）没有内容，网关就不再等它。计时从请求发给这家上游的 那一刻起，等空位的时间不算在内；每来一段内容重新计时：正文、推理、工具调用都算，心跳 @@ -861,7 +871,9 @@ security: - 还没有内容交给客户端时，这家上游记一次失败（和 5xx 一样计入 `failures_to_pause`）， 尝试链上这一跳的结果是 `idle_timeout`，请求换到下一个候选。在此之前上游的回答不交给 客户端，下一家从头开始回答。没有候选了，客户端收到它自己格式的超时错误（504）。 - 最后一个候选的流是边收边交给客户端的，它的响应开始之后再超时，回答以一条错误事件收尾。 + 客户端收的是流、请求还能在 `wait_secs` 之内重试时，最后一个候选的流同样暂存；否则它的 + 流边收边交给客户端，响应开始之后再超时，回答以一条错误事件收尾。最后一个候选的整包 + 回答一律边收边交。 - 流式回答最多暂存 15 秒。到时还没有内容，客户端先收到 `200` 和流式响应头，以免它自己 等响应头的时限先到，之后每 15 秒收到一行 SSE 注释（`: keep-alive`）；Gemini 的客户端 不发注释。上游的事件照旧暂存到第一段内容，故障转移在同一个 `200` 之下照常进行：下一家 @@ -882,14 +894,14 @@ failover: 暂停 5、10、20、40 秒，之后每次 60 秒；上游回了别的答复就重新计数。暂停最长 `rate_limit_max_pause_secs` 秒，熔断和 `load-balance` 都不把它算作失败。收到 429 的 请求换下一个候选，或者等暂停结束，客户端察觉不到。流式回答在内容之前报的限流也按 429 -处理；只有请求的最后一个候选、而且这家五分钟内没被限流过时，客户端才会收到这个错误（这家照样 -暂停）。暂停结束时先放一个请求过去，这家接受了它（流式回答要等到内容开始），其余的请求再 +处理；只有请求的最后一个候选、而且这个请求已经不能再等（`wait_secs`）时，客户端才会收到这个 +错误（这家照样暂停）。暂停结束时先放一个请求过去，这家接受了它（流式回答要等到内容开始），其余的请求再 跟上；这个请求没有得到答复、超时，或者遇到这家额度用完，其余的请求也照常发出。暂停只保留 到 twcore 重启为止。 一个请求等暂时不能发的上游合计最多 `wait_secs` 秒：并发数满了（`max_concurrent`）的、 回了 429 正在暂停的、到了 `minute` 或 `hour` 用量上限的都算。等密钥的 `minute`、`hour` -上限也算在这段时间里，几样加起来不超过 `wait_secs`。进行中的对话等它留在的那一家，到时 +上限和重试之前的等待也算在这段时间里，几样加起来不超过 `wait_secs`。进行中的对话等它留在的那一家，到时 还没有恢复就换下一家，缓存在那边从头建；新的对话遇到这样的上游直接跳过。用完了天、周、 月用量上限的上游也跳过。没有候选了，请求等最先恢复的那一家；流式回答照上文先收到 `200` 和保活，连接不会因此断开。都没有恢复，客户端收到 429，`Retry-After` 和 `retry-after-ms` @@ -908,7 +920,7 @@ failover: | `quota_pause_secs` | 整数 | `3600` | 上游报告额度用完、但没有给出重置时间时停用的秒数。给出了重置时间的，停用到那一刻。 | | `rate_limit_max_pause_secs` | 整数 | `3600` | 上游回 429（限流）后暂停这家的最长秒数。上游给出了等待时间的（`retry-after-ms`、`Retry-After` 或错误正文里的等待时间）按它给的暂停；没有给出的，按这家连续被限流的次数退避：5、10、20、40 秒，之后每次 60 秒。暂停不算失败：请求在 `wait_secs` 之内等这家恢复，熔断也不计入。 | | `idle_timeout_secs` | 整数 | `300` | 上游多少秒没有发出内容，网关就不再等它。从请求发出的那一刻算起，每来一段内容重新计时：正文、推理、工具调用都算，心跳不算。整包（非流式）的回答从发出算到整份回来。还没有内容交给客户端时，这家上游记一次失败，请求换到下一家；没有下一家了，客户端收到超时错误。已经有内容交给客户端的，回答以一条错误收尾。取值 30 到 3600。 | -| `wait_secs` | 整数 | `120` | 一个请求合计最多等的秒数，从过了密钥自己的 `max_concurrent` 时算起：等密钥的 `minute`、`hour` 上限空出名额，和等暂时不能发的上游 —— 并发数满了（`max_concurrent`）、回了 429 正在暂停、到了这家 `minute` 或 `hour` 的用量上限 —— 都算在里面。密钥的上限到时空不出来就拒绝。上游暂时不能发的先换下一家，没有下一家了就等最先恢复的那一家；留在某一家的对话等这一家。到时都没有恢复，客户端收到带 `Retry-After` 的 429。`0`：不等。取值 0 到 600。 | +| `wait_secs` | 整数 | `120` | 一个请求合计最多等的秒数，从过了密钥自己的 `max_concurrent` 时算起：等密钥的 `minute`、`hour` 上限空出名额，等暂时不能发的上游 —— 并发数满了（`max_concurrent`）、回了 429 正在暂停、到了这家 `minute` 或 `hour` 的用量上限 —— 和上游在内容之前暂时出错（过载、服务端出错、连不上）后重试之前的等待，都算在里面。密钥的上限到时空不出来就拒绝。上游暂时不能发的先换下一家，没有下一家了就等最先恢复的那一家；留在某一家的对话等这一家。到时都没有恢复，客户端收到带 `Retry-After` 的 429；到时重试仍未成功，客户端收到最后一次的上游错误。`0`：不等，也不重试。取值 0 到 600。 | ### `aliases` @@ -972,6 +984,8 @@ groups: 无论哪种类型，一段对话都留在上次回答它的那一家上游，让上游缓存着的那部分被再次读取，而不是换一家全价重算。同一轮之内（客户端正在回传工具结果）一律不换；跨轮时，上一次回答读或写了至少 1024 个 token 的 prompt cache、且距今不到五分钟，才继续留下。上游因失败进入冷却时，对话随之放开；故障转移之后接下回答的那一家，就是之后留下的那一家。一轮开始时命中的规则也沿用到这一轮结束：按输入大小或图片分流的规则不会让一轮半路换家，除非输入已经超出规则所指模型的上下文窗口。因此 `load-balance` 的权重是长期看各家分到的请求的比例：进行中的对话留在原来的上游，也算进那一家的份额。 +带着上游加密内容的对话另当别论。OpenAI Responses 把下一轮还要用的内容加密交给客户端，每一轮原样带回（`encrypted_content`：推理、压缩后的前文、Codex 子代理之间传递的消息），只有加密它的那个账号读得出来。这样的对话一律送回上次回答它的那一家，不论规则现在选中哪一家，配置改过、缓存已凉或隔了一天也一样（记 30 天）。加密的不只是推理时，候选只有那一家：不做故障转移，那一家已从配置中删除时请求被拒绝并说明原因。只加密了推理时，推理可以去掉，其余上游仍作为故障转移的备选。twcore 重启之后不再知道对话由哪一家加密；送到别家而解不开时，失败说明会写明原因，这段对话需要改回原来的上游，或者重新开始。 + `balance_by` 让 `load-balance` 组再看各上游最近的表现：每个成员的权重乘上一个系数，组内请求按乘出来的结果照上文的方式分。 - `weights`（默认）：只按权重。 @@ -1075,6 +1089,10 @@ routes: `set.model` 可以写别名，每家上游收到各自对它的叫法。和别名一样，回答里的模型名写成客户端请求的名称。 +`set.reasoning` 按客户端的格式写推理强度：`output_config.effort`、`reasoning_effort`、`reasoning.effort`、Gemini 的 `thinkingLevel`；客户端给的是思考预算时改预算。转换成另一种格式发给上游的请求按那种格式的写法写：`off` 发给 OpenAI Chat 或 Responses 上游时写 `none`，发给 Gemini 2.5 Flash 时写思考预算 0。 + +上游因为推理强度回 400 时（无论这一档是客户端写的还是规则改的），请求在同一家上游再发一次：改用上游原话里列出的、离原来那一档最近的一档，原话给了上限的用上限，一档都没列的去掉推理强度。被拒的是 `none` 时改用原话里列出的最低一档，推理保持在上游允许的最低。重发在请求详情里是新的一跳，不算重试。重发成功后，这一档按上游和模型记 7 天，存在数据目录的 `effort-memory.json` 里，之后的请求直接发可用的那一档。上游的地址、密钥或协议变了，或者上游被删除，它的记录随之作废。只给了思考预算的请求不重发。WebSocket 连接（Codex）上，一轮因推理强度被拒时不重发、也不记录，拒绝原样交给客户端。 + @@ -1082,7 +1100,7 @@ routes: |---|---|---|---| | `model` | 字符串 | — | 换成另一个模型发送。该会话的 prompt cache 随之失效。 | | `max_tokens` | 整数 | — | 替换 `max_tokens`。 | -| `thinking` | 布尔 | — | 开启或关闭扩展思考。 | +| `reasoning` | `off` \| `low` \| `medium` \| `high` \| `xhigh` \| `max` | — | 推理。`off` 关闭推理：去掉推理设置和推理强度，OpenAI 的格式改为写 `none`。写一档则改为该档强度。只改写已开启推理的请求，未开启推理的请求（如生成标题）保持不变。 | | `only_at_session_start` | 布尔 | `false` | 只在会话开始时应用。目前只记录和显示，尚未生效。 | diff --git a/src/data/core-docs/manifest.json b/src/data/core-docs/manifest.json index 720d9cb..7d84819 100644 --- a/src/data/core-docs/manifest.json +++ b/src/data/core-docs/manifest.json @@ -1,9 +1,9 @@ { "repository": "ThinkWatchProject/ThinkWatch-Core", - "ref": "v0.79.0", + "ref": "v0.83.1", "files": { - "docs/config.md": "6418013cd4dd862fb9735a05d19a5e0efa4e8f070ab89fc238b2c0e71622f8bb", - "docs/config.zh-CN.md": "4ac624456040885086a94b5795c8a9ef9f1fb8db83c51e0e3a9ef8c5dbfa0fcf", + "docs/config.md": "67cc13e428e91ab5584fba1eaca8c1a900a442b6da56d59d8b55dea4d26d62ad", + "docs/config.zh-CN.md": "bb33b04c8048758b0e9e900cce24f5eeb796e5cc434908cd8b848749956df54f", "docs/server.md": "5e1e9b901bef2b46d417aea057a1db24c78457d3c1938b8540ecb4766515b68f", "docs/server.zh-CN.md": "1f508e7c39b8ded28653773ca4d8701e6bdc2247bcd359c3a8fd00fd1401a6ff" } diff --git a/src/i18n/index.ts b/src/i18n/index.ts index d532308..b4b4f1d 100644 --- a/src/i18n/index.ts +++ b/src/i18n/index.ts @@ -86,7 +86,7 @@ const dict = { tagline: "A single port for all supported models, with drop-in compatibility.", bullets: [ { title: "Multi-format proxy", body: "OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, and Gemini APIs on a single port. Cursor, Continue, Cline, Claude Code, and the official SDKs connect by changing only the base URL." }, - { title: "Per-model routing: automatic or manual", body: "Each model has its own routing setting. Automatic modes weight upstreams by latency, by success rate, or by both combined (the default); manual mode splits traffic by fixed weights set on a drag bar. A circuit breaker takes failing upstreams out of rotation, a request that fails with a retryable error is sent to another upstream, and the console shows live health for every route." }, + { title: "Per-model routing: automatic or manual", body: "Each model has its own routing setting, and can set the reasoning effort its requests carry. Automatic modes weight upstreams by latency, by success rate, or by both combined (the default); manual mode splits traffic by fixed weights set on a drag bar. A circuit breaker takes failing upstreams out of rotation, a request that fails with a retryable error is sent to another upstream, or again after a short wait when every route failed for the moment, and the console shows live health for every route." }, { title: "Multiple providers and a model-level kill switch", body: "OpenAI, Anthropic, Google Gemini, Azure OpenAI, AWS Bedrock, or any OpenAI-compatible endpoint, with automatic format conversion. A single model can be paused without disabling the whole provider, and every request log records the upstream model that actually served the request." }, { title: "Virtual API keys", body: "Each tw- key belongs to one user and can be limited to the AI gateway, the MCP gateway, the admin API, or any combination, and tagged with a cost center. Keys support expiry, inactivity timeouts, and rotation with a grace period in which the old key keeps working; revoked and disabled keys are listed on a separate tab. The plaintext is shown once, and only a hash is stored." }, { title: "Rate limits & budgets", body: "Sliding windows from one minute to one week limit requests or tokens, and daily, weekly, or monthly budgets cap token use, weighted per model. Both attach to users, API keys, or roles and apply to MCP tool calls as well as model requests. Request counts are checked before a request; tokens and budgets are counted after the response, and once a budget is used up further requests are refused. A client can ask the gateway what its key has used and how much room its limits leave. If Redis or the database cannot be read, limits are skipped by default; a setting makes the gateway refuse those requests instead." }, @@ -318,7 +318,7 @@ const dict = { tagline: "单一端口接入所有支持的模型，可直接替换现有接入。", bullets: [ { title: "多格式代理", body: "OpenAI Chat Completions、OpenAI Responses、Anthropic Messages 与 Gemini 四种 API 在同一端口提供。Cursor、Continue、Cline、Claude Code 及官方 SDK 只需更换 base URL 即可接入。" }, - { title: "按模型路由：自动或手动", body: "每个模型有独立的路由设置。自动模式按延迟、成功率或两者结合（默认）为上游分配权重；手动模式通过拖拽流量条按固定比例分配。熔断器将故障上游移出轮换，因可重试错误而失败的请求会转发到其他上游，控制台展示每条路由的实时健康状态。" }, + { title: "按模型路由：自动或手动", body: "每个模型有独立的路由设置，并可指定请求的推理强度。自动模式按延迟、成功率或两者结合（默认）为上游分配权重；手动模式通过拖拽流量条按固定比例分配。熔断器将故障上游移出轮换，因可重试错误而失败的请求会转发到其他上游，所有路由都暂时失败时稍候重发，控制台展示每条路由的实时健康状态。" }, { title: "多 Provider 与模型级开关", body: "OpenAI、Anthropic、Google Gemini、Azure OpenAI、AWS Bedrock，或任意 OpenAI 兼容端点，请求格式自动转换。可在不停用整个 Provider 的前提下暂停单个模型；每条请求日志均记录实际提供服务的上游模型。" }, { title: "虚拟 API 密钥", body: "每把 tw- 密钥归属于一位用户，可限定用于 AI 网关、MCP 网关、管理 API 或其任意组合，并可标注成本中心。密钥支持到期时间、闲置超时，以及带宽限期的轮换（宽限期内旧密钥仍可使用）；已吊销和已停用的密钥列在单独的分页中。明文仅展示一次，只存储哈希。" }, { title: "限流与预算", body: "滑动窗口（1 分钟至 1 周）限制请求数或 token 数，按日、周或月设置的预算限制 token 用量（按模型加权）。两者均可设置在用户、API 密钥或角色上，同时作用于模型请求与 MCP 工具调用。请求数在请求前检查；token 与预算在响应后计入，预算用尽后的请求会被拒绝。客户端可以向网关查询自己的密钥已用多少、限额还剩多少。Redis 或数据库读取失败时默认不做限制，也可设置为直接拒绝这些请求。" }, diff --git a/src/i18n/pages/core.ts b/src/i18n/pages/core.ts index b82845d..090a0e5 100644 --- a/src/i18n/pages/core.ts +++ b/src/i18n/pages/core.ts @@ -46,7 +46,7 @@ export const coreCopy = { }, { title: "Failover before the first byte", - body: "Until the first byte reaches the client, a failing upstream is replaced by the next one without the client noticing; after that point, the failure is reported. A failing upstream is paused for as long as its reason calls for: an exhausted balance, or a used-up quota until its reset. An upstream that answers 429 is paused for the wait it asks, without counting as a failure, and its requests go to the next upstream or wait for it.", + body: "Until the first byte reaches the client, a failing upstream is replaced by the next one, and a brief failure of every upstream is retried after a short wait, without the client noticing; after that point, the failure is reported. A failing upstream is paused for as long as its reason calls for: an exhausted balance, or a used-up quota until its reset. An upstream that answers 429 is paused for the wait it asks, without counting as a failure, and its requests go to the next upstream or wait for it.", }, { title: "Cost accounting", @@ -170,7 +170,7 @@ export const coreCopy = { }, { title: "首字节前的故障转移", - body: "首字节到达客户端之前，出错的上游由下一个上游替换，客户端无从察觉；此后发生的故障如实报告。出错的上游按原因暂停：余额不足与额度用完（到重置时间为止）各有时长。返回 429 的上游按其要求的等待时间暂停，不计为失败，发给它的请求转到下一个上游或等待它恢复。", + body: "首字节到达客户端之前，出错的上游由下一个上游替换，所有上游都只是暂时出错时稍候重试，客户端无从察觉；此后发生的故障如实报告。出错的上游按原因暂停：余额不足与额度用完（到重置时间为止）各有时长。返回 429 的上游按其要求的等待时间暂停，不计为失败，发给它的请求转到下一个上游或等待它恢复。", }, { title: "费用核算", diff --git a/src/i18n/pages/lite.ts b/src/i18n/pages/lite.ts index ade3ed6..be04a99 100644 --- a/src/i18n/pages/lite.ts +++ b/src/i18n/pages/lite.ts @@ -108,7 +108,7 @@ export const liteCopy = { lines: ["macOS 12 or later, Apple silicon", "Windows 10 21H2 or later, x64 or ARM64", "Linux, x86_64 or aarch64"], }, { label: "Interface", lines: ["English and Simplified Chinese"] }, - { label: "Updates", lines: ["Installed by the app itself; through Homebrew for a Homebrew installation"] }, + { label: "Updates", lines: ["Installed by the app itself; through Homebrew for a Homebrew installation", "The gateway keeps serving during an update"] }, { label: "License", lines: ["MIT, free and open source"] }, ], }, @@ -142,7 +142,7 @@ export const liteCopy = { { id: "routing", title: "Routing by rule, with failover", - body: "Rules send requests to different upstreams by model, tools, images, extended thinking and more. When an upstream fails before the answer begins, the next one takes over, and each session stays on one upstream so its prompt cache keeps hitting. Auxiliary requests such as title generation and warm-ups can be answered locally without using any quota.", + body: "Rules send requests to different upstreams by model, tools, images, extended thinking and more, and can set the reasoning effort. When an upstream fails before the answer begins, the next one takes over, a brief failure is retried after a short wait, and each session stays on one upstream so its prompt cache keeps hitting. Auxiliary requests such as title generation and warm-ups can be answered locally without using any quota.", alt: "The Routing page: a map from keys through routes and groups to upstreams, and the routes with the rules each applies in order", }, { @@ -351,7 +351,7 @@ export const liteCopy = { lines: ["macOS 12 及以上（Apple silicon）", "Windows 10 21H2 及以上（x64、ARM64）", "Linux（x86_64、aarch64）"], }, { label: "界面语言", lines: ["英文、简体中文"] }, - { label: "更新", lines: ["应用自动更新；通过 Homebrew 安装的随 Homebrew 更新"] }, + { label: "更新", lines: ["应用自动更新；通过 Homebrew 安装的随 Homebrew 更新", "更新期间网关照常转发请求"] }, { label: "许可证", lines: ["MIT，免费开源"] }, ], }, @@ -385,7 +385,7 @@ export const liteCopy = { { id: "routing", title: "按规则分流，失败自动换", - body: "按模型、工具、图片、扩展思考等条件把请求分给不同上游。回答开始前上游出错时自动换用下一个，同一会话固定使用同一上游，提示缓存保持有效。标题生成、预热等辅助请求可在本地应答，不占用额度。", + body: "按模型、工具、图片、扩展思考等条件把请求分给不同上游，并可指定推理强度。回答开始前上游出错时自动换用下一个，暂时出错时稍候重试，同一会话固定使用同一上游，提示缓存保持有效。标题生成、预热等辅助请求可在本地应答，不占用额度。", alt: "路由页：从密钥经路由与策略组到上游的链路图，下方逐条列出每条路由的规则", }, { diff --git a/src/pages/llms.txt.ts b/src/pages/llms.txt.ts index 5bf5a70..a210969 100644 --- a/src/pages/llms.txt.ts +++ b/src/pages/llms.txt.ts @@ -30,7 +30,7 @@ ThinkWatch Lite runs a local AI API gateway on macOS (Apple silicon), Windows (x - After that, upstreams and models change inside the gateway; client configuration files are not edited again and clients do not restart. - Converts between the Anthropic Messages, OpenAI Chat Completions, OpenAI Responses and Gemini APIs, so Claude Code can use GPT or Gemini models and Codex can use upstreams that only offer Chat Completions. - Upstreams: API keys for Anthropic, OpenAI, Gemini, DeepSeek, Amazon Bedrock and any compatible API, relays such as OpenRouter, local models such as Ollama, and ChatGPT or Z.ai accounts signed in from the app. -- Routing rules match on model, token count, tools, images and extended thinking, and can rewrite the model name. Strategy groups: in order (failover), round robin with session affinity, lowest latency, lowest cost, manual selection. +- Routing rules match on model, token count, tools, images and extended thinking, and can rewrite the model name and the reasoning effort. Strategy groups: in order (failover), round robin with session affinity, lowest latency, lowest cost, manual selection. - Every request records its cost, tokens, cache hits, time to first token, generation speed, the rule it matched, each upstream it tried, and the full request and response. - Before a request leaves, API keys, tokens, private keys, ID numbers and card numbers can be replaced with placeholders, so a relay does not receive them. Tool calls that download and run code or send out credentials can be cut off mid-stream. MCP servers, skills and hooks are scanned. - Claude Pro and Max subscriptions cannot be used through the gateway; Claude Code needs an API key or a relay.