Repository navigation
fix(reader): 反爬挑战页独立报错 + 失败冷却负缓存 (#160) - #162
Merged
Merged
Conversation
掘金等站被 WAF 按 IP 限流时返回 200 的 JS 挑战空壳页,此前走完 提取链路后误报 not_readerable「没有可提取的正文」。现在按响应体签名 (字节 WAF waf_jschallenge/_wafchallengeid、Cloudflare challenge-platform) 识别,抛独立的 anti_bot_challenge(503,文案如实说明来源暂时限制访问), 并保留 curl_cffi 第二层重试。签名匹配加 64KB 体积上限,避免讨论反爬的 正常长文因正文出现词干被误判。
失败此前不缓存,用户每次点击都再打一次源站,对被反爬限流的站点 (拦截窗口分钟级)只会加深限流。现在按 content_id 做进程内负缓存,默认 120 秒(ARTICLE_READER_FAILURE_COOLDOWN_SECONDS,0 关闭)内直接重抛上次 错误;refresh=true 绕过,成功读取后清除。与 _robots_cache 同为进程级口径。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
变更
Closes #160。两个切片,对应两个 commit:
waf_jschallenge/_wafchallengeid),此前走完提取链路后误报not_readerable「没有可提取的正文」。现在按响应体签名识别(字节两家 + Cloudflarechallenge-platform),抛独立错误码anti_bot_challenge(503,文案如实说明「来源暂时限制访问」)。签名匹配带 64KB 体积上限——真实挑战页都是轻量空壳,讨论反爬的正常长文不会因正文出现词干被误判。刻意不进_no_retry:TLS 指纹型挑战 curl_cffi 仍可能救回。ARTICLE_READER_FAILURE_COOLDOWN_SECONDS,默认 120s,0 关闭),期内直接重抛上次错误;refresh=true绕过、成功后清除。与_robots_cache同为进程级口径。根因证据(本地复现):同机 6 连抓掘金同文,第 5、6 次连续两层(httpx+curl_cffi)失败拿到挑战页;约 15 分钟后自动解除。掘金 robots 允许
/post/。前端无需改动:API 已透传
detail.message,ReaderDrawer 直接渲染。明确不做
服务端解 WAF PoW / 种 cookie 伪装通过 = 绕过反爬,违反 article_reader 设计契约(no cookies / no stealth escalation)。
Verification
tests/test_article_reader.py:19/19 绿(12 存量 + 7 新增:签名识别正/负/体积上限、httpx 路径分类、tiered 保留 curl_cffi 重试、冷却抑制/refresh 绕过/成功清除/0 关闭)pytest tests/:exit 0,1372 通过 0 失败ruff format --check+ruff check:改动文件全过uv run python -m py_compile:通过