Repository navigation
feat(ingest): 掘金等信源入库前补全文存 raw_content (#161) - #163
Merged
Merged
Conversation
站内阅读点击时才外呼源站,会被掘金按 IP 频率限流的 WAF 挑战页挡住。 现在入库管线在去重后、落库前对命中 INGEST_FULLTEXT_HOSTS(默认 juejin.cn) 且缺正文的候选预抓一次原文,从提取出的结构块重建受控 HTML 存进 raw_content,站内阅读优先走 ingested 路径、点击时零外呼。 抓取复用 article_reader 分层链路(SSRF/robots/体积上限/挑战识别全继承), 每 sync 上限 6 条 + 进程级 2 秒最小间隔,任何失败只跳过该条目不阻断入库。 ingested 提取路径同步补上 trafilatura 丢短标题/小列表时的整篇回退, 入库全文与点击时抓取的结构还原口径一致。
fxbin
force-pushed
the
issue-161-ingest-fulltext
branch
from
October 27, 2026 08:59
45dacd5 to
4a7fd03
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
变更
Closes #161,与 #160(PR #162)互补:那边把点击时抓取的失败兜住,这边让掘金条目点击时根本不需要外呼。
content_enrichment.py:入库管线 Step 3.5(去重后、落库前)对命中INGEST_FULLTEXT_HOSTS(默认juejin.cn,逗号分隔可扩展,空关闭)且raw_content缺失/过短(<200 字)的候选预抓一次原文,从提取出的结构块重建受控 HTML(heading/p/li/quote/pre/img,文本全转义)存进raw_content。站内阅读因此优先走 ingested 路径,点击时零外呼,从根上避开掘金按 IP 频率限流的 WAF。fetch_readable_article):SSRF 校验、robots、体积上限、站内阅读把反爬挑战页误报为「没有可提取的正文」,且失败无冷却会加深限流 #160 的挑战识别全部继承。INGEST_FULLTEXT_MAX_PER_SYNC(默认 6)+ 进程级 2 秒最小间隔(asyncio.Lock + 单调时钟),多个信源并发入库也不会密集外呼同一源站。_extract_from_ingested_content补上_extract_from_html已有的「trafilatura 丢短标题/小列表时整篇回退手工提取」口径,入库全文与点击时抓取的结构还原一致(标题/代码块/列表在站内阅读里完整保留)。方案取舍(调研记录在 #161):掘金 content detail API 对数字 article_id 不开放(int→404、string→参数错误,内部 UUID 无映射),推荐流只带 90 字 brief,故走 www SSR 页面抓取。
Verification
tests/test_content_enrichment.py6/6:非目标域名/已有正文不外呼、补全 + ingested 路径结构还原 roundtrip、恶意文本转义、抓取失败 fail-open、每 sync 上限、hosts 留空关闭tests/test_content_pipeline.py:存量全绿 + 新增接线测试(去重后的新候选才进补全,重复条目零外呼)tests/test_article_reader.py12/12 存量绿(ingested 路径结构回退不破坏现有行为)pytest tests/:exit 0,1372 通过 0 失败make layering:通过(service→service 方向合法)ruff format --check+ruff check+py_compile:通过