Repository navigation
fix: 内容入库按参数上限分片 + 聚类写回批量化分段提交 (#155) - #157
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
变更内容
修复日志巡检发现的两个高优先级缺陷,分支
issue-155-ingest-chunk-clustering-batch↔ issue #155。1. 批量入库撞 asyncpg 参数上限(Vercel News / OpenAI Blog 反复导入失败)
_persist_new_content_items把整批新记录拼成一条多值 INSERT,每行 29 个绑定参数,asyncpg 单语句上限 32767,单次同步新条目 >1129 即在客户端抛ValueError。整批全有或全无 → 失败后一行都写不进 → 下个周期条目仍全是新的 → 自维持失败循环。IN查询按 800 一批分片,结果合并。2. post_sync_clustering 超时循环(24h 内 17 次 >300s)
UPDATE ... WHERE id=?(2000 条积压 = 2000 个串行往返);所有写入末尾一次性 commit,超时取消即整轮回滚(含已花费的 LLM 命名),积压原样留给下一次;外层 track_job 300s 超时小于内层租约 600s,且取消时内层租约以 SUCCESS 落库。WHERE id IN (...)批量 UPDATE;每 50 组分段提交,超时最多损失最后一批;stage_ms(fetch / cluster_name / write)进 stats 落job_execution_logs.result_summary;外层超时对齐 600s;取消时租约记 FAILED。Verification
bash backend/scripts/test_full_local.sh(一次性 PG 5433)uv run python -m py_compile(5 个改动文件)make layeringCloses #155