Skip to content

fix: 内容入库按参数上限分片 + 聚类写回批量化分段提交 (#155) - #157

Merged
fxbin merged 4 commits into
mainfrom
issue-155-ingest-chunk-clustering-batch
Oct 25, 2026
Merged

fxbin merged 4 commits into
mainfrom
issue-155-ingest-chunk-clustering-batch

Conversation

@fxbin

@fxbin fxbin commented Oct 25, 2026

Copy link
Copy Markdown
Owner

变更内容

修复日志巡检发现的两个高优先级缺陷,分支 issue-155-ingest-chunk-clustering-batch ↔ issue #155。

1. 批量入库撞 asyncpg 参数上限(Vercel News / OpenAI Blog 反复导入失败)

  • 根因:_persist_new_content_items 把整批新记录拼成一条多值 INSERT,每行 29 个绑定参数,asyncpg 单语句上限 32767,单次同步新条目 >1129 即在客户端抛 ValueError。整批全有或全无 → 失败后一行都写不进 → 下个周期条目仍全是新的 → 自维持失败循环。
  • 修复:INSERT 按参数上限/列数分片(默认 500 行/批);hash 去重 IN 查询按 800 一批分片,结果合并。

2. post_sync_clustering 超时循环(24h 内 17 次 >300s)

  • 根因:写回逐条 UPDATE ... WHERE id=?(2000 条积压 = 2000 个串行往返);所有写入末尾一次性 commit,超时取消即整轮回滚(含已花费的 LLM 命名),积压原样留给下一次;外层 track_job 300s 超时小于内层租约 600s,且取消时内层租约以 SUCCESS 落库。
  • 修复:每话题组一条 WHERE id IN (...) 批量 UPDATE;每 50 组分段提交,超时最多损失最后一批;stage_ms(fetch / cluster_name / write)进 stats 落 job_execution_logs.result_summary;外层超时对齐 600s;取消时租约记 FAILED。

Verification

命令 结果
bash backend/scripts/test_full_local.sh(一次性 PG 5433) 1344 passed,无失败无跳过
独立一次性 PG 单跑两个新回归测试 2 passed(1200 条目首跑全量入库 / 二跑全判重;聚类强制多批提交仍完整落库)
uv run python -m py_compile(5 个改动文件) 通过
make layering 通过

Closes #155

@fxbin
fxbin merged commit 1f741e5 into main Oct 25, 2026
8 checks passed
@fxbin
fxbin deleted the issue-155-ingest-chunk-clustering-batch branch October 25, 2026 08:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

内容导入批量 INSERT 超过 asyncpg 参数上限;post_sync_clustering 超时后整轮回滚

1 participant