[wToTw123]

Summary

A page containing only stopwords still has candidate chunks, but cleaning leaves an empty BM25 vocabulary. rank_bm25 then divides by zero while calculating average IDF. Return no content when every cleaned candidate is empty.

Found from the current develop source; no corresponding issue has been filed. This PR does not claim to close an existing issue.

List of files changed and why

• crawl4ai/content_filter_strategy.py: return an empty result before constructing BM25Okapi for a vocabulary with no terms.
• tests/unit/test_bm25_empty_corpus.py: add offline behavior regression tests.

How Has This Been Tested?

The six new tests produce four failures on unchanged develop and all pass with the fix. They use the real BM25 implementation, with stemming enabled/disabled, thresholds 0/1 and mixed meaningful/empty candidates. The existing BM25 and #1213 deduplication tests produce the same 24 passes and five failures before and after this change. The unchanged failures are test_basic_extraction, test_user_query_override (outdated extract_page_query call missing body), test_header_extraction, test_no_metadata_fallback and test_large_content.

Run python -m pytest tests/unit/test_bm25_empty_corpus.py -q from the repository root. The new test file and changed source block pass Black. The full source-file Black check fails on unchanged develop too; unrelated formatting is excluded. git diff --check passes. Browser/LLM integration tests and the full repository test suite were not run for these offline changes.

Checklist:

• My code follows the style guidelines of this project — The new test file and changed source block pass Black. The full source-file Black check fails on unchanged develop too; unrelated formatting is excluded.
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas (or the change is self-explanatory)
• I have made corresponding changes to the documentation — No public API change; an unscorable page now follows the existing empty-result contract.
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes — the new tests pass; five unchanged existing failures are documented above

Prepared with AI assistance; validation scope and limitations are listed above.