Repository navigation
get_textbox(), get_text_selection(): leave out text hidden by a clip path - #5262
Open
thomas-villani wants to merge 1 commit into
Open
thomas-villani wants to merge 1 commit into
thomas-villani wants to merge 1 commit into
Conversation
…path Both build their own TextPage when none is passed, without TEXT_MEDIABOX_CLIP. Text that a clip path removes from the rendered page then comes back, although get_text() and search_for() leave it out. Add only that flag; the other defaults are unchanged, and an explicit textpage still decides what is extracted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Page.get_textbox()andPage.get_text_selection()build their ownTextPagewhen none is passed, and neither setsTEXT_MEDIABOX_CLIP. As a result they return text that a clip path removes from the rendered page.get_text()andsearch_for()both set the flag and leave that text out.The docs give checking a
search_for()hit as the typical use ofget_textbox(), andtest_2730/test_2553expectget_textbox(page.rect)to agree withget_text(). Hidden text breaks both expectations:search_for()reports the text as absent, andget_textbox()returns it.This occurs in real files. I tested 66 open-access PMC articles (736 pages). Five are journal proofs that carry an older, wider typesetting inside a clipped form XObject. On 41 of their pages,
get_textbox(page.rect)returns about 10,200 words that are not on the rendered page.The change. Only the clip flag is added:
get_textboxgoes fromflags=0toflags=TEXT_MEDIABOX_CLIP.get_text_selectiongoes fromTEXT_DEHYPHENATEtoTEXT_DEHYPHENATE | TEXT_MEDIABOX_CLIP.I know from #3686 that
get_textboxuses different flags fromget_text()on purpose, so the other defaults stay as they are: ligatures, CID mapping and whitespace are untouched. Passingtextpage=still lets callers choose their own flags. That includes callers who want the hidden text, as some pymupdf4llm users did for clipped spreadsheet cells (pymupdf/pymupdf4llm#490).Annot.get_textbox, which goes throughutils.get_textbox, gets the same change.Effect on the same 66 articles. Extra words that
get_textbox(page.rect)returns beyondget_text()fall from 12,590 to 2,557. Visible words that it misses stay at 3,034 before and 3,036 after; that gap comes from the flags left at 0. So the fix removes hidden text without dropping visible text.Tests.
test_textbox_clipped_textfails without the change and passes with it.test_textextract.py,test_textsearch.py,test_textbox.py,test_annots.py,test_general.py::test_2730andtest_2553give the same results before and after.test_textextract.pytests fail both before and after, because this branch's tests are newer than the 2.1 wheel.This changes what two helpers return by default, so it probably deserves a line in
changes.txt. I've left that to you.