Skip to content

get_textbox(), get_text_selection(): leave out text hidden by a clip path - #5262

Open
thomas-villani wants to merge 1 commit into
pymupdf:mainfrom
thomas-villani:ghost-text-clip
Open

thomas-villani wants to merge 1 commit into
pymupdf:mainfrom
thomas-villani:ghost-text-clip

Conversation

@thomas-villani

Copy link
Copy Markdown

Page.get_textbox() and Page.get_text_selection() build their own TextPage when none is passed, and neither sets TEXT_MEDIABOX_CLIP. As a result they return text that a clip path removes from the rendered page. get_text() and search_for() both set the flag and leave that text out.

import pymupdf
src = pymupdf.open(); sp = src.new_page(width=400, height=200)
sp.insert_text((20, 50), "VISIBLE left part", fontsize=12)
sp.insert_text((250, 50), "HIDDEN right part", fontsize=12)
doc = pymupdf.open(); page = doc.new_page(width=400, height=200)
left = pymupdf.Rect(0, 0, 200, 200)
page.show_pdf_page(left, src, 0, clip=left)          # right half is clipped away

page.get_text()                                # 'VISIBLE left part\n'
page.search_for("HIDDEN")                      # []
page.get_textbox(page.rect)                    # 'VISIBLE left part\nHIDDEN right part'
page.get_text_selection((0, 40), (400, 60))    # 'VISIBLE left part\nHIDDEN right part'

The docs give checking a search_for() hit as the typical use of get_textbox(), and test_2730/test_2553 expect get_textbox(page.rect) to agree with get_text(). Hidden text breaks both expectations: search_for() reports the text as absent, and get_textbox() returns it.

This occurs in real files. I tested 66 open-access PMC articles (736 pages). Five are journal proofs that carry an older, wider typesetting inside a clipped form XObject. On 41 of their pages, get_textbox(page.rect) returns about 10,200 words that are not on the rendered page.

The change. Only the clip flag is added:

  • get_textbox goes from flags=0 to flags=TEXT_MEDIABOX_CLIP.
  • get_text_selection goes from TEXT_DEHYPHENATE to TEXT_DEHYPHENATE | TEXT_MEDIABOX_CLIP.

I know from #3686 that get_textbox uses different flags from get_text() on purpose, so the other defaults stay as they are: ligatures, CID mapping and whitespace are untouched. Passing textpage= still lets callers choose their own flags. That includes callers who want the hidden text, as some pymupdf4llm users did for clipped spreadsheet cells (pymupdf/pymupdf4llm#490). Annot.get_textbox, which goes through utils.get_textbox, gets the same change.

Effect on the same 66 articles. Extra words that get_textbox(page.rect) returns beyond get_text() fall from 12,590 to 2,557. Visible words that it misses stay at 3,034 before and 3,036 after; that gap comes from the flags left at 0. So the fix removes hidden text without dropping visible text.

Tests.

  • The new test_textbox_clipped_text fails without the change and passes with it.
  • test_textextract.py, test_textsearch.py, test_textbox.py, test_annots.py, test_general.py::test_2730 and test_2553 give the same results before and after.
  • These runs used the 2.1 wheel with this change applied, on Windows. Eight test_textextract.py tests fail both before and after, because this branch's tests are newer than the 2.1 wheel.

This changes what two helpers return by default, so it probably deserves a line in changes.txt. I've left that to you.

…path

Both build their own TextPage when none is passed, without
TEXT_MEDIABOX_CLIP. Text that a clip path removes from the rendered page
then comes back, although get_text() and search_for() leave it out.
Add only that flag; the other defaults are unchanged, and an explicit
textpage still decides what is extracted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant