Skip to content

feat(dgw): page recording search with a cursor - #2137

Open
irvingouj@Devolutions (irvingoujAtDevolution) wants to merge 1 commit into
masterfrom
feat/recording-search-pagination
Open

irvingouj@Devolutions (irvingoujAtDevolution) wants to merge 1 commit into
masterfrom
feat/recording-search-pagination

Conversation

@irvingoujAtDevolution

@irvingoujAtDevolution irvingouj@Devolutions (irvingoujAtDevolution) commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Searching every recording doesn't fit in one response, so POST /jet/jrec/search now returns pages.

A page ends after reading about 16 MiB of logs or after maxHits hits, from 1 to 1000 and 1000 by default; any other value is rejected, never clamped. nextCursor replaces limitReached: sending the same query, recordings and startTime with it as cursor gets the next page, until a page has no nextCursor.

Gateway keeps no state between pages. The cursor holds the recording, file, byte offset and line number to resume at, plus a hash of the search, so a cursor sent with another search is a 400. A cursor whose position cannot exist is a 400 too, and its file name is only matched against the recording's manifest, never joined into a path. Pages never go back, so a recording that starts behind the cursor while paging isn't searched. The paging rules are in search.intent.md.

🤖 Generated with Claude Code

Issue: DGW-463

@irvingoujAtDevolution

irvingouj@Devolutions (irvingoujAtDevolution) commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Lines changed, against #2112:

Added Removed
Production code (search.rs) 245 63
Tests (search.rs unit tests, recording_search.rs) 351 29
Other (search.intent.md, hex for the testsuite) 14 3

@irvingoujAtDevolution

Copy link
Copy Markdown
Contributor Author

Intent drafts: handoff

The Pages section of search.intent.md replaces the Limits section of #2112. The rules come from the same interview, which read #2112 at 4de203f.

Provenance

Levels: 1 = my words verbatim, 2 = my words tidied, 3 = agent wording I adopted, 4 = agent fill-in, not intent.

# Commitment as written My words (if tidied) Level
1 Results come in pages, so that searching every recording takes a bounded time per request. "I efel like some sort of pagination is needed"; "there might be thouands of videos" 2
2 A page ends after reading about 16 MiB of logs or after maxHits hits, whichever comes first. "so one page, internally, for gateway to read is 16mb at most, but we return at max 1000 line as long as 16mb not hit, correct?" 2
3 maxHits is from 1 to 1,000 and defaults to 1,000; a value outside that range is rejected, never clamped. "reject 0 and > 1000, just reject"; "default 1000" 2
4 A page carries a nextCursor until every log is read, and a page with fewer hits than maxHits may still have one, since the read limit can end it first. "acceptable, we can do this" (cursor design) 3
5 Gateway keeps no state between pages; the cursor holds where the next page starts. "acceptable, we can do this" 3
6 A cursor is accepted only with the same query, recordings and startTime as the page that gave it. "acceptable, we can do this" (a hash of the request in the cursor) 3
7 The hit that does not fit in a page opens the next one, so that every page moves the search forward and returns each hit once. "acceptable, we can do this" 3
8 Pages never go back: a recording that starts, or a log that grows, behind the cursor while paging is not searched. "acceptable, we can do this" (no catch-up while paging) 3
9 A log replaced while paging, as an AI analysis rerun does, is read from the cursor's byte offset on, so lines can be missed or misnumbered; this is accepted for now. "let's keep it simple for today, just accept it" 2
10 "the same on every request" becomes "the same on every page". tidy only 3

Agent fill-in (level 4, not in the intent file): the cursor is hex-encoded JSON, not signed. A changed cursor can only move the read position inside the logs the same search reads, since its recording ID and file name are matched against the listing and never joined into a path.

Left out on purpose

  • Skip-N offset paging and server-side search sessions were rejected for the stateless cursor.
  • A file-version check in the cursor is not built, per commitment 9.

Placement

devolutions-gateway/src/api/jrec/search.intent.md, same file as #2112.

Pending experiments

None.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Numeric page sizes exceeding u32 return 422 instead of the promised 400.

1 open issue
What changed in this PR

Adds stateless cursor pagination to recording search so clients can retrieve results across bounded pages.

Changes:

  • Replaces limitReached with nextCursor and adds maxHits.
  • Documents paging behavior and cursor constraints.
  • Adds pagination and cursor-validation tests.
File Description
testsuite/​tests/​cli/​dgw/​recording_search.rs Adds HTTP pagination and cursor tests.
testsuite/​Cargo.toml Adds hex support for cursor tests.
devolutions-gateway/​src/​api/​jrec/​search.rs Implements cursor pagination and unit tests.
devolutions-gateway/​src/​api/​jrec/​search.intent.md Defines pagination semantics.
Cargo.lock Records the test dependency.

Review effort: Balanced

Comment thread devolutions-gateway/src/api/jrec/search.rs
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) force-pushed the feat/recording-search-pagination branch 7 times, most recently from f05df42 to 170f536 Compare October 21, 2026 18:36
@irvingoujAtDevolution
irvingouj@Devolutions (irvingoujAtDevolution) force-pushed the feat/recording-search-pagination branch 3 times, most recently from 7606fd7 to 29adde5 Compare October 22, 2026 17:26
Base automatically changed from feat/recording-search to master October 22, 2026 17:57
Searching every recording doesn't fit in one response, so
`POST /jet/jrec/search` now returns pages.

A page ends after reading about 16 MiB of logs or after `maxHits`
hits, from 1 to 1000 and 1000 by default; 0 or more than 1000 is a
400. `nextCursor` replaces `limitReached`: sending the same `query`,
`recordings` and `startTime` with it as `cursor` gets the next page,
until a page has no `nextCursor`.

Gateway keeps no state between pages. The cursor holds the recording,
file, byte offset and line number to resume at, plus a hash of the
search, so a cursor sent with another search is a 400. A cursor whose
position cannot exist is a 400 too, and its file name is only matched
against the recording's manifest, never joined into a path. Pages
never go back, so a recording that starts behind the cursor while
paging isn't searched. The paging rules are in `search.intent.md`.

Issue: DGW-463

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Gateway keeps no state between pages; the cursor holds where the next page starts.
A cursor is accepted only with the same `query`, `recordings` and `startTime` as the page that gave it.
The hit that does not fit in a page opens the next one, so that every page moves the search forward and returns each hit once.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intent is good, but I think one line could make the paging invariant explicit and remove ambiguity:

Page boundaries are only between hits (line results), never inside one.
If the next hit would exceed page limits, that hit is deferred to the next page.

This makes it clear that:

  • pages advance strictly by whole hits,
  • no line/hit is truncated to fit a page,
  • and each hit appears exactly once across pages.

@CBenoit Benoît Cortier (CBenoit) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants