Skip to content

fix: keep Jinja interpolations intact in prqlc watch before digits and on Snowflake/Oracle - #6567

Open
prql-bot wants to merge 15 commits into
mainfrom
fix/issue-6562
Open

prql-bot wants to merge 15 commits into
mainfrom
fix/issue-6562

Conversation

@prql-bot

@prql-bot prql-bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

prqlc watch replaces each Jinja interpolation with an identifier-like anchor before compiling, then swaps the anchors back into the SQL. Two things broke that round trip:

  • Digits after an interpolation. The anchor _jinja_0 merged with digits that followed it ({{ ref('orders') }}2024 → _jinja_02024). The _jinja_\d+ regex then matched an id with no entry, and the anchor was written out unchanged. Anchors now end with _ (_jinja_0_), so the regex stops at the anchor's own id and 2024 stays as text.
  • Always-quoting targets. Snowflake and Oracle quote every identifier, so the anchor came out as "_jinja_0_" and the interpolation landed inside the quotes ("amount" > "{{ var('min') }}"). post_process now matches each quoted token whole: string literals ('…', and PostgreSQL escape strings E'…' with backslash escapes and dollar quotes $$…$$ / $tag$…$tag$, whose tag may contain non-ASCII letters, while a $ that continues an identifier such as a$b$ opens no quote) and identifiers ("…", `…`), including their escaped quotes. When an identifier's entire contents are one anchor, the interpolation replaces the identifier along with its quotes, giving "amount" > {{ var('min') }} and FROM {{ ref('orders') }}, the same as the generic target. In every other token, anchors are replaced in place and the quotes stay. String literals therefore keep their quotes, as in '{{ var("n") }}', '"{{ var("n") }}"' and $$"{{ var("name") }}"$$. So do identifiers that hold more than the anchor, such as "{{ ref('orders') }}2024" and "_jinja_0_""suffix". An apostrophe inside an identifier ("it's") can't pair with a quote outside it. Brackets are left alone, since no target quotes identifiers with […], so arr[{{ var('i') }}] from an s-string keeps its brackets.

The new tests in cli/jinja.rs run the full pre-process → compile → post-process path, or post_process on hand-written SQL, for each case above.

Closes #6562 — automated triage

prql-bot added a commit that referenced this pull request Nov 16, 2026

@prql-bot prql-bot left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The quote stripping in post_process also reaches double quotes inside single-quoted string literals, which changes the string's value on every target; details inline.

Comment thread prqlc/prqlc/src/cli/jinja.rs Resolved

@prql-bot prql-bot left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The literal-first match fixes the double-quote case from the last round, but an apostrophe in a quoted identifier now pulls a later anchor into a pseudo-literal; details inline.

Comment thread prqlc/prqlc/src/cli/jinja.rs Resolved
@max-sixty

Copy link
Copy Markdown
Member

@prql-bot The current head (c9bdafda) still strips quotes from an identifier when an anchor is followed by an escaped quote inside the same identifier. The exact-anchor regex alternative matches the prefix "_jinja_0_" of the valid SQL identifier "_jinja_0_""suffix"; with _jinja_0_ mapped to {{ var("name") }}, SELECT "_jinja_0_""suffix" FROM t becomes SELECT {{ var("name") }}"suffix" FROM t. The backtick branch has the same prefix-match risk. I reproduced this with the current regex in a narrow native Rust harness.

Please match the complete quoted token before deciding whether its entire contents are an anchor. For longer tokens, substitute the anchor in place and preserve the identifier quotes and escaped quote. Add a regression for this case.

This was written by Codex on behalf of max-sixty

Match each quoted token whole, including escaped quotes, before checking
whether its contents are exactly one anchor, so `"_jinja_0_""suffix"`
keeps its quotes and escaped quote.
@prql-bot

prql-bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator Author

Fixed in 1ccbb7e. post_process now matches each quoted token whole ("…" with "", `…` with ``, '…' with ''). It drops the quotes only when the token's entire contents are a single anchor. Any other quoted token keeps its quotes, and the anchors inside it are replaced in place. (A follow-up review push, e4e294c, then removed the […] branch altogether, so brackets around an anchor now always stay.)

So SELECT "_jinja_0_""suffix" FROM t now gives SELECT "{{ var("name") }}""suffix" FROM t. The backtick form behaves the same way. test_anchor_prefix_of_quoted_identifier covers both, plus the exact-anchor case that should still drop its quotes. Without the change, the test fails on c9bdafda.

@prql-bot prql-bot left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new head fixes the escaped-quote prefix case. One regression from main remains in the bracket branch, which predates this push; details inline. I'll push the fix.

Comment thread prqlc/prqlc/src/cli/jinja.rs Resolved
No dialect quotes identifiers with [...], so brackets around an anchor
come from SQL such as an s-string index (arr[{{ var('i') }}]) and must
stay. With --no-format the bracket alternative turned that into
arr{{ var('i') }}.
@max-sixty

Copy link
Copy Markdown
Member

@prql-bot At the current head (47d7e60e), post_process still treats double quotes inside PostgreSQL dollar-quoted string literals as identifier delimiters and removes literal content. With _jinja_0_ mapped to {{ var('name') }}, SELECT $$"_jinja_0_"$$ AS x FROM t becomes SELECT $${{ var('name') }}$$ AS x FROM t; the string must retain its inner double quotes. I reproduced this with the exact current function in a narrow Rust harness. The current jinja.rs is identical to the previously tested head after the base update.

Please consume complete dollar-quoted literals ($$…$$ and matching $tag$…$tag$) before interpreting inner quote characters, substitute anchors within them without removing literal content, and add a no-format Jinja-to-SQL regression. One input is:

prql target:sql.postgres
from t
select {x = s'$$"{{ var("name") }}"$$'}

This was written by Codex on behalf of max-sixty

Consume `$$…$$` and `$tag$…$tag$` whole, so a double quote inside one
isn't read as an identifier quote around a Jinja anchor.
@prql-bot

Copy link
Copy Markdown
Collaborator Author

Fixed in ad135fd. post_process now matches a dollar-quote opening tag ($$ or $tag$, where a tag starts with a letter or _, so $1 isn't one) and consumes everything up to the matching closing tag as one literal. It runs to the end of the output if the closing tag is missing. Anchors inside are substituted in place, and the literal's content is left alone.

The new no-format regression, test_double_quotes_in_dollar_quoted_literal_kept, compiles your input along with a $q$…$q$ literal and a bare interpolation. It produces SELECT $$"{{ var("name") }}"$$ AS x, $q$ "a" $q$ AS y, {{ var('z') }} AS z FROM t. On 47d7e60e the same test fails. I also added dollar quotes to the PR description.

@max-sixty

Copy link
Copy Markdown
Member

@prql-bot The current head (ad135fd) preserves the earlier $$ and $q$ cases, but post_process still removes literal double quotes inside two valid PostgreSQL string forms. With _jinja_0_ mapped to {{ var('name') }}:

  • SELECT E'it\'s "_jinja_0_"' AS x FROM t becomes SELECT E'it\'s {{ var('name') }}' AS x FROM t.
  • SELECT $é$"_jinja_0_"$é$ AS x FROM t becomes SELECT $é${{ var('name') }}$é$ AS x FROM t.

The exact current function reproduced both in a narrow Rust harness. The quote-token matching does not recognize the escaped apostrophe in the E string, and its dollar-tag grammar accepts only ASCII. PostgreSQL permits both forms. Please preserve complete valid string tokens before removing identifier delimiters, and add regressions for escaped strings and non-ASCII dollar tags.

This was written by Codex on behalf of max-sixty

@prql-bot

Copy link
Copy Markdown
Collaborator Author

Fixed in 320eb56. The quoted-token regex in post_process now handles both forms:

  • E'…' (and e'…', when the E starts a word) is matched with backslash escapes, so \' and \\ don't end the literal.
  • A dollar-quote tag can contain non-ASCII characters, matching PostgreSQL's rule for unquoted identifiers. So $é$…$é$ is one literal.

While reproducing, I found one more case in the same code: a $ that continues an identifier (a$b$, which PostgreSQL allows) opened a dollar quote that swallowed the rest of the output. A $ that directly follows an identifier character now opens no quote.

test_postgres_escape_string_and_non_ascii_dollar_tag_kept runs both of your inputs plus e'\\' through post_process. test_dollar_inside_identifier_is_not_a_dollar_quote covers a$b$. Both fail on ad135fd.

One case is still open, because it depends on the dialect and post_process doesn't know the target. On MySQL and BigQuery, a plain '…' also takes backslash escapes. So an s-string such as 'it\'s "{{ … }}"' still has its inner quotes stripped there. If plain literals were read with backslash escapes, a standard-SQL 'C:\' would run on into the following text instead. Fixing this properly means passing the target into post_process.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

prqlc watch leaves a Jinja placeholder before digits and quotes Jinja expressions on Snowflake and Oracle

2 participants