diff --git a/README.md b/README.md index 97cae92c..48474cf5 100644 --- a/README.md +++ b/README.md @@ -323,8 +323,9 @@ If planned renames clash, the process is halted and no change is applied to any | `-s` | **`--dot-all`** Have `.` also match newline. [boolean] | | `-M` | **`--void-multiline`** Void multiline search pattern. Makes ^ and $ match start/end of whole content rather than each line. [boolean] | | `-u` | **`--unicode`** Treat pattern as a sequence of unicode code points. [boolean] | +| | **`--flags`** The flags of the regex, like `gimsv`, in place of the defaults: `gim`, or `gimvxn` with `-E regex+`. `x` and `n` need `-E regex+`. Not with -G, -I, -M, -s or -u. [string] | | `-e` | **`--encoding`** Encoding of files/piped data. [default: "utf8"] | -| `-E` | **`--engine`** What regex engine to use. `default` is the engine of the runtime: `V8` with Node and `JSC` (JavaScriptCore) with Bun. The engine of the other runtime can not be picked. [choices: "default", "V8", "JSC"] [default: "default"] | +| `-E` | **`--engine`** The regex engine. `rr` (also `default`) reads patterns like the native engine and adds the extended syntax of Regex+: atomic groups `(?>…)`, possessive quantifiers like `a++`, and subroutines `\g`. `regex+` is Regex+ with the flags v, x and n. `runtime` is the native engine of the runtime: V8 in Node and JavaScriptCore in Bun. [choices: "rr", "default", "regex+", "runtime"] [default: "rr"] | | `-q` | **`--quiet`** Hide file names and warnings. Errors are still shown, and output from flags like --output and --output-match is still shown. [boolean] | | `-Q` | **`--quiet-total`** Never display errors or info [boolean] | | `-f` | **`--full-path`** Print the full path of the named files instead of the relative path. [boolean] | @@ -366,8 +367,8 @@ If planned renames clash, the process is halted and no change is applied to any | `-g` | **`--glob-pipe`** Each line of the piped data is a path or a glob of the files to work on. They are used together with the ones given as arguments. [boolean] | | | **`--null-in`** The piped paths and globs of --glob-pipe are separated by null bytes, like from `find -print0` or --null-out. [boolean] | | `-R` | **`--replacement-pipe`** Replacement will be piped in. You still need to provide a dummy value (like `_`) as replacement parameter. [boolean] | -| `-j` | **`--replacement-js`**, **`--js`** Treat replacement as javascript source code. The statement from the last expression will become the replacement string. Purposefully implemented the most insecure way possible to remove _any_ incentive to consider running code from an untrusted part. The full match will be available as a javascript variable named $0 while each captured group will be available as $1, $2, $3, ... and so on. At some point, the $ char _will_ give you a headache when used from the command line, so use €0, €1, €2, €3... instead. The code has access to the following variables: `r` as an alias for `require` with both expanded to understand a relative path even if it is not starting with `./`, `fs` from node, `path` from node, `globs` from npm, `pipe`: the data piped into the command with --js-pipe (null if no piped data, and `pipe_` is `❌ `), `find`: pattern searched for (the needle), `text`: full text being searched i.e. file content or piped data (the haystack), `bytes`: total size of the haystack in bytes, `size`: human-friendly representation of the total size of the haystack, `time`: String representing the local time when the command was invoked, `time_obj`: date object representing `time`, `now`: alias for `time`, `cwd`: current process working dir, `nl`: a new-line char, `_`: a single space char (for easy string concatenation). The following values defaults to `❌` if haystack does not originate from a file: `file`: contains the full path of the active file being searched (including full filename), `file_rel`: contains `file` relative to current process working dir, `dirpath`: contains the full path without filename of the active file being searched, `dirpath_rel`: contains `dirpath` relative to current process working dir, `filename`: is the full filename of the active file being searched without path, `name`: filename of the active file being searched with no extension, `ext`: extension of the filename including leading dot, `mtime`: ISO inspired representation of the last local modification time of the current file, `ctime`: ISO representation of the local creation time of the current file. `mtime_obj`: date object representing `mtime`, `ctime_obj`: date object representing `ctime`. The numbers: `$n`: the number of the file, `$t`: the number of files, `$i`: the number of the file among the files with a match so far, `$c`: among the files changed so far, `$m`: the number of the match in the file, `$M`: across all files. Each has cousins with zeros in front (`$n0` has 2 digits, `$n00` 3, and so on to 10) and with a space at the end (`$n_`). For speed, the code runs once per file unless it names the match, a group, `$m` or `$M`. Access a named group via the name prepended with `rr_`. `(?...)` gives you `rr_year`. A named group name can not end with `_`. All variables, except from module, date objects, `nl` and `_`, has a corresponding variable name followed by `_` where the content has an extra space at the end (for easy concatenation). [boolean] | -| `-x` | **`--exclude-re`** Files with a path that matches this regular expression will be excluded from processing. Can be provided multiple times. [string] | +| `-j` | **`--replacement-js`**, **`--js`** Treat replacement as javascript source code. The statement from the last expression will become the replacement string. Purposefully implemented the most insecure way possible to remove _any_ incentive to consider running code from an untrusted part. The full match will be available as a javascript variable named $0 while each captured group will be available as $1, $2, $3, ... and so on. At some point, the $ char _will_ give you a headache when used from the command line, so use €0, €1, €2, €3... instead. The code has access to the following variables: `r` as an alias for `require` with both expanded to understand a relative path even if it is not starting with `./`, `fs` from node, `path` from node, `globs` from npm, `pipe`: the data piped into the command with --js-pipe (null if no piped data, and `pipe_` is `❌ `), `find`: pattern searched for (the needle), `text`: full text being searched i.e. file content or piped data (the haystack), `bytes`: total size of the haystack in bytes, `size`: human-friendly representation of the total size of the haystack, `time`: String representing the local time when the command was invoked, `time_obj`: date object representing `time`, `now`: alias for `time`, `cwd`: current process working dir, `nl`: a new-line char, `_`: a single space char (for easy string concatenation). The following values defaults to `❌` if haystack does not originate from a file: `file`: contains the full path of the active file being searched (including full filename), `file_rel`: contains `file` relative to current process working dir, `dirpath`: contains the full path without filename of the active file being searched, `dirpath_rel`: contains `dirpath` relative to current process working dir, `filename`: is the full filename of the active file being searched without path, `name`: filename of the active file being searched with no extension, `ext`: extension of the filename including leading dot, `mtime`: ISO inspired representation of the last local modification time of the current file, `ctime`: ISO representation of the local creation time of the current file. `mtime_obj`: date object representing `mtime`, `ctime_obj`: date object representing `ctime`. The numbers: `$n`: the number of the file, `$t`: the number of files, `$i`: the number of the file among the files with a match so far, `$c`: among the files changed so far, `$m`: the number of the match in the file, `$M`: across all files. Each has cousins with zeros in front (`$n0` has 2 digits, `$n00` 3, and so on to 10) and with a space at the end (`$n_`). For speed, the code runs once per file unless it names the match, a group, `$m` or `$M`. Access a named group as `$` and the name: `(?...)` gives you `$year`. A group named like a number, like `n`, wins over the number. With the flag `n` of `-E regex+` only named groups capture, so there is no `$1`. All variables, except from module, date objects, `nl` and `_`, has a corresponding variable name followed by `_` where the content has an extra space at the end (for easy concatenation). [boolean] | +| `-x` | **`--exclude-re`** Files with a path that matches this regular expression will be excluded from processing. It uses the same engine and flags as the pattern. Can be provided multiple times. [string] | | `-X` | **`--exclude-glob`** Files found with this glob will be excluded from being processed. Can be provided multiple times and will act as one group of globs, meaning you can use ! notation to negate. [string] | | `-h` | **`--help`** Display help. [boolean] | @@ -389,6 +390,7 @@ Files identified for potential processing are identified by: - Patterns are described as javascript notation regex - Pattern defaults to global multiline case-insensitive search - Supports regex lookaheads in the pattern +- Supports the extended syntax of [Regex+](https://github.com/slevithan/regex) in the pattern: atomic groups `(?>…)`, possessive quantifiers like `a++`, and subroutines `\g` with `(?(DEFINE)…)`. A pattern that works in the native engine of the runtime works the same, with one difference: `\g` calls the group, where the native engine reads it as the text `g` (use `-E runtime` for that). Regex+ emulates atomic groups and possessive quantifiers with a lookahead and a backreference, and a lookbehind reads from right to left, so inside a lookbehind they look forward in place of back, and match in the wrong place with no error: `(?<=(?>a))` finds the places in front of an `a`, not after it. A subroutine needs a named group that exists, as Regex+ has no numbered ones like `\g<1>`. A subroutine copies the groups inside the group it calls, and an unnamed one there gets a number of its own: in `(?

(a)(b))\g

`, `$4` and `$5` are the copies of `$2` and `$3`, and `--output-match` prints them too. Name the groups inside, or leave them out, to avoid it. Without `-u`, the native engine reads `[[]` as a class with a `[` in it, but Regex+ sees the class as still open, so the extended syntax after it is not read: `[[](?>a)` stops the run, and `[\[](?>a)` works. Counting the groups Regex+ adds, a replacement text can only reach the first 99 groups, as in javascript. With `-E regex+`, `-u` gives `u` in place of `v`. The flags `u` and `v` can not go together. - Supports backreference to matching groups in the replacement - Data can be piped in, both as the text to search and as the replacement - The names of the changed files are printed to stdout, so they can be piped on (to stderr with `--output` and `--output-match`) diff --git a/biome.json b/biome.json index 4043a395..cf537517 100644 --- a/biome.json +++ b/biome.json @@ -63,6 +63,6 @@ } ], "files": { - "includes": ["**", "!**/dist", "!**/bin", "!**/node_modules"] + "includes": ["**", "!**/dist", "!**/bin", "!**/node_modules", "!src/libs"] } } diff --git a/package.json b/package.json index e7c25b65..0736f929 100644 --- a/package.json +++ b/package.json @@ -26,8 +26,8 @@ "build": "yarn bundle-node --outfile=bin/rexreplace.cli.node.js && yarn bundle-bun --outfile=bin/rexreplace.cli.bun.js", "build-minify": "yarn build && yarn minify", "minify": "yarn bundle-node --minify --outfile=bin/rexreplace.cli.node.min.js && yarn bundle-bun --minify --outfile=bin/rexreplace.cli.bun.min.js", - "bundle-node": "bun build src/cli.node.ts --target=node --format=cjs --packages=external", - "bundle-bun": "bun build src/cli.bun.ts --target=bun --format=cjs --packages=external", + "bundle-node": "bun build src/cli.node.ts --target=node --format=cjs --external fast-glob --external yargs", + "bundle-bun": "bun build src/cli.bun.ts --target=bun --format=cjs --external fast-glob --external yargs", "prebuild": "rm -fr bin && yarn format", "sync-version": "bun npm/sync-version.js", "test-js": "AGENT=1 bun test ./test/engine ./test/cli ./test/fix", @@ -72,6 +72,8 @@ }, "dependencies": { "fast-glob": "^3.3.3", + "regex": "6.1.0", + "regex-utilities": "2.3.0", "yargs": "16" } } diff --git a/src/cli.ts b/src/cli.ts index c50d68e5..d9beb2f4 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -118,6 +118,15 @@ const yargs = require('yargs') .describe('u', 'Treat pattern as a sequence of unicode code points.') .alias('u', 'unicode') + .string('flags') + .conflicts('flags', ['I', 'G', 'M', 's', 'u']) + // Like the other options, the last one wins when it is given twice + .coerce('flags', value => [].concat(value).pop()) + .describe( + 'flags', + 'The flags of the regex, like `gimsv`, in place of the defaults: `gim`, or `gimvxn` with `-E regex+`. `x` and `n` need `-E regex+`. Not with -G, -I, -M, -s or -u.' + ) + .default('e', 'utf8') .alias('e', 'encoding') .describe('e', 'Encoding of files/piped data.') @@ -127,7 +136,7 @@ const yargs = require('yargs') .string('x') .describe( 'x', - 'Files with a path that matches this regular expression will be excluded from processing. Can be provided multiple times.' + 'Files with a path that matches this regular expression will be excluded from processing. It uses the same engine and flags as the pattern. Can be provided multiple times.' ) .alias('x', 'exclude-re') @@ -141,10 +150,10 @@ const yargs = require('yargs') .alias('E', 'engine') .describe( 'E', - 'What regex engine to use. `default` is the default engine of the runtime (`V8` for Node and `JSC` (JavaScriptCore) for Bun.' + 'The regex engine. `rr` (also `default`) reads patterns like the native engine and adds the extended syntax of Regex+: atomic groups `(?>…)`, possessive quantifiers like `a++`, and subroutines `\\g`. `regex+` is Regex+ with the flags v, x and n. `runtime` is the native engine of the runtime: V8 in Node and JavaScriptCore in Bun.' ) - .choices('E', ['default', 'V8', 'JSC' /*'RE2' /*'sd', 'stream'*/]) - .default('E', 'default') + .choices('E', ['rr', 'default', 'regex+', 'runtime']) + .default('E', 'rr') .boolean('q') .describe( @@ -414,7 +423,7 @@ The following values defaults to \`❌\` if haystack does not originate from a f The numbers: \`$n\`: the number of the file, \`$t\`: the number of files, \`$i\`: the number of the file among the files with a match so far, \`$c\`: among the files changed so far, \`$m\`: the number of the match in the file, \`$M\`: across all files. Each has cousins with zeros in front (\`$n0\` has 2 digits, \`$n00\` 3, and so on to 10) and with a space at the end (\`$n_\`). For speed, the code runs once per file unless it names the match, a group, \`$m\` or \`$M\`. -Access a named group via the name prepended with \`rr_\`. \`(?...)\` gives you \`rr_year\`. A named group name can not end with \`_\`. +Access a named group as \`$\` and the name: \`(?...)\` gives you \`$year\`. A group named like a number, like \`n\`, wins over the number. With the flag \`n\` of \`-E regex+\` only named groups capture, so there is no \`$1\`. All variables, except from module, date objects, \`nl\` and \`_\`, has a corresponding variable name followed by \`_\` where the content has an extra space at the end (for easy concatenation). ` diff --git a/src/engine.ts b/src/engine.ts index 7b961097..6a5c8ce5 100644 --- a/src/engine.ts +++ b/src/engine.ts @@ -24,6 +24,14 @@ const re = { // With a name the location is in parentheses at the end, like `at fn (/path/file.js:2:15)`. stackLineWithoutName: /^\s*at (?:file:\/\/)?((?:\/|[A-Za-z]:[\\/]).*?)(:\d+:\d+)\s*$/, stackLineWithName: /^\s*at .*? \((?:file:\/\/)?(.+?)(:\d+:\d+)\)\s*$/, + subroutineSyntax: /\\g<|\(\?\(DEFINE\)/, + atomicSyntax: /\(\?>/, + // v is a native flag too, but only Regex+ knows x and n + regexPlusOnlyFlags: /[xn]/, + regexFlagNames: /[dgimsuvyxn]/g, + // The flags Regex+ takes as options, and not as flags + regexPlusFlags: /[uvxn]/g, + replacementToken: /\$(\$|&|`|'|<[^>]*>|(\d)(\d?))/g, // Only the chars with a meaning outside a character class. Unicode mode does not allow to escape the others (#686). regexSpecialChars: /[\[\]{}()*+?.\/\\^$|]/g, byteOrSize: /bytes|size/, @@ -39,15 +47,29 @@ const re = { // Only the version ends up in the bundle import {version} from '../package.json'; + +// Regex+ by Steven Levithan. Its subroutines are copied into src/libs until it exports them. +import {rewrite} from 'regex'; +import {atomic, possessive} from 'regex/internals'; +import {Context, hasUnescaped} from 'regex-utilities'; +import {subroutines} from './libs/regex/subroutines.js'; export {version}; // The regex engine comes with the runtime: Node has V8 and Bun has JSC (JavaScriptCore) const runtime = process.versions.bun ? 'Bun' : 'Node'; const runtimeEngine = process.versions.bun ? 'JSC' : 'V8'; +// Node 18 has no v flag, and then Regex+ uses u +const supportsFlagV = (() => { + try { + return Boolean(new RegExp('', 'v')); + } catch (e) { + return false; + } +})(); // Resolves with the status of the run when all files are done. // When the run can not go on, a panic is thrown with the status so far. -export async function engine(config: any = {engine: 'default'}) { +export async function engine(config: any = {engine: 'rr'}) { config.statusInfo = newStatus(config); // A new run has had no panic yet, has read no replacement file, and has its own time, also when the same config is used again config.plzStop = false; @@ -154,23 +176,34 @@ function unpackConfig(config) { // --output-match wins over --output config.output = config.output && !config.outputMatch; - config.regex = getFinalRegex(config.pattern, config); + const {regex, hiddenCaptures, namedOnly} = compileRegex(config.pattern, config); + config.regex = regex; + config.hiddenCaptures = hiddenCaptures; + // With the flag n of Regex+ only named groups capture, so --js gets no $1 and on + config.namedOnly = namedOnly; - config.codeUses = config.replacementJs ? codeUses(config.replacementOri, config.regex) : null; - - // In --js a named group is a variable like rr_year, and rr_year_ has a space at the end. - // A group that ends with _ would have the same name as another group with a space, so it can not be used. - // With --output-match the code is not run, so there are no variables. - const endsWith_ = config.outputMatch - ? null - : config.codeUses?.groups.find(name => name.endsWith('_')); - if (endsWith_) { - panic('A named group can not end with _', endsWith_); + // The user does not know the groups Regex+ adds for atomic groups and possessive quantifiers + if ( + hiddenCaptures.length && + 'string' === typeof config.replacement && + !config.replacementJs && + !config.outputMatch && + !config.literalReplacement + ) { + config.replacement = visibleGroupNumbers( + config.replacement, + hiddenCaptures, + emptyMatch(regex).length - 1 + ); } + config.codeUses = config.replacementJs ? codeUses(config.replacementOri, config.regex) : null; + // What the regex engine is asked to do. With --replacement-js and --output-match it gets a function. if (config.statusInfo.engine) { config.statusInfo.engine.execution = { + engine: engineName(config.engine), + hiddenCaptures, search: config.regex.source, flags: config.regex.flags, replacement: config.replacementJs || config.outputMatch ? null : config.replacement, @@ -298,6 +331,8 @@ function openFile(file, config, number, total) { function doReplacement(file: string, config: any, data: string, number: number, total: number) { verbose('Work on content from: ' + file); + // A regex with the flag y and not g starts where the last file ended, so each file starts from the beginning + config.regex.lastIndex = 0; // Main regexp of the whole thing. With --output-match the replacement prints the matches, so the code of --replacement-js is not run. const result = data.replace( config.regex, @@ -423,6 +458,7 @@ function planRenames(files, config) { config.replacementJs && !config.outputMatch ? codeReplacement(file, config, text, fileNumbers(config, index + 1, files.length)) : config.replacement; + config.regex.lastIndex = 0; const result = text.replace(config.regex, replacement); countFile(config, text, result); if (result === text) { @@ -634,7 +670,7 @@ function matchPrinter(config) { warn('The replacement file is not read with --output-match, as only the matches are printed.'); } return (...args) => { - const [match, ...groups] = splitMatch(args).groups; + const [match, ...groups] = splitMatch(args, config.hiddenCaptures).groups; print( groups.length ? groups.join('') : match, config.contentDelimiter, @@ -689,29 +725,128 @@ function codeReplacement(file, config, text, numbers) { }; } -function getFinalRegex(pattern, config) { - const engine = 'default' === config.engine ? runtimeEngine : config.engine; - // RE2 is not in yet, so it uses the engine of the runtime - if (engine !== runtimeEngine && 'RE2' !== engine) { - panic( - 'V8' === engine || 'JSC' === engine - ? `When running via the ${runtime} runtime you can only use default or ${runtimeEngine} as engine` - : `Engine ${config.engine} not supported` - ); +const engineName = name => ('default' === name ? 'rr' : (name ?? 'rr')); + +// regex+ has v, or u with -u +function regexFlags(config, regexPlus) { + if (null != config.flags) { + // The runtime does not always say which flag it does not know, so it is told here + const unknown = String(config.flags).replace(re.regexFlagNames, ''); + if (unknown) { + panic('Unknown flags in --flags', unknown); + } + // The runtime does not take a flag twice, and there is no harm in it + return [...new Set(String(config.flags))].join(''); } - const flags = + return ( (config.voidGlobal ? '' : 'g') + (config.voidIgnoreCase ? '' : 'i') + (config.voidMultiline ? '' : 'm') + (config.dotAll ? 's' : '') + - (config.unicode ? 'u' : ''); + (regexPlus ? (config.unicode ? 'uxn' : 'vxn') : config.unicode ? 'u' : '') + ); +} + +function compileRegex(pattern, config) { + const engine = engineName(config.engine); + if (!['rr', 'regex+', 'runtime'].includes(engine)) { + panic(`Engine ${config.engine} not supported`); + } + const regexPlus = 'regex+' === engine; + const flags = regexFlags(config, regexPlus); + if (!regexPlus && re.regexPlusOnlyFlags.test(flags)) { + panic('The flags x and n need -E regex+', flags); + } + // The runtime does not say why it does not take them + if (flags.includes('u') && flags.includes('v')) { + panic('The flags u and v can not go together', flags); + } + // Regex+ takes v, u, x and n as options, and gives an error when they come as flags + const nativeFlags = regexPlus ? flags.replace(re.regexPlusFlags, '') : flags; + const unicode = !flags.includes('u') && supportsFlagV ? 'v' : 'u'; try { - return new RegExp(pattern, flags); + if (config.literal) { + // The text as it is, without the free spacing of Regex+ + return { + regex: new RegExp(pattern, nativeFlags + (regexPlus ? unicode : '')), + hiddenCaptures: [], + namedOnly: false, + }; + } + if (regexPlus) { + // The subroutines and the atomic groups are left to extendedSyntax, as rewrite() does not tell which groups they add + const disable = { + v: flags.includes('u'), + x: !flags.includes('x'), + n: !flags.includes('n'), + atomic: true, + subroutines: true, + }; + const result = rewrite(pattern, {flags: nativeFlags, disable}); + const extended = extendedSyntax(result.pattern, result.flags); + return { + regex: new RegExp(extended.pattern, result.flags), + hiddenCaptures: extended.hiddenCaptures, + namedOnly: flags.includes('n'), + }; + } + if ('rr' === engine) { + const extended = extendedSyntax(pattern, flags); + return { + regex: new RegExp(extended.pattern, flags), + hiddenCaptures: extended.hiddenCaptures, + namedOnly: false, + }; + } + return {regex: new RegExp(pattern, flags), hiddenCaptures: [], namedOnly: false}; } catch (e) { return panic(e.message); } } +// Only the transforms of Regex+ run, in its own order, and not its flags, so a pattern that works natively gives the same regex. +// A transform only runs when its syntax is outside a character class, as Regex+ does not always look, like for [(?>]\1 (#858). +function extendedSyntax(pattern, flags) { + let hiddenCaptures = []; + const transform = (plugin, syntax = null) => { + if (syntax && !hasUnescaped(pattern, syntax.source, Context.DEFAULT)) { + return; + } + const result = plugin(pattern, {flags, hiddenCaptures}); + pattern = result.pattern; + hiddenCaptures = result.hiddenCaptures ?? hiddenCaptures; + }; + transform(subroutines, re.subroutineSyntax); + transform(possessive); + transform(atomic, re.atomicSyntax); + return {pattern, hiddenCaptures}; +} + +// Always two digits, so a digit after it is not read as part of the number. A number that is no group of the user stays text, as in javascript. +function visibleGroupNumbers(replacement, hiddenCaptures, total) { + const real = [0]; + for (let group = 1; group <= total; group++) { + if (!hiddenCaptures.includes(group)) { + real.push(group); + } + } + const visible = real.length - 1; + return replacement.replace(re.replacementToken, (token, kind, first, second) => { + if (undefined === first) { + return token; + } + const two = '' !== second ? Number(first + second) : 0; + if (1 <= two && two <= visible) { + return '$' + String(real[two]).padStart(2, '0'); + } + const one = Number(first); + if (1 <= one && one <= visible) { + return '$' + String(real[one]).padStart(2, '0') + second; + } + return '$$' + first + second; + }); +} + function readableSize(size) { if (1 === size) { return '1 Byte'; @@ -739,10 +874,9 @@ function countFile(config, text, result) { function codeUses(code, regex) { const named = name => new RegExp('\\$' + name + '(?![A-Za-z])').test(code); const groups = namedGroups(regex); - // A named group is used as rr_year, or as rr_year_ with a space at the end const usesGroup = name => new RegExp( - '(^|[^\\p{ID_Continue}$])rr_' + name.replace(/\$/g, '\\$') + '_?(?![\\p{ID_Continue}$])', + '(^|[^\\p{ID_Continue}$])\\$' + name.replace(/\$/g, '\\$') + '_?(?![\\p{ID_Continue}$])', 'u' ).test(code); const matchNumbers = named('m') || named('M'); @@ -757,12 +891,15 @@ function codeUses(code, regex) { }; } +// With | at the end the regex matches an empty text, and the match has all the groups +function emptyMatch(regex) { + return new RegExp(regex.source + '|', regex.flags.replace(/[gy]/g, '')).exec(''); +} + // The names of the named groups of a regex, like year for (?…). A name can have any letter, like año. // They come from the regex itself, so text that only looks like a group, like \(? or [(?], is not one. -// With | at the end the regex matches an empty text, and the match has all the groups. function namedGroups(regex) { - const match = new RegExp(regex.source + '|', regex.flags.replace(/[gy]/g, '')).exec(''); - return Object.keys(match?.groups ?? {}); + return Object.keys(emptyMatch(regex)?.groups ?? {}); } // The numbers of --js and their cousins: $n0 is $n with zeros in front to 2 digits, $n00 to 3 and so on to 10, @@ -850,11 +987,18 @@ function cut(text, length = 300) { return chars.length > length ? chars.slice(0, length).join('') + '…' : text; } -// The arguments of a replace function: the match and the groups, then the offset and the text, and the named groups if the pattern has any -function splitMatch(args) { +// The arguments of a replace function: the match and the groups, then the offset and the text, and the named groups if the pattern has any. +// The groups the engine added are left out, so $1 is the first group of the user. +function splitMatch(args, hiddenCaptures) { const named = args[args.length - 1]; const hasNamed = 'object' === typeof named && null !== named; - return {groups: args.slice(0, hasNamed ? -3 : -2), named: hasNamed ? named : {}}; + const groups = args.slice(0, hasNamed ? -3 : -2); + return { + groups: hiddenCaptures.length + ? groups.filter((_, group) => !hiddenCaptures.includes(group)) + : groups, + named: hasNamed ? named : {}, + }; } // The function that runs the code of --js. It is made once, as the names are the same for every file. @@ -998,14 +1142,17 @@ function dynamicReplacement(file, config, text, numbers = {n: 1, t: 1, i: 1, c: foundMatch(); match++; config.numbers.matches++; - const {groups, named} = splitMatch(matchArgs); + const {groups, named} = splitMatch(matchArgs, config.hiddenCaptures); let variables = ''; + // $0 is the match. With the flag n the groups have no numbers, only names. groups.forEach((value, i) => { - variables += 'var $' + i + '=' + JSON.stringify(value) + '; '; + if (!i || !config.namedOnly) { + variables += 'var $' + i + '=' + JSON.stringify(value) + '; '; + } }); - // A named group is a variable with rr_ in front, like rr_year for (?…), and rr_year_ has a space at the end + // After the numbers, so a group named like a number, like n, wins over $n for (const [name, value] of Object.entries(named)) { - variables += `var rr_${name} = ${JSON.stringify(value)}, rr_${name}_ = rr_${name} + ' '; `; + variables += `var $${name} = ${JSON.stringify(value)}, $${name}_ = $${name} + ' '; `; } return runCode({m: match, M: config.numbers.matches}, variables); }; @@ -1041,9 +1188,12 @@ function getFilePaths(conf) { } excludeRe - .map(pattern => getFinalRegex(regexPattern(pattern, conf), conf)) + .map(pattern => compileRegex(regexPattern(pattern, conf), conf).regex) .forEach(exclude => { - files = files.filter(file => !file.match(exclude)); + files = files.filter(file => { + exclude.lastIndex = 0; + return !file.match(exclude); + }); }); files = sortFiles(files, sort); diff --git a/src/libs/README.md b/src/libs/README.md new file mode 100644 index 00000000..33bbc619 --- /dev/null +++ b/src/libs/README.md @@ -0,0 +1,12 @@ +# Copied code + +RexReplace uses [Regex+](https://github.com/slevithan/regex) by Steven Levithan, from the npm packages [`regex`](https://www.npmjs.com/package/regex) and [`regex-utilities`](https://www.npmjs.com/package/regex-utilities). Both are dependencies, and the build bundles them, as they are ES modules only. + +One part of `regex` 6.1.0 is not exported: the subroutines. So it is copied here, as it is upstream, until upstream exports it: + +- `regex/subroutines.js` and `regex/utils-internals.js`, unchanged. +- `regex/utils.js`, only the three parts of it that `subroutines.js` uses. + +The license of `regex` (MIT) is in `regex/LICENSE`. Biome leaves this folder alone, so it stays easy to compare with upstream. + +When `regex` exports `subroutines`, import it from `regex/internals`, and delete this folder and its line in `biome.json`. diff --git a/src/libs/regex/LICENSE b/src/libs/regex/LICENSE new file mode 100644 index 00000000..42eabea2 --- /dev/null +++ b/src/libs/regex/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2025 Steven Levithan + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/src/libs/regex/subroutines.js b/src/libs/regex/subroutines.js new file mode 100644 index 00000000..6ad569a2 --- /dev/null +++ b/src/libs/regex/subroutines.js @@ -0,0 +1,375 @@ +import {capturingDelim, countCaptures, namedCapturingDelim} from './utils.js'; +import {incrementIfAtLeast, spliceStr} from './utils-internals.js'; +import {Context, execUnescaped, forEachUnescaped, getGroupContents, hasUnescaped, replaceUnescaped} from 'regex-utilities'; +/** +@import {PluginData, PluginResult} from './regex.js'; +*/ + +/** +@param {string} expression +@param {PluginData} [data] +@returns {PluginResult} +*/ +function subroutines(expression, data) { + // NOTE: subroutines and definition groups fully support numbered backreferences and unnamed + // captures (from interpolated regexes or from turning implicit flag n off), and all of the + // complex forward and backward backreference adjustments that can result + const namedGroups = getNamedCapturingGroups(expression, {includeContents: true}); + const transformed = processSubroutines(expression, namedGroups, data?.hiddenCaptures ?? []); + return { + pattern: processDefinitionGroup(transformed.pattern, namedGroups), + hiddenCaptures: transformed.hiddenCaptures, + }; +} + +// Explicitly exclude `&` from subroutine name chars because it's used by extension +// `regex-recursion` for recursive subroutines via `\g` +const subroutinePattern = String.raw`\\g<(?[^>&]+)>`; +const token = new RegExp(String.raw` +${subroutinePattern} +| (?${capturingDelim}) +| \\(?[1-9]\d*) +| \\k<(?[^>]+)> +| \\?. +`.replace(/\s+/g, ''), 'gsu'); + +/** +@typedef { + Map} NamedCapturingGroupsMap +*/ +/** +Apply transformations for subroutines: `\g`. +@param {string} expression +@param {NamedCapturingGroupsMap} namedGroups +@param {Array} hiddenCaptures +@returns {PluginResult} +*/ +function processSubroutines(expression, namedGroups, hiddenCaptures) { + if (!/\\g]+>)', Context.DEFAULT); + const subroutineWrapper = hasBackrefs ? '(' : '(?:'; + const openSubroutines = new Map(); + const openSubroutinesStack = []; + const captureNumMap = [0]; + const addedHiddenCaptures = []; + let numCapturesPassedOutsideSubroutines = 0; + let numCapturesPassedInsideSubroutines = 0; + let numCapturesPassedInsideThisSubroutine = 0; + let numSubroutineCapturesTrackedInRemap = 0; + let numCharClassesOpen = 0; + let match; + token.lastIndex = 0; + while (match = token.exec(expression)) { + const {0: m, index, groups: {subroutineName, capturingStart, backrefNum, backrefName}} = match; + if (m === '[') { + numCharClassesOpen++; + } else if (!numCharClassesOpen) { + + if (subroutineName) { + if (!namedGroups.has(subroutineName)) { + throw new Error(`Invalid named capture referenced by subroutine ${m}`); + } + if (openSubroutines.has(subroutineName)) { + throw new Error(`Subroutine ${m} followed a recursive reference`); + } + const contents = namedGroups.get(subroutineName).contents; + // Wrap value in case it has top-level alternation or is followed by a quantifier. The + // wrapper also marks the end of the expanded contents, which we'll track using + // `unclosedGroupCount`. If there are any backrefs in the expression, wrap with `()` + // instead of `(?:)` in case there are backrefs inside the subroutine that refer to their + // containing capturing group + const subroutineValue = `${subroutineWrapper}${contents})`; + if (hasBackrefs) { + numCapturesPassedInsideThisSubroutine = 0; + numCapturesPassedInsideSubroutines++; + updateHiddenCaptureTracking( + hiddenCaptures, + addedHiddenCaptures, + numCapturesPassedOutsideSubroutines + numCapturesPassedInsideSubroutines + ); + } + openSubroutines.set(subroutineName, { + // Incrementally decremented to track when we've left the group + unclosedGroupCount: countOpenParens(subroutineValue), + }); + openSubroutinesStack.push(subroutineName); + // Expand the subroutine's contents into the pattern we're looping over + expression = spliceStr(expression, index, m, subroutineValue); + token.lastIndex -= m.length - subroutineWrapper.length; + } else if (capturingStart) { + // Somewhere within an expanded subroutine + if (openSubroutines.size) { + if (hasBackrefs) { + numCapturesPassedInsideThisSubroutine++; + numCapturesPassedInsideSubroutines++; + updateHiddenCaptureTracking( + hiddenCaptures, + addedHiddenCaptures, + numCapturesPassedOutsideSubroutines + numCapturesPassedInsideSubroutines + ); + } + // Named capturing group + if (m !== '(') { + // Replace named with unnamed capture. Subroutines ideally wouldn't create any new + // captures, but it can't be helped since we need any backrefs to this capture to work. + // Given that flag n prevents unnamed capture and thereby requires you to rely on named + // backrefs and `groups`, switching to unnamed essentially accomplishes not creating a + // capture. Can fully avoid capturing if there are no backrefs in the expression + expression = spliceStr(expression, index, m, subroutineWrapper); + token.lastIndex -= m.length - subroutineWrapper.length; + } + } else if (hasBackrefs) { + captureNumMap.push( + lastOf(captureNumMap) + 1 + + numCapturesPassedInsideSubroutines - + numSubroutineCapturesTrackedInRemap + ); + numSubroutineCapturesTrackedInRemap = numCapturesPassedInsideSubroutines; + numCapturesPassedOutsideSubroutines++; + } + } else if ((backrefNum || backrefName) && openSubroutines.size) { + // Unify handling for named and unnamed by always using the backref num + const num = backrefNum ? +backrefNum : namedGroups.get(backrefName)?.groupNum; + let isGroupFromThisSubroutine = false; + // Search for the group in the contents of the subroutine stack + for (const s of openSubroutinesStack) { + const group = namedGroups.get(s); + if (num >= group.groupNum && num <= (group.groupNum + group.numCaptures)) { + isGroupFromThisSubroutine = true; + break; + } + } + if (isGroupFromThisSubroutine) { + const group = namedGroups.get(lastOf(openSubroutinesStack)); + // Replace the backref with metadata we'll need to rewrite it later, using + // `\k<$$bNsNrNcN>` as a temporary wrapper: + // - b: The unmodified matched backref num, or the corresponding num of a named backref + // - s: The capture num of the subroutine we're most deeply nested in, including captures + // added by expanding the contents of preceding subroutines + // - r: The original capture num of the group that the subroutine we're most deeply + // nested in references, not counting the effects of subroutines + // - c: The number of captures within `r`, not counting the effects of subroutines + const subroutineNum = numCapturesPassedOutsideSubroutines + numCapturesPassedInsideSubroutines - numCapturesPassedInsideThisSubroutine; + const metadata = `\\k<$$b${num}s${subroutineNum}r${group.groupNum}c${group.numCaptures}>`; + expression = spliceStr(expression, index, m, metadata); + token.lastIndex += metadata.length - m.length; + } + } else if (m === ')') { + if (openSubroutines.size) { + const subroutine = openSubroutines.get(lastOf(openSubroutinesStack)); + subroutine.unclosedGroupCount--; + if (!subroutine.unclosedGroupCount) { + openSubroutines.delete(openSubroutinesStack.pop()); + } + } + } + + } else if (m === ']') { + numCharClassesOpen--; + } + } + + hiddenCaptures.push(...addedHiddenCaptures); + + if (hasBackrefs) { + // Second pass to adjust backrefs + expression = replaceUnescaped( + expression, + String.raw`\\(?:(?[1-9]\d*)|k<\$\$b(?\d+)s(?\d+)r(?\d+)c(?\d+)>)`, + ({0: m, groups: {bNum, bNumSub, subNum, refNum, refCaps}}) => { + if (bNum) { + const backrefNum = +bNum; + if (backrefNum > captureNumMap.length - 1) { + throw new Error(`Backref "${m}" greater than number of captures`); + } + return `\\${captureNumMap[backrefNum]}`; + } + const backrefNumInSubroutine = +bNumSub; + const subroutineGroupNum = +subNum; + const refGroupNum = +refNum; + const numCapturesInRef = +refCaps; + if (backrefNumInSubroutine < refGroupNum || backrefNumInSubroutine > (refGroupNum + numCapturesInRef)) { + return `\\${captureNumMap[backrefNumInSubroutine]}`; + } + return `\\${subroutineGroupNum - refGroupNum + backrefNumInSubroutine}`; + }, + Context.DEFAULT + ); + } + + return { + pattern: expression, + hiddenCaptures, + }; +} + +// `(?:)` allowed because it can be added by flag x's preprocessing of whitespace and comments +const defineGroupToken = new RegExp(String.raw`${namedCapturingDelim}|\(\?:\)|(?\\?.)`, 'gsu'); + +/** +Remove valid subroutine definition groups: `(?(DEFINE)…)`. +@param {string} expression +@param {NamedCapturingGroupsMap} namedGroups +IMPORTANT: Avoid using the `contents` property of `namedGroups` objects, because at this point +subroutine substitution has been performed on the corresponding substrings in `expression` +@returns {string} +*/ +function processDefinitionGroup(expression, namedGroups) { + const defineMatch = execUnescaped(expression, String.raw`\(\?\(DEFINE\)`, 0, Context.DEFAULT); + if (!defineMatch) { + return expression; + } + const defineGroup = getGroup(expression, defineMatch); + if (defineGroup.afterPos < expression.length) { + // Supporting DEFINE at positions other than the end would complicate backref handling. + // NOTE: Flag x's preprocessing permits trailing whitespace and comments + throw new Error('DEFINE group allowed only at the end of a regex'); + } else if (defineGroup.afterPos > expression.length) { + throw new Error('DEFINE group is unclosed'); + } + let match; + defineGroupToken.lastIndex = 0; + while (match = defineGroupToken.exec(defineGroup.contents)) { + const {captureName, invalid} = match.groups; + if (captureName) { + const group = getGroup(defineGroup.contents, match); + let duplicateName; + if (!namedGroups.get(captureName).isUnique) { + duplicateName = captureName; + } else { + const nestedNamedGroups = getNamedCapturingGroups(group.contents, {includeContents: false}); + for (const name of nestedNamedGroups.keys()) { + if (!namedGroups.get(name).isUnique) { + duplicateName = name; + break; + } + } + } + if (duplicateName) { + throw new Error(`Duplicate group name "${duplicateName}" within DEFINE`); + } + defineGroupToken.lastIndex = group.afterPos; + } else if (invalid) { + // Since a DEFINE group is stripped from its expression, we can't easily determine whether + // unreferenced top-level syntax within it is valid. Such syntax serves no purpose, so it's + // easiest to not allow it + throw new Error(`DEFINE group includes unsupported syntax at top level`); + } + } + return expression.slice(0, defineMatch.index); +} + +/** +Counts unescaped open parens outside of character classes, regardless of group type +@param {string} expression +@returns {number} +*/ +function countOpenParens(expression) { + let num = 0; + forEachUnescaped(expression, '\\(', () => num++, Context.DEFAULT); + return num; +} + +/** +@param {string} expression +@param {string} groupName +@returns {number} +*/ +function getCaptureNum(expression, groupName) { + let num = 0; + let pos = 0; + let match; + while (match = execUnescaped(expression, capturingDelim, pos, Context.DEFAULT)) { + const {0: m, index, groups: {captureName}} = match; + num++; + if (captureName === groupName) { + break; + } + pos = index + m.length; + } + return num; +} + +/** +@param {string} expression +@param {RegExpExecArray} delimMatch +@returns {{contents: string; afterPos: number}} +*/ +function getGroup(expression, delimMatch) { + const contentsStart = delimMatch.index + delimMatch[0].length; + const contents = getGroupContents(expression, contentsStart); + const afterPos = contentsStart + contents.length + 1; + return { + contents, + afterPos, + }; +} + +/** +@param {string} expression +@param {{includeContents: boolean}} options +@returns {NamedCapturingGroupsMap} +*/ +function getNamedCapturingGroups(expression, {includeContents}) { + const namedGroups = new Map(); + forEachUnescaped( + expression, + namedCapturingDelim, + ({0: m, index, groups: {captureName}}) => { + // If there are duplicate capture names, subroutines refer to the first instance of the given + // group (matching the behavior of PCRE and Perl) + if (namedGroups.has(captureName)) { + namedGroups.get(captureName).isUnique = false; + } else { + const group = {isUnique: true}; + if (includeContents) { + const contents = getGroupContents(expression, index + m.length); + Object.assign(group, { + contents, + groupNum: getCaptureNum(expression, captureName), + numCaptures: countCaptures(contents), + }); + } + namedGroups.set(captureName, group); + } + }, + Context.DEFAULT + ); + return namedGroups; +} + +/** +@param {Array} arr +@returns {any} +*/ +function lastOf(arr) { + // Remove when support for ES2022 array method `at` (Node.js 16.6) is no longer an issue: + // + return arr[arr.length - 1]; +} + +/** +@param {Array} hiddenCaptures +@param {Array} addedHiddenCaptures +@param {number} addedCaptureNum +*/ +function updateHiddenCaptureTracking(hiddenCaptures, addedHiddenCaptures, addedCaptureNum) { + addedHiddenCaptures.push(addedCaptureNum); + incrementIfAtLeast(hiddenCaptures, addedCaptureNum); +} + +export { + subroutines, +}; diff --git a/src/libs/regex/utils-internals.js b/src/libs/regex/utils-internals.js new file mode 100644 index 00000000..5fe3581c --- /dev/null +++ b/src/libs/regex/utils-internals.js @@ -0,0 +1,33 @@ +// Separating some utils for improved tree shaking of the `./internals` export + +const noncapturingDelim = String.raw`\(\?(?:[:=!>A-Za-z\-]|<[=!]|\(DEFINE\))`; + +/** +Updates the array in place by incrementing each value greater than or equal to the threshold. +@param {Array} arr +@param {number} threshold +*/ +function incrementIfAtLeast(arr, threshold) { + for (let i = 0; i < arr.length; i++) { + if (arr[i] >= threshold) { + arr[i]++; + } + } +} + +/** +@param {string} str +@param {number} pos +@param {string} oldValue +@param {string} newValue +@returns {string} +*/ +function spliceStr(str, pos, oldValue, newValue) { + return str.slice(0, pos) + newValue + str.slice(pos + oldValue.length); +} + +export { + incrementIfAtLeast, + noncapturingDelim, + spliceStr, +}; diff --git a/src/libs/regex/utils.js b/src/libs/regex/utils.js new file mode 100644 index 00000000..e5101b03 --- /dev/null +++ b/src/libs/regex/utils.js @@ -0,0 +1,21 @@ +// rexreplace: the parts of utils.js of regex 6.1.0 that subroutines.js uses, copied as they are +import {Context, forEachUnescaped} from 'regex-utilities'; + +const namedCapturingDelim = String.raw`\(\?<(?![=!])(?[^>]+)>`; +const capturingDelim = String.raw`\((?!\?)(?!(?<=\(\?\()DEFINE\))|${namedCapturingDelim}`; + +/** +@param {string} expression +@returns {number} +*/ +function countCaptures(expression) { + let num = 0; + forEachUnescaped(expression, capturingDelim, () => num++, Context.DEFAULT); + return num; +} + +export { + capturingDelim, + countCaptures, + namedCapturingDelim, +}; diff --git a/test/cli/debug.test.js b/test/cli/debug.test.js index f5754ab6..534b7535 100644 --- a/test/cli/debug.test.js +++ b/test/cli/debug.test.js @@ -16,6 +16,8 @@ describe('--debug', () => { expect(status.files).toEqual({searched: ['my.file'], updated: ['my.file']}); expect(status.engine.config).toMatchObject({pattern: 'o', replacement: 'x'}); expect(status.engine.execution).toEqual({ + engine: 'rr', + hiddenCaptures: [], search: 'o', flags: 'gim', replacement: 'x', diff --git a/test/cli/run.sh b/test/cli/run.sh index 88777bdf..85b3ce6c 100644 --- a/test/cli/run.sh +++ b/test/cli/run.sh @@ -74,14 +74,13 @@ assert "rexreplace o x my.file --output" "fxxbar" assert "cat my.file" "foobar" -# --engine: `default` is the engine of the runtime: V8 with Node and JSC with Bun +# --engine: `rr` (also `default`) reads patterns like the native engine, with the extended syntax of Regex+, like atomic groups. +# `runtime` is the native engine: V8 with Node and JSC with Bun. reset assert "rexreplace o x my.file --output --engine default" "fxxbar" -if [[ "$(rexreplace --version)" == *+bun ]]; then - assert_failure "rexreplace o x my.file --output --engine V8" -else - assert_failure "rexreplace o x my.file --output --engine JSC" -fi +assert "rexreplace '(?>o+)(b)' '[\$1]' my.file --output" "f[b]ar" +assert "rexreplace o x my.file --output --engine runtime" "fxxbar" +assert_failure "rexreplace o x my.file --output --engine V8" # --encoding diff --git a/test/engine/engine.test.js b/test/engine/engine.test.js index 7e8d71d8..7e2c2f5b 100644 --- a/test/engine/engine.test.js +++ b/test/engine/engine.test.js @@ -3,23 +3,12 @@ import fs from 'fs'; import {rr} from '../helper.js'; describe('--engine', () => { - test('default is the engine of the runtime', async () => { + test('default is rr', async () => { expect((await rr('o', 'x', {pipedData: 'foobar', engine: 'default'})).out).toBe('fxxbar'); }); - // The tests are run by Bun, and Bun has JSC - test('JSC does the job with Bun as the runtime', async () => { - expect((await rr('o', 'x', {pipedData: 'foobar', engine: 'JSC'})).out).toBe('fxxbar'); - }); - - test('stops with an error if the engine is the one of another runtime', async () => { - fs.writeFileSync('my.file', 'foobar'); - const result = await rr('o', 'x', {includeGlob: ['my.file'], engine: 'V8'}); - expect(result.exit).toBe(3); - expect(result.err).toContain( - 'When running via the Bun runtime you can only use default or JSC as engine' - ); - expect(fs.readFileSync('my.file', 'utf8')).toBe('foobar'); + test('runtime is the native engine', async () => { + expect((await rr('o', 'x', {pipedData: 'foobar', engine: 'runtime'})).out).toBe('fxxbar'); }); test('stops with an error if the engine is unknown', async () => { @@ -31,14 +20,14 @@ describe('--engine', () => { }); test('is case sensitive', async () => { - expect((await rr('o', 'x', {pipedData: 'foobar', engine: 'jsc'})).exit).toBe(3); + expect((await rr('o', 'x', {pipedData: 'foobar', engine: 'Runtime'})).exit).toBe(3); }); describe('combined', () => { - test('JSC with all the regex flags', async () => { + test('runtime with all the regex flags', async () => { const config = { pipedData: 'Foo\nfoo 🦖', - engine: 'JSC', + engine: 'runtime', voidIgnoreCase: true, voidGlobal: true, voidMultiline: true, diff --git a/test/fix/705.test.js b/test/fix/705.test.js index 7baeea14..ea709163 100644 --- a/test/fix/705.test.js +++ b/test/fix/705.test.js @@ -55,7 +55,7 @@ describe('#705 the engine returns a promise that resolves when all files are don {includeGlob: ['nope.file'], quietTotal: true}, ]) { const done = engine({ - engine: 'default', + engine: 'rr', encoding: 'utf8', pipedData: null, includeGlob: ['my.file'], diff --git a/test/fix/742.test.js b/test/fix/742.test.js index bc93c2a9..5693cbcb 100644 --- a/test/fix/742.test.js +++ b/test/fix/742.test.js @@ -141,6 +141,8 @@ describe('#742 the engine part of the status', () => { ...flags, }); expect(status.engine.execution).toEqual({ + engine: 'rr', + hiddenCaptures: [], search: 'o+', flags: 'gi', replacement, diff --git a/test/fix/799.test.js b/test/fix/799.test.js index d7c4ddf8..4c097f08 100644 --- a/test/fix/799.test.js +++ b/test/fix/799.test.js @@ -9,16 +9,16 @@ describe('#799 groups in --js', () => { expect((await rr('(?a)(b)', '$1 + $2 + typeof $3', config)).out).toBe('abundefined'); }); - test('a named group is a variable with rr_ in front, and _ at the end has a space', async () => { + test('a named group is a variable with $ in front, and _ at the end has a space', async () => { const config = {pipedData: '2024-05 1999-12', replacementJs: true}; const pattern = '(?\\d{4})-(?\\d\\d)'; - expect((await rr(pattern, 'rr_month + "/" + rr_year', config)).out).toBe('05/2024 12/1999'); - expect((await rr(pattern, 'rr_month_ + rr_year', config)).out).toBe('05 2024 12 1999'); + expect((await rr(pattern, '$month + "/" + $year', config)).out).toBe('05/2024 12/1999'); + expect((await rr(pattern, '$month_ + $year', config)).out).toBe('05 2024 12 1999'); }); test('a named group that took no part in the match is undefined', async () => { const config = {pipedData: 'ab', replacementJs: true}; - expect((await rr('(?a)|(?b)', 'String(rr_x) + String(rr_y) + ","', config)).out).toBe( + expect((await rr('(?a)|(?b)', 'String($x) + String($y) + ","', config)).out).toBe( 'aundefined,undefinedb,' ); }); diff --git a/test/fix/813.test.js b/test/fix/813.test.js index 2a76e2ca..d782252a 100644 --- a/test/fix/813.test.js +++ b/test/fix/813.test.js @@ -30,7 +30,7 @@ describe('#813 a run that stops', () => { test('tells the panic of a second run with the same config again (#844)', async () => { const config = { - engine: 'default', + engine: 'rr', encoding: 'utf8', pipedData: 'foo', includeGlob: [], diff --git a/test/fix/815.test.js b/test/fix/815.test.js index 9a9028e0..18504a75 100644 --- a/test/fix/815.test.js +++ b/test/fix/815.test.js @@ -10,7 +10,7 @@ describe('#815 --js code can declare its own variables', () => { ['a function', 'o', 'function $t() { return "F"; } $t()', 'fFF'], ['a number with let', 'o', 'let $t = "T"; $t', 'fTT'], ['a group with const', '(o)', 'const $1 = "X"; $1', 'fXX'], - ['a named group with let', '(?o)', 'let rr_o = "N"; rr_o', 'fNN'], + ['a named group with let', '(?o)', 'let $o = "N"; $o', 'fNN'], ])('with the name of %s', async (name, pattern, code, out) => { expect((await rr(pattern, code, {pipedData: 'foo', replacementJs: true})).out).toBe(out); }); diff --git a/test/fix/816.test.js b/test/fix/816.test.js index 40bae980..e5c7ea67 100644 --- a/test/fix/816.test.js +++ b/test/fix/816.test.js @@ -5,21 +5,19 @@ import {rr} from '../helper.js'; describe('#816 named groups with letters that are not ASCII', () => { test.each([ - ['año', '(?\\d)', '"[" + rr_año + "]"', 'a[1]b[2]'], - ['日', '(?<日>\\d)', 'rr_日 * 2', 'a2b4'], - ['año with a space at the end', '(?\\d)', 'rr_año_ + "|"', 'a1 |b2 |'], - ['𝒜, outside the BMP', '(?<𝒜>\\d)', 'rr_𝒜 * 2', 'a2b4'], - ['$é, with $', '(?<$é>\\d)', 'rr_$é * 2', 'a2b4'], - ['Æble, in uppercase', '(?<Æble>\\d)', 'rr_Æble * 2', 'a2b4'], - ['x٣, with a digit that is not ASCII', '(?\\d)', 'rr_x٣ * 2', 'a2b4'], + ['año', '(?\\d)', '"[" + $año + "]"', 'a[1]b[2]'], + ['日', '(?<日>\\d)', '$日 * 2', 'a2b4'], + ['año with a space at the end', '(?\\d)', '$año_ + "|"', 'a1 |b2 |'], + ['𝒜, outside the BMP', '(?<𝒜>\\d)', '$𝒜 * 2', 'a2b4'], + ['$é, with $', '(?<$é>\\d)', '$$é * 2', 'a2b4'], + ['Æble, in uppercase', '(?<Æble>\\d)', '$Æble * 2', 'a2b4'], + ['x٣, with a digit that is not ASCII', '(?\\d)', '$x٣ * 2', 'a2b4'], ])('can be used, like %s', async (name, pattern, code, out) => { expect((await rr(pattern, code, {pipedData: 'a1b2', replacementJs: true})).out).toBe(out); }); - test('can not end with _ either', async () => { - const result = await rr('(?x)', 'rr_año', {pipedData: 'x', replacementJs: true}); - expect(result.err).toStartWith(' ❌ A named group can not end with _\n año_'); - expect(result.exit).toBe(3); + test('can end with _', async () => { + expect((await rr('(?x)', '$año_', {pipedData: 'x', replacementJs: true})).out).toBe('x'); }); test('are read from the regex, so text that only looks like a group is not one', async () => { @@ -28,14 +26,14 @@ describe('#816 named groups with letters that are not ASCII', () => { expect((await rr('[(?]', '"z"', {...config, pipedData: 'ab'})).out).toBe('zb'); }); - test('are only seen as the whole name, so rr_rr_year is not the group year', async () => { - const result = await rr('(?\\d)', 'rr_rr_year', {pipedData: '1', replacementJs: true}); - expect(result.err).toContain('ReferenceError: rr_rr_year is not defined'); + test('are only seen as the whole name, so $years is not the group year', async () => { + const result = await rr('(?\\d)', '$years', {pipedData: '1', replacementJs: true}); + expect(result.err).toContain('ReferenceError: $years is not defined'); expect(result.exit).toBe(3); }); - test('can be used with a space at the end on its own, like rr_year_', async () => { + test('can be used with a space at the end on its own, like $year_', async () => { const config = {pipedData: '2024', replacementJs: true}; - expect((await rr('(?\\d+)', 'rr_year_ + "|"', config)).out).toBe('2024 |'); + expect((await rr('(?\\d+)', '$year_ + "|"', config)).out).toBe('2024 |'); }); }); diff --git a/test/fix/825.test.js b/test/fix/825.test.js index d9e9e4f8..cd668437 100644 --- a/test/fix/825.test.js +++ b/test/fix/825.test.js @@ -2,30 +2,29 @@ // A named group was a variable like _year, which could clash: with groups a and a_, _a_ was both a_ and a with a space, // and a group named _require_ broke r(). import {describe, test, expect} from 'bun:test'; -import fs from 'fs'; import {rr} from '../helper.js'; const js = {pipedData: 'in 1999 and 2024', replacementJs: true}; describe('#825 named groups in --js', () => { test.each([ - ['a name in lowercase', '(?\\d+)', '"<" + rr_year + ">"', 'in <1999> and <2024>'], + ['a name in lowercase', '(?\\d+)', '"<" + $year + ">"', 'in <1999> and <2024>'], [ 'the cousin with a space at the end', '(?\\d+)', - 'rr_year + "," + rr_year_', + '$year + "," + $year_', 'in 1999,1999 and 2024,2024 ', ], - ['a name in uppercase', '(?\\d+)', '"Y" + rr_Year', 'in Y1999 and Y2024'], - ['a name with a number', '(?\\d+)', '"Y" + rr_y2k', 'in Y1999 and Y2024'], - ['a name with $', '(?\\d+)', '"Y" + rr_a$', 'in Y1999 and Y2024'], - ['two names that only differ by case', '(?\\d)(?\\d)', 'rr_Y + rr_y', 'in 9199 and 0242'], - ['a name next to its number', '(?\\d+)', '$1 + rr_year', 'in 19991999 and 20242024'], + ['a name in uppercase', '(?\\d+)', '"Y" + $Year', 'in Y1999 and Y2024'], + ['a name with a number', '(?\\d+)', '"Y" + $y2k', 'in Y1999 and Y2024'], + ['a name with $', '(?\\d+)', '"Y" + $a$', 'in Y1999 and Y2024'], + ['two names that only differ by case', '(?\\d)(?\\d)', '$Y + $y', 'in 9199 and 0242'], + ['a name next to its number', '(?\\d+)', '$1 + $year', 'in 19991999 and 20242024'], ])('gives %s', async (name, pattern, code, out) => { expect(await rr(pattern, code, js)).toEqual({out, err: '', exit: null}); }); - test('have rr_ in front, so they can not clash with what RexReplace uses', async () => { + test('have $ in front, so they can not clash with what RexReplace uses', async () => { // With _ in front, the group _require with a space was __require_, which r() uses itself. // $1 makes the code run once per match, where the variables of the groups are made. const config = {pipedData: 'ab', replacementJs: true}; @@ -37,30 +36,15 @@ describe('#825 named groups in --js', () => { }); }); -describe('#825 a named group that ends with _', () => { - test.each([['a_'], ['Year_'], ['_'], ['y2_']])( - 'panics for %s, as that is the name of another group with a space, and nothing is changed', - async name => { - fs.writeFileSync('a.md', 'in 1999'); - const result = await rr(`(?<${name}>\\d+)`, '"x"', { - includeGlob: ['a.md'], - replacementJs: true, - }); - expect(result).toEqual({ - out: '', - err: ` ❌ A named group can not end with _\n ${name}\n No further processing.\n\n`, - exit: 3, - }); - expect(fs.readFileSync('a.md', 'utf8')).toBe('in 1999'); - } - ); - - test('is fine with --output-match, as the code is not run', async () => { - const config = {pipedData: 'ab', replacementJs: true, outputMatch: true}; - expect((await rr('(?a)', 'x', config)).exit).toBe(null); +describe('#825 a named group with the name of another variable', () => { + // The variables of the groups come after the others, so the group wins + test('wins over the number, like n over $n, but not over its cousins with zeros', async () => { + expect((await rr('(?\\w)', '$n + $n_ + $n00', {...js, pipedData: 'ab'})).out).toBe( + 'aa 001bb 001' + ); }); - test('is fine without --js, as there are no variables then', async () => { - expect((await rr('(?o)', 'x', {pipedData: 'foo'})).out).toBe('fxx'); + test('can end with _, and then wins over the cousin with a space of the group without it', async () => { + expect((await rr('(?\\d)(?\\w)', '$a + $a_', {...js, pipedData: '1x'})).out).toBe('1x'); }); }); diff --git a/test/fix/854.test.js b/test/fix/854.test.js new file mode 100644 index 00000000..3b4d2064 --- /dev/null +++ b/test/fix/854.test.js @@ -0,0 +1,253 @@ +// https://github.com/mathiasrw/rexreplace/issues/854 +// Regex+ is behind the engines. rr, the default, reads patterns like the native engine, with the extended syntax of Regex+ on top. +// regex+ is Regex+ with the flags v, x and n, and --flags sets all the flags of any engine. +import {describe, test, expect} from 'bun:test'; +import fs from 'fs'; +import {$} from 'bun'; +import {rr, rrStatus, sh, cli} from '../helper.js'; + +// What the regex engine was asked to do: the source and the flags of the regex +async function compiled(pattern, config = {}) { + const status = await rrStatus(pattern, 'x', { + pipedData: '', + verbose: true, + status: true, + ...config, + }); + return status.engine.execution; +} + +const out = async (pattern, replacement, text, config = {}) => + (await rr(pattern, replacement, {pipedData: text, ...config})).out; + +describe('#854 the rr engine', () => { + test.each([ + [`(['"])([^\\1]+)\\1`], + [`\\"`], + ['a]'], + ['a{'], + ['\\01'], + ['[(]'], + ['[a-z-]'], + ['\\++'], + ['[+]+'], + ['\\{2}+'], + ['[(?>]'], + // #858 + ['[(?>]\\1'], + ['[\\g]'], + ['[a++]'], + ['\\(?>'], + ['(?a)\\k'], + ['\\k'], + ['a+?'], + ['x{1,2}?'], + ['(foo)(.*)'], + ['[\\]+]+'], + ['\\c'], + ['\\x4'], + ['(?x)\\1'], + ['$'], + ['[[]'], + ])('gives the same regex as the native engine for %s', async pattern => { + const native = new RegExp(pattern, 'gim'); + expect(await compiled(pattern)).toMatchObject({search: native.source, flags: 'gim'}); + }); + + test('only uses the u flag with -u', async () => { + expect((await compiled('a', {unicode: true})).flags).toBe('gimu'); + }); + + test.each([ + ['an atomic group', '(?>a+)(b)', '[$1]', 'aab', '[b]'], + ['a possessive quantifier', 'a++b', 'X', 'aaab', 'X'], + ['a possessive quantifier that gives nothing back', 'a++a', 'X', 'aaa', 'aaa'], + ['a subroutine', '(?\\d)-\\g', 'X', '1-2', 'X'], + [ + 'a DEFINE group', + '\\b(?\\g\\.\\g\\.\\g)(?(DEFINE)(?0|[1-9]\\d*))', + 'v$', + '3.12.0 and 01.2.3', + 'v3.12.0 and 01.2.3', + ], + ])('reads %s', async (name, pattern, replacement, text, result) => { + expect(await out(pattern, replacement, text)).toBe(result); + }); + + test('is the engine for default too', async () => { + expect(await out('(?>a+)(b)', '[$1]', 'aab', {engine: 'default'})).toBe('[b]'); + }); +}); + +describe('#854 the groups Regex+ adds', () => { + // (?>a+) adds a group of its own in front of (b) and (c) + test.each([ + ['$1', 'b'], + ['$2$1', 'cb'], + ['$3', '$3'], + ['$10', 'b0'], + ['$$1', '$1'], + ['$&', 'aabc'], + ])('are not counted in the replacement %s', async (replacement, result) => { + expect(await out('(?>a+)(b)(c)?', replacement, 'aabc')).toBe(result); + }); + + test('are not counted in --js', async () => { + expect(await out('(?>a+)(b)', "$1 + '!'", 'aab', {replacementJs: true})).toBe('b!'); + }); + + test('are not printed by --output-match', async () => { + expect(await out('(?>a+)(b)', 'x', 'aab', {outputMatch: true})).toBe('b\n'); + }); + + test('are not hidden for the unnamed groups a subroutine copies, as the README says', async () => { + expect(await out('(?

(a)(b))\\g

', '[$4$5]', 'abab')).toBe('[ab]'); + }); + + test('are told in the status report', async () => { + expect(await compiled('(?>a+)(b)')).toMatchObject({engine: 'rr', hiddenCaptures: [1]}); + }); +}); + +describe('#854 the other engines', () => { + // The tests are run by Bun, so the native engine is JSC + test.each([['runtime']])('%s is the native engine, without the extended syntax', async engine => { + expect(await out('(?a)\\g', 'X', 'ag', {engine})).toBe('X'); + }); + + test.each([['regex+']])( + '%s has free spacing, only named groups capture, and the v flag', + async engine => { + expect(await out('foo bar', 'X', 'foobar', {engine})).toBe('X'); + expect(await out('(foo)(?.*)', '$1', 'foox', {engine})).toBe('x'); + expect((await compiled('a', {engine})).flags).toContain('v'); + } + ); + + test.each([['V'], ['V8'], ['JSC'], ['regex+n'], ['regex+x'], ['regex+u'], ['regex+v']])( + '%s is not an engine', + async engine => { + expect(await rr('a', 'b', {pipedData: 'a', engine})).toEqual({ + out: '', + err: ` ❌ Engine ${engine} not supported\n No further processing.\n\n`, + exit: 3, + }); + } + ); + + test('regex+ stops the run on an escape that is not strict, and the file is not changed', async () => { + fs.writeFileSync('a.txt', 'a"b'); + const result = await rr('\\"', 'x', {includeGlob: ['a.txt'], engine: 'regex+'}); + expect(result.exit).toBe(3); + expect(result.err).toStartWith(' ❌ Invalid regular expression'); + expect(fs.readFileSync('a.txt', 'utf8')).toBe('a"b'); + }); + + test('--literal gives the text as it is, also with the free spacing of regex+', async () => { + expect(await out('a b #c (d)', 'X', 'a b #c (d)!', {engine: 'regex+', literal: true})).toBe( + 'X!' + ); + }); +}); + +describe('#854 --flags', () => { + const plus = {engine: 'regex+'}; + + test('leaves free spacing out of regex+ without x', async () => { + expect(await out('foo bar', 'X', 'foo bar', {...plus, flags: 'gimvn'})).toBe('X'); + }); + + test('lets all groups of regex+ capture without n', async () => { + expect(await out('(foo)', '[$1]', 'foo', {...plus, flags: 'gimvx'})).toBe('[foo]'); + }); + + test('sets all the flags, so without i the search is case sensitive and without g only the first match', async () => { + expect(await out('o', 'x', 'oOo', {flags: 'm'})).toBe('xOo'); + }); + + test('with y, starts each file from the beginning', async () => { + const names = ['a1.txt', 'a2.txt', 'a3.txt']; + names.forEach(name => fs.writeFileSync(name, 'axx')); + await rr('a', '_', {includeGlob: names, flags: 'y'}); + expect(names.map(name => fs.readFileSync(name, 'utf8'))).toEqual(['_xx', '_xx', '_xx']); + }); + + test('with y, checks each path of --exclude-re from the beginning', async () => { + const names = ['a1.txt', 'a2.txt', 'b.txt']; + names.forEach(name => fs.writeFileSync(name, 'o')); + const result = await rr('o', 'x', { + includeGlob: names, + excludeRe: ['a'], + flags: 'y', + simulate: true, + }); + expect(result.out).toBe('b.txt\n'); + }); + + test('with y, starts each name of --rename from the beginning', async () => { + ['a1.txt', 'a2.txt'].forEach(name => fs.writeFileSync(name, 'o')); + await rr('a', 'b', {includeGlob: ['a1.txt', 'a2.txt'], flags: 'y', rename: true}); + expect(fs.readdirSync('.').sort()).toEqual(['b1.txt', 'b2.txt']); + }); + + test('takes a flag that is given twice once', async () => { + expect(await out('o', 'x', 'oO', {flags: 'gg'})).toBe('xO'); + }); + + test('gives regex+ the flag v when it has neither v nor u', async () => { + expect((await compiled('a', {...plus, flags: 'gx'})).flags).toContain('v'); + }); + + test('takes the last one when it is given twice in the CLI', async () => { + fs.writeFileSync('a.txt', 'Aa'); + expect((await sh($`bun ${cli} a _ a.txt --flags g --flags i -o`)).out).toBe('_a\n'); + }); + + test('can not go with -I, -G, -M, -s or -u in the CLI', async () => { + fs.writeFileSync('a.txt', 'a'); + const result = await sh($`bun ${cli} a b a.txt --flags g -I`); + expect(result.exit).toBe(3); + expect(result.err).toEndWith('Arguments flags and I are mutually exclusive\n'); + expect(fs.readFileSync('a.txt', 'utf8')).toBe('a'); + }); + + test('gives regex+ the flag u in place of v with -u too', async () => { + const {flags} = await compiled('a', {engine: 'regex+', unicode: true}); + expect(flags).toContain('u'); + expect(flags).not.toContain('v'); + }); + + test('gives regex+ the flag u in place of v', async () => { + const {flags} = await compiled('a', {...plus, flags: 'gu'}); + expect(flags).toContain('u'); + expect(flags).not.toContain('v'); + }); + + test.each([ + ['x without regex+', {flags: 'gx'}, ' ❌ The flags x and n need -E regex+\n gx'], + ['n without regex+', {flags: 'gn'}, ' ❌ The flags x and n need -E regex+\n gn'], + ['a flag that is not a flag', {flags: 'gqz'}, ' ❌ Unknown flags in --flags\n qz'], + ['u and v', {flags: 'guv'}, ' ❌ The flags u and v can not go together\n guv'], + ])('stops the run with %s, and says so', async (name, config, message) => { + expect(await rr('a', 'b', {pipedData: 'a', ...config})).toEqual({ + out: '', + err: message + '\n No further processing.\n\n', + exit: 3, + }); + }); +}); + +describe('#854 --js with the flag n of regex+', () => { + const config = {engine: 'regex+', replacementJs: true}; + + test('has the named groups and $0, but no group numbers', async () => { + expect(await out('(?\\d)(?\\d)', '$m + $y + $0', '12', config)).toBe('2112'); + expect((await rr('(?\\d)', '$1', {pipedData: '1', ...config})).err).toContain( + 'ReferenceError: $1 is not defined' + ); + }); + + test('has the group numbers again without n', async () => { + expect(await out('(\\d)(\\d)', '$2 + $1', '12', {...config, flags: 'gimvx'})).toBe('21'); + }); +}); diff --git a/test/helper.js b/test/helper.js index 041a53d2..6807c6fc 100644 --- a/test/helper.js +++ b/test/helper.js @@ -56,7 +56,7 @@ async function run(pattern, replacement, config) { try { const options = { - engine: 'default', + engine: 'rr', encoding: 'utf8', pipedData: null, includeGlob: [], diff --git a/yarn.lock b/yarn.lock index dce61abe..20f409b7 100644 --- a/yarn.lock +++ b/yarn.lock @@ -461,6 +461,18 @@ queue-microtask@^1.2.2: resolved "https://registry.yarnpkg.com/queue-microtask/-/queue-microtask-1.2.3.tgz#4929228bbc724dfac43e0efb058caf7b6cfb6243" integrity sha512-NuaNSa6flKT5JaSYQzJok04JzTL1CA6aGhv5rfLW3PgqA+M2ChpZQnAC8h8i4ZFkBS8X5RqkDBHA7r4hej3K9A== +regex-utilities@2.3.0, regex-utilities@^2.3.0: + version "2.3.0" + resolved "https://registry.yarnpkg.com/regex-utilities/-/regex-utilities-2.3.0.tgz#87163512a15dce2908cf079c8960d5158ff43280" + integrity sha512-8VhliFJAWRaUiVvREIiW2NXXTmHs4vMNnSzuJVhscgmGav3g9VDxLrQndI3dZZVVdp0ZO/5v0xmX516/7M9cng== + +regex@6.1.0: + version "6.1.0" + resolved "https://registry.yarnpkg.com/regex/-/regex-6.1.0.tgz#d7ce98f8ee32da7497c13f6601fca2bc4a6a7803" + integrity sha512-6VwtthbV4o/7+OaAF9I5L5V3llLEsoPyq9P1JVXkedTP33c7MfCG0/5NOPcSJn0TzXcG9YUrR0gQSWioew3LDg== + dependencies: + regex-utilities "^2.3.0" + require-directory@^2.1.1: version "2.1.1" resolved "https://registry.yarnpkg.com/require-directory/-/require-directory-2.1.1.tgz#8c64ad5fd30dab1c976e2344ffe7f792a6a6df42"