Make VM data and generated helpers harder to identify

doc/advanced/register-vm.md

@@ -16,28 +16,38 @@ compiler, or `loadstring` to execute the original source.
 - `vm/optimize.lua`: recursive, conservative basic-block optimizations and
   cleanup batching before emission.
 - `vm/opcodes.lua`: semantic instruction IDs.
-- `vm/layout.lua`: a per-emission opcode and operand layout.
+- `vm/layout.lua`: per-emission opcode, operand, and numeric metadata keys.
 - `vm/runtime.lua` and `vm/runtime_dispatch.lua`: runtime templates and the
   generated dispatcher, configured for the selected opcode IDs and operand
   positions.
 - `vm/runtime_calls.lua`: fixed-arity and packed call, return, and register
   cleanup handlers used by the generated runtime.
-- `vm/emit.lua`: instruction encoding, native prototype table literals, and the
-  runtime function IIFE.
+- `vm/data.lua`: scattered table graph encoding and startup reconstruction.
+- `vm/emit.lua`: shard literals, the reconstruction helper, and the runtime
+  function IIFE.
 - `vm/strings.lua`: optional hoisting of generated interpreter string literals
   into an outer initializer.
 - `vm/flatten.lua`: the Cff integration for generated VM interpreters.
 
 A prototype contains instructions, constants, child prototypes, parameter
-metadata, and capture descriptors. The emitter passes the native prototype
-tables as an argument to a runtime function IIFE. The generated interpreter
-then uses that table data; the prototypes are not encoded as strings.
+metadata, and capture descriptors. Prototype fields use unique randomized
+numeric keys instead of fixed string keys. The emitter scatters assignments
+for the complete table graph across three to five native shard tables,
+including individual instruction operands and constant entries. Table IDs and
+record order are shuffled. A startup helper allocates all destination tables,
+resolves references, and passes the reconstructed root to the runtime function
+IIFE. It releases the consumed shards. Reconstruction happens once per chunk
+execution; the interpreter uses ordinary direct table lookups afterward.
+Scattering increases output size and startup work without adding shard lookups
+to the VM loop. The prototypes are not encoded as strings.
 
 Each invocation owns its register table; vararg functions also carry a counted
 vararg tuple. Closures retain only
 their referenced upvalue cells; leaving a scope detaches its cells without
-destroying values shared by live closures. Value lists carry an explicit `n`
-so trailing nils survive calls, varargs, table constructors, and returns.
+destroying values shared by live closures. Value lists carry an explicit count
+under a per-emission negative numeric key, keeping it separate from positive
+array indices so trailing nils survive calls, varargs, table constructors, and
+returns.
 The compiler keeps ordinary scalar values in individual registers. Small calls
 and returns pass those registers directly to native Lua instead of first making
 a value-list table. Temporary registers are cleared after consumption, and
@@ -66,22 +76,24 @@ initialization path, which also builds the vararg tuple when needed.
 `Cff` normally flattens control flow throughout its input AST. When it receives
 the AST emitted by `Vmify`, a private `vmRuntime` annotation selects a special
 path: by default, `vm/flatten.lua` flattens the interpreter function only. The
-large native prototype table initialization remains outside the flattened
-function, keeping it out of the Cff control structure and jump limits used by
-Lua and LuaJIT.
+large native shard initialization and startup reconstruction remain outside
+the flattened function, keeping them out of the Cff control structure and jump
+limits used by Lua and LuaJIT.
 
 Set `Cff`'s `ExcludeVm` setting to `true` to leave both the generated
-interpreter and the native prototype table literal out of flattening. Cff then
-flattens the surrounding code, including helpers introduced by intervening
-steps such as string decoders or tamper checks. The interpreter remains native
-Lua, so its lexical captures continue to share variables with surrounding
+interpreter and the native shard literal out of flattening. Cff then flattens
+the startup reconstruction helper and its call along with surrounding code,
+including helpers introduced by intervening steps such as string decoders or
+tamper checks. The interpreter remains native Lua, so its lexical captures
+continue to share variables with surrounding
 closures according to ordinary Lua upvalue semantics.
 
 `cff/native.lua` emits the excluded expressions directly and bridges their
 external locals through Cff's shared upvalue cells. Garbage-collected capture
 proxies keep those cells alive for native closures; captures are not value
-snapshots. The `vmData` annotation identifies the entire prototype initializer,
-including its child prototypes.
+snapshots. The `vmData` annotation identifies only the shard literal containing
+the complete prototype graph. It does not exclude the reconstruction helper
+or its invocation from Cff.
 
 With the default `ExcludeVm = false` behavior, the special path passes
 read-only external lexical helpers through function arguments. This supports
@@ -94,8 +106,9 @@ arity and cleanup instructions use the same ID scheme. At emission,
 `layout.lua` chooses a permutation of the opcode IDs and a permutation of
 operand slots 2 through 5; slot 1 always contains the opcode. The emitter
 encodes each instruction with that layout without changing the compiler
-prototypes. One layout is shared by the root prototype and all child
-prototypes in the emitted chunk.
+prototypes. The layout also selects numeric prototype field keys and the
+packed-value count key. One layout is shared by the root prototype and all
+child prototypes in the emitted chunk.
 
 Before emission, `optimize.lua` visits the root prototype and its children and
 applies conservative basic-block-local rewrites. At emission,
@@ -111,15 +124,17 @@ the generated interpreter's literal strings into an outer initializer, where
 normal string encryption and later `ConstantArray` processing apply. The
 initializer decodes them once at startup and binds them to native lexical
 locals used by the interpreter, avoiding decoder or wrapper calls on hot
-paths. Strings in `vmData` prototype tables remain encrypted by the normal
+paths. Strings in the `vmData` shards remain encrypted by the normal
 `EncryptStrings` behavior. `vm/strings.lua` implements this transformation.
 
 Medium then applies `AntiTamper`, `ConstantArray`, and `WrapInFunction`,
-followed by `Cff` with `ExcludeVm = true`. The interpreter and prototype table
-initializers stay native while Cff flattens the surrounding code and generated
-helpers. `NumbersToExpressions` runs after Cff: it adds no helper functions,
-and this ordering prevents Cff from virtualizing arithmetic constant
-expressions.
+followed by `Cff` with `ExcludeVm = true`. The interpreter and shard literal
+stay native while Cff protects startup reconstruction, surrounding code, and
+generated helpers. After generating its helpers, Cff puts repeated string
+literals in a shared outer table, including any new diagnostics and counted-value
+keys. Each value is embedded only once in Medium output. `NumbersToExpressions`
+runs after Cff: it adds no helper functions, and this ordering prevents Cff
+from virtualizing arithmetic constant expressions.
 
 `Strong` applies one `Vmify` pass followed by `EncryptStrings`, `AntiTamper`,
 and one default `Cff` pass over the generated VM interpreter. It retains the

doc/reference/steps/cff.md

@@ -8,6 +8,15 @@ When `Cff` receives `Vmify` output, it recognizes the private VM runtime annotat
 
 Set `ExcludeVm` to `true` to leave the generated interpreter and native prototype table literal out of Cff flattening. Cff then flattens the surrounding code, including helpers such as decoders and tamper checks, and the outer string initializer emitted by `EncryptStrings` when `HoistVmStrings` is enabled. The interpreter keeps its native Lua lexical captures, preserving shared variable behavior between the interpreter and surrounding closures.
 
+After generating its runtime and helpers, Cff shares repeated string literals
+through one outer table with shuffled slots. Each repeated value is embedded
+once and reused through indexed references, including diagnostics and counted
+argument/result keys introduced by Cff itself. Unique literals stay in place.
+This also applies to literals inside preserved native VM expressions; their
+control flow stays native. Sharing does not encrypt the remaining literal.
+Later steps may introduce new strings. Medium only runs `NumbersToExpressions`
+after Cff, so its final output retains one literal per distinct string value.
+
 ## Settings
 
 | Setting | Type | Default | Description |

doc/reference/steps/vmify.md

@@ -10,6 +10,16 @@ slot 1 always stores the opcode. The same layout applies to the root function
 and its child prototypes. For the same input, configuration, and Lua runtime,
 a fixed pipeline seed reproduces the layout.
 
+Prototype fields and packed-value counts use per-emission numeric keys;
+Vmify introduces no fixed string keys for these internal tables. The complete
+prototype graph, including instructions and constants, is scattered across
+three to five tables with shuffled records and table IDs. A generated helper
+reconstructs ordinary tables once at chunk startup and releases the shards,
+leaving direct lookups during VM execution. This adds output size and startup
+work. In Medium, Cff protects the reconstruction helper and its invocation;
+only the interpreter and large shard literal remain excluded from flattening.
+Source-program strings and keys retain their normal semantics.
+
 ## Settings
 
 None.

src/prometheus/cff/strings.lua

@@ -0,0 +1,46 @@
+-- Share repeated literals after Cff has introduced its own helper strings.
+local Ast = require('prometheus.ast')
+local visitast = require('prometheus.visitast')
+local shuffle = require('prometheus.util').shuffle
+local Kind = Ast.AstKind
+
+return function(ast)
+    local counts, strings = {}, {}
+    visitast(ast, function(node)
+        if node.kind == Kind.StringExpression then
+            local value = node.value
+            counts[value] = (counts[value] or 0) + 1
+            if counts[value] == 2 then strings[#strings + 1] = value end
+        end
+    end)
+    if #strings == 0 then return ast end
+
+    shuffle(strings)
+    local scope = ast.body.scope
+    local pool = scope:addVariable()
+    local indices, entries = {}, {}
+    for i, value in ipairs(strings) do
+        indices[value] = i
+        entries[i] = Ast.TableEntry(Ast.StringExpression(value))
+    end
+    -- Track blocks explicitly: strings after a nested function literal still
+    -- belong to the enclosing block, regardless of the visitor's last scope.
+    local scopes = {}
+    visitast(ast, function(node)
+        if node.kind == Kind.Block then scopes[#scopes + 1] = node.scope end
+    end, function(node)
+        if node.kind == Kind.Block then
+            scopes[#scopes] = nil
+        elseif node.kind == Kind.StringExpression then
+            local index = indices[node.value]
+            if index then
+                scopes[#scopes]:addReferenceToHigherScope(scope, pool)
+                return Ast.IndexExpression(Ast.VariableExpression(scope, pool), Ast.NumberExpression(index))
+            end
+        end
+    end)
+    -- Insert only after replacement so each pooled value remains embedded once.
+    table.insert(ast.body.statements, 1,
+        Ast.LocalVariableDeclaration(scope, {pool}, {Ast.TableConstructorExpression(entries)}))
+    return ast
+end

src/prometheus/steps/Cff.lua

@@ -7,6 +7,7 @@
 local Step = require("prometheus.step");
 local Compiler = require("prometheus.cff.compiler");
 local flattenVm = require("prometheus.vm.flatten");
+local shareStrings = require("prometheus.cff.strings");
 
 local Cff = Step:extend();
 Cff.Description = "Flattens control flow into Lua statements executed by a randomized block dispatcher.";
@@ -26,15 +27,15 @@ function Cff:apply(ast)
         local flattened = compiler:compile(ast);
         flattened.vmRuntime = compiler.nativeResults[ast.vmRuntime];
         flattened.vmData = compiler.nativeResults[ast.vmData];
-        return flattened;
+        return shareStrings(flattened);
     end
     if ast.vmRuntime then
         local flattened = flattenVm(ast);
-        if flattened then return flattened; end
+        if flattened then return shareStrings(flattened); end
     end
 	local compiler = Compiler:new();
 
-    return compiler:compile(ast);
+    return shareStrings(compiler:compile(ast));
 end
 
 return Cff;

src/prometheus/vm/data.lua

@@ -0,0 +1,82 @@
+-- Scatter the emitted table graph; rebuild ordinary tables once before VM startup.
+local Ast = require('prometheus.ast')
+local Unparser = require('prometheus.unparser')
+local Enums = require('prometheus.enums')
+local shuffle = require('prometheus.util').shuffle
+local Data = {}
+
+Data.rebuild = [=[
+local function rebuild(shards, count, root)
+    local tables = {}
+    for i = 1, count do tables[i] = {} end
+    for i = 1, #shards do
+        local shard = shards[i]
+        for j = 1, #shard do
+            local record = shard[j]
+            local value = record[3]
+            if record[4] then value = tables[value] end
+            tables[record[1]][record[2]] = value
+        end
+        shards[i] = nil
+    end
+    return tables[root]
+end
+]=]
+
+function Data.encode(proto, layout)
+    local function prototype(value)
+        local result, code, children = {}, {}, {}
+        for i, instruction in ipairs(value.code) do
+            code[i] = layout:encode(instruction)
+        end
+        for i, child in ipairs(value.children) do children[i] = prototype(child) end
+        result[layout.fields.code] = code
+        result[layout.fields.children] = children
+        for _, name in ipairs({'constants', 'captures', 'params', 'vararg', 'maxRegister'}) do
+            result[layout.fields[name]] = value[name]
+        end
+        return result
+    end
+    local root = {prototype(proto)}
+    local tables, indices, keys = {}, {}, {}
+    local function collect(value)
+        if indices[value] then return end
+        tables[#tables + 1] = value
+        indices[value] = #tables
+        local sorted = {}
+        for key in pairs(value) do sorted[#sorted + 1] = key end
+        table.sort(sorted)
+        keys[value] = sorted
+        for _, key in ipairs(sorted) do
+            if type(value[key]) == 'table' then collect(value[key]) end
+        end
+    end
+    collect(root)
+    local ids = {}
+    for i = 1, #tables do ids[i] = i end
+    shuffle(ids)
+    local unparser = Unparser:new({LuaVersion = Enums.LuaVersion.Lua51})
+    local records = {}
+    for i, value in ipairs(tables) do
+        for _, key in ipairs(keys[value]) do
+            local entry, reference = value[key], ''
+            if type(entry) == 'table' then
+                entry, reference = ids[indices[entry]], ',1'
+            end
+            local literal = unparser:unparseExpression(Ast.ConstantNode(entry), 0)
+            records[#records + 1] = '{' .. ids[i] .. ',' .. key .. ',' .. literal .. reference .. '}'
+        end
+    end
+    shuffle(records)
+    local shards = {}
+    local count = math.random(3, 5)
+    for i = 1, count do shards[i] = {} end
+    for i, record in ipairs(records) do
+        local shard = shards[(i - 1) % count + 1]
+        shard[#shard + 1] = record
+    end
+    for i, shard in ipairs(shards) do shards[i] = '{' .. table.concat(shard, ',') .. '}' end
+    return '{' .. table.concat(shards, ',') .. '}', #tables, ids[indices[root]]
+end
+
+return Data

src/prometheus/vm/emit.lua

@@ -1,50 +1,26 @@
--- Emit native table data and a generated interpreter, then parse the complete AST.
-local Ast = require('prometheus.ast')
+-- Emit scattered data, startup reconstruction, and a generated interpreter.
 local Parser = require('prometheus.parser')
-local Unparser = require('prometheus.unparser')
 local Enums = require('prometheus.enums')
 local Runtime = require('prometheus.vm.runtime')
 local Layout = require('prometheus.vm.layout')
+local Data = require('prometheus.vm.data')
 local Emit = {}
 
 function Emit.source(proto, layout)
     layout = layout or Layout.new()
-    local unparser = Unparser:new({LuaVersion = Enums.LuaVersion.Lua51})
-    local function literal(value)
-        if type(value) ~= 'table' then
-            return unparser:unparseExpression(Ast.ConstantNode(value), 0)
-        end
-        local parts = {}
-        for _, entry in ipairs(value) do parts[#parts + 1] = literal(entry) end
-        return '{' .. table.concat(parts, ',') .. '}'
-    end
-    local function prototype(value)
-        local instructions = {}
-        for _, instruction in ipairs(value.code) do
-            local encoded, length = layout:encode(instruction)
-            local operands = {}
-            for slot = 1, length do operands[slot] = literal(encoded[slot]) end
-            instructions[#instructions + 1] = '{' .. table.concat(operands, ',') .. '}'
-        end
-        local parts = {'code={' .. table.concat(instructions, ',') .. '}'}
-        for _, key in ipairs({'constants', 'captures', 'params', 'vararg', 'maxRegister'}) do
-            parts[#parts + 1] = key .. '=' .. literal(value[key])
-        end
-        local children = {}
-        for _, child in ipairs(value.children) do children[#children + 1] = prototype(child) end
-        parts[#parts + 1] = 'children={' .. table.concat(children, ',') .. '}'
-        return '{' .. table.concat(parts, ',') .. '}'
-    end
-    local data = '{' .. prototype(proto) .. '}'
-    local source, count = Runtime.build(layout):gsub('__PROTOTYPES__', function() return data end)
+    local data, tables, root = Data.encode(proto, layout)
+    local initializer = 'rebuild(' .. data .. ',' .. tables .. ',' .. root .. ')'
+    local source, count = Runtime.build(layout):gsub('__PROTOTYPES__', function() return initializer end)
     assert(count == 1, 'Runtime template must contain one prototype marker')
-    return source
+    return Data.rebuild .. source
 end
 
 function Emit.ast(proto)
     local ast = Parser:new({LuaVersion = Enums.LuaVersion.Lua51}):parse(Emit.source(proto))
-    ast.vmRuntime = ast.body.statements[1].args[1].base
-    ast.vmData = ast.body.statements[1].args[1].args[1]
+    local invocation = ast.body.statements[#ast.body.statements].args[1]
+    ast.vmRuntime = invocation.base
+    -- Only the large shard literal stays native. Cff protects the reconstruction call/helper.
+    ast.vmData = invocation.args[1].args[1]
     return ast
 end
 

src/prometheus/vm/layout.lua

@@ -1,6 +1,7 @@
--- Choose one instruction encoding for each generated VM. Compiler IR stays semantic.
+-- Choose instruction encoding and metadata keys per VM. Compiler IR stays semantic.
 local opcodes = require('prometheus.vm.opcodes')
 local shuffle = require('prometheus.util').shuffle
+local fields = {'code', 'constants', 'captures', 'params', 'vararg', 'maxRegister', 'children'}
 local Layout = {}
 Layout.__index = Layout
 
@@ -18,6 +19,16 @@ function Layout.new()
         layout.opcodes[entry.name] = ids[i]
         layout.ids[entry.id] = ids[i]
     end
+    layout.fields = {}
+    local used = {}
+    for _, name in ipairs(fields) do
+        local key
+        repeat key = math.random(1024, 16777215) until not used[key]
+        used[key] = true
+        layout.fields[name] = key
+    end
+    -- Counted tuples also contain positive array indices, so keep metadata negative.
+    layout.countKey = -math.random(1024, 16777215)
     return layout
 end