6.0.0 release checklist #154

What has to happen before libhxcpp 6.0.0 goes to haxelib. Pre-releases follow RELEASING.md: 6.0.0-rc.1 is out, and anything fixed before 6.0.0 ships as -rc.2 and on.

The bar for 6.0.0: every known problem that libhxcpp can fix without a change in Haxe is fixed, shown not to reproduce, or moved out with a reason (#192 for the critical ones). That covers wrong results, crashes and leaks, performance and memory, each performance fix with a before and after measurement, and cppia being usable and debuggable on every CPU, from VS Code, with clear errors.

Open for 6.0.0

Wrong results, lost data or objects freed in use, found since rc.1:

• SQLite: SQLite TEXT reads truncate values at embedded NUL characters #272 (TEXT cut at an embedded NUL), Failed SQLite opens leave native connection handles allocated #273 (failed opens leak the connection), SQLite result column names accumulate in the permanent string pool #274 (column names pile up in the permanent string pool), GC of a SQLite connection silently truncates a still-live result set #290 (collecting the connection truncates a live result set)
• Sockets and TLS: Prepared socket polls do not retain registered wrappers, allowing GC to close active connections #286 (a prepared Poll does not keep its sockets alive), Peer certificate objects lose their native data when the SSL socket closes #291 (a peer certificate loses its data when the socket closes), Windows Poll misses failed connections beyond 64 read sockets and in write-only polls #304 (Windows Poll misses failed connections past 64 sockets and in write-only polls), Windows IPv6 Socket.accept fails with an IPv4-sized peer address buffer #309 (Windows IPv6 accept fails with an IPv4-sized address buffer), Windows IPv6 address formatting returns null during concurrent first use #310 (Windows IPv6 address formatting returns null on concurrent first use), Windows empty socket polls ignore timeouts and spin at full CPU #311 (an empty Windows Poll ignores its timeout and spins), Retry resolver buffer exhaustion instead of reporting valid hosts as unknown #312 (a full resolver buffer reports valid hosts as unknown)
• Int64: ObjectMap Float insertion rounds Int64 values already stored under other keys #282 (ObjectMap Float insertion rounds stored Int64 values), Writable cached Int64 boxes let reflection change independent and future values #295 (cached Int64 boxes are writable through reflection), Boxed Int64 high-word updates are overwritten by sign extension from the low word #296 (setting hi on a boxed Int64 sign-extends the low word), Handle minimum Int64 division and remainder by negative one without terminating the process #326 (the minimum Int64 divided by -1, or its remainder, ends the process)
• Arrays and strings: Array formatting shares its recursion depth across threads #277 (array formatting shares its recursion depth across threads), String.substr overflows before clamping a large positive length #279 (String.substr overflows before clamping a large length), Failed dynamic-array searches box the whole array and can change element types #281 (a failed dynamic-array search boxes the array and can change its element type), Dynamic array concat changes source storage and can discard later string pushes #293 (Array.concat changes the source's storage and can lose later pushes)
• Other: Zero-length file reads grow empty buffers and change read-ahead EOF state #275 (zero-length file reads), cpp.Random.int cannot reach the full range of large positive bounds #276 (cpp.Random.int with large bounds), CFFI field IDs misinterpret UTF-8 names, causing missed reads and duplicate fields #288 (CFFI field IDs misread UTF-8 names), Loading a second CPPIA module hides resources of a still-loaded module #292 (a second cppia module hides the resources of one still loaded), Preserve file timestamps beyond 2038 in FileSystem.stat #321 (FileSystem.stat loses timestamps after 2038), Preserve Unicode arguments and results in ordinary CFFI wrappers for Prime exports #323 (CFFI Prime wrappers lose Unicode arguments and results)

Performance and memory found since rc.1, each with a before and after measurement:

• Boxing: Int64-key hashes abandon specialized value storage on the second boxed Int64 #278, Avoid temporary numeric boxes when pushing into specialized dynamic arrays #294, Avoid temporary numeric boxes when popping typed arrays #306, Avoid boxing typed start offsets in array and string searches #307, Store SQLite row primitives inline instead of retaining a box for each cell #308
• Searching and parsing: String.split and lastIndexOf still repeat long comparisons at every candidate #280, Numeric parsing transcodes unused UTF-16 suffixes before reading a short prefix #284, Bound the linear pointer scan for literal field names on wide anonymous objects #285
• Hashing and maps: Release excess weak-map bucket capacity after GC removes the keys #204 (a weak map keeps its buckets after most keys die), Int64 hashing collapses packed coordinate pairs and equal halves into long chains #289
• Buffers and the large-object allocator: Chunked array growth repeatedly copies whole buffers in socket reads and CFFI appends #287, NativeArray.reserve ignores empty dynamic arrays and causes avoidable buffer growth #297, Large-buffer release scans unrelated live allocations while holding the shared allocator lock #298, Index recycled large buffers by size to avoid repeated scans under the allocator lock #300, Include recycled large buffers in reserved-memory reporting #313, Avoid clearing large atomic copy destinations immediately before overwriting them #315
• Generational collector: Track dirty pointer-array ranges to reduce minor GC work after sparse writes #301, Avoid scanning every old finalizer during minor collections #302, Avoid scanning entire old maps after sparse writes during minor GC #305, Avoid rescanning stable old weak references during every minor GC #314, Skip stable old large allocations during minor GC sweeps #318
• Native memory the collector cannot see: Account for PCRE2 native allocations in GC memory pressure #303 (PCRE2), Account for SQLite native database and statement memory in GC pressure #319 (SQLite), Account for native TLS context memory in GC pressure #325 (TLS)
• Threads: Reclaim oversized backing storage from long-lived empty Deques #316, Avoid quadratic front insertion in sys.thread.Deque #317 (sys.thread.Deque)
• CFFI: Reuse UTF-8 conversion work across repeated CFFI string access #322, Avoid temporary argument arrays for small fixed-arity CFFI vector calls #324
• Sockets: Use a persistent event backend for Poll with many mostly idle sockets #320 (a persistent event backend for Poll with many mostly idle sockets)

cppia:

• A host fingerprint scripts can require (cppia: a host fingerprint that scripts can require #263)
• What keeps a released module loaded (cppia: report what keeps a released module loaded #264)
• Moving live instances onto a replaced class (cppia: move live instances of a replaced class onto its replacement #265), once its design is agreed

Before the release:

• Haxe 5 on every change (Test Haxe 5 on every change #153): hxb and compile-server builds, which CI does not run yet
• Architecture pass over the GC, threads, cppia and the build tool, filing what it finds
• Lime: lime setup installs and runs libhxcpp setup
• Every item in 6.0.0: critical runtime and cppia issues #192 fixed, shown not to reproduce, or moved out with a reason
• 6.0.0-rc.2 from a green main, then 6.0.0 from a green main

Done since rc.1

• GC: GcCensus does not find a local array on macOS #269 (objects freed when only a pointer into them was on the stack, every clang build, 740e0b9), HXCPP_SANITIZE=address builds free objects that only a local holds #270 (ASan builds, 9bee2f4), Intermittent segfault in test/haxe on macOS arm64 (generational GC) #254 (GetNextFree on ARM64, very likely EXC_BAD_ACCESS Unclosed string > KERN_INVALID_ADDRESS at 0x627. HaxeFoundation/hxcpp#1177, d734565), Reduce repeated ephemeron sweeps for weak-map dependency chains #203 (weak-map chains, c3ab80a), Release-mode heap census by class #82 (release-build heap census). Generational GC: a nursery object occasionally escapes a collection under stress on Windows #168 closed as not reproducible
• cppia JIT: cppia JIT: a Bool stored in a dynamic field is boxed as 1 #259 and cppia jit: Array contains, remove and Array<Bool> element results are boxed as Int #260 (Bools boxed as Int), cppia JIT: executable memory on Apple platforms, and saying when the JIT is not used #266 (executable memory on Apple platforms, 9d64e1d), cppia JIT: bring up the arm64 code generator #271 (the JIT runs on arm64, fe67557), cppia JIT: with HXCPP_GC_GENERATIONAL, objects only the stack holds are freed while in use #299 (objects only the stack held freed under HXCPP_GC_GENERATIONAL, Calling Gc.compact() in cppia collects the current CppiaClosure/ScriptCallable? HaxeFoundation/hxcpp#1196, 0fe6cbe)
• cppia: cppia: check a script against its host before linking and report every missing class and member #262 (one error naming everything a script needs from its host), cppia frames store objects, Strings and doubles at misaligned offsets #157 (frame slots read and written without misaligned access; the sanitizer job suppresses no alignment reports)
• Debugging cppia from VS Code, interpreted and with the JIT, on Windows, Linux and macOS (cppia: a debugger that attaches before a script loads cannot set breakpoints in it #283, Breakpoints don't work on cppia compiled with HXCPP_DEBUGGER HaxeFoundation/hxcpp#498): libhxcpp's half is on main, and the server's half is Set breakpoints in cppia scripts loaded after the server attaches vshaxe/hxcpp-debugger#48. docs/CppiaDebugging.md describes the setup
• Processes: Windows: Process.close waits for a descendant that holds the output pipe while another thread reads it #261 (Windows Process.close waiting on a descendant's pipe, d8e534c), cpp.Function fields don't compile with HXCPP_GC_GENERATIONAL #167 (cpp.Function fields under HXCPP_GC_GENERATIONAL), Generated files whose names differ only in case overwrite each other #70 (the build tool stops when generated file names differ only in case)
• Infrastructure: Haxe 5.0.0-preview.1 builds against the API level 430 runtime and runs the suites on every change (b26ce09), output compared with the other Haxe targets (Compare test program output with the other Haxe targets #150), nightly downstream builds (Nightly downstream builds of Lime, OpenFL, Flixel, HaxeUI and CrossByte #149), the build tool stops a Haxe older than 4.2

Done before rc.1

• Sanitizer CI (UBSan, ASan, TSan, CodeQL) and everything it reported (Run the test suites under sanitizers and CodeQL #148, Constant allocations are only 4-byte aligned #155, String headers are read from unaligned string data #156, SSL casts the socket handle to its own copy of SocketWrapper #158, cppia: functions with an untyped return read an uninitialised return type #159)
• GC and thread races and the thread API: Generational GC: a thread attaching during a mode switch gets no remembered set #139, ARM64: the collector can miss a thread leaving a GC-free zone #142, The moving GC ignores the pin flag on alloc finalizers #143, ThreadLocal reuses a slot without clearing its old values #68, Sys.exit destroys GC state while other threads still allocate #67, Process.exitCode returns 0 when called again on POSIX #114, Thread wait timeouts break on negative, NaN, huge and sub-millisecond values #122, Natively attached threads all get thread id 0 #123, Windows semaphores cap at 134 million permits #125
• Security and dependencies: the PCRE2 backports, SQLite 3.54.0 with the WAL-reset fix, Actions off Node 20 (Dependency upgrades by priority #147), the mbedTLS provider for static builds (Let a library provide mbedTLS for static builds #161)
• Wrong results: Comparing a String with a Dynamic depends on the static types #97 to haxe.Utf8 truncates characters above U+FFFF #107, Bytes.ofString and urlEncode throw on a lone surrogate #110, Date.getTimezoneOffset ignores daylight saving time on Windows #112, Negative fractional times round toward zero #116, Sys.environment decodes the Windows environment as UTF-8 #117, Haxe 5 thread objects: pseudo-handles for names, and Scratch copy assignment #124, cppia JIT: String += drops null and turns an empty string into null #129, cppia: isOfType, cast and catch ignore interfaces inherited through other interfaces #131 to cppia: safe casts to Int, Float, Bool and String never throw #135, the narrow-trigger group (Zero-length process pipe operations grow empty byte buffers #209, POSIX child stdin is closed during exec when the parent has no fd 0 #210, Socket select and poll restart finite timeouts after every signal interruption #217, Whole-file reads return empty data for readable zero-size procfs files #219, Positive sub-microsecond socket timeouts disable the timeout on POSIX #220, Reflect.hasField misses fields in fixed anonymous objects #222, Anonymous-object formatting shares its recursion limit across threads #223, Reflected Array.filter rereads source values after the predicate changes them #224, Native heap freezing duplicates shared objects instead of preserving graph identity #226, Native cleanup callbacks erase outer frames from a propagating exception stack #229, Print floats with the shortest digits that round-trip #93, HXCPP_GC_VERIFY reports unmarked row HaxeFoundation/hxcpp#1415) and the high-severity triage of @:nativeGen interfaces: second interface gets the object's address, and the collector misses it #193 to Native cleanup callbacks erase outer frames from a propagating exception stack #229 (6.0.0: critical runtime and cppia issues #192)
• Performance against stock hxcpp 4.3.53 and its regression suite (Add a native benchmark suite, recorded baselines and performance regression checks #151, GC heap does not grow; HXCPP_MINIMUM_WORKING_MEMORY is ignored (allocation ~3x slower than stock) #230 to Peak memory is 4-22% higher than stock hxcpp 4.3.53 on most benchmarks #235 and the items in 6.0.0: critical runtime and cppia issues #192)
• Haxe 5 API level 500 fixes tested on Haxe 5: Scratch can reuse a live buffer when allocations are released out of order #184 to Endian-aware native pointer readers instantiate the wrong pointer type #188, Weak bound-method liveness does not follow callable adapter chains #202
• Docs: README, Defines.md, Changes.md, SECURITY.md and supported versions (Release process: versions, a stable branch and a security policy #152), the performance table
• Release mechanics: haxelib path output for hxp (extraParams -D lines make hxp misread libhxcpp's path #165), Lime 8.4.0's own mbedTLS 3.6 copy, fixes for a released line on a release/<major>.<minor> branch (Release process: versions, a stable branch and a security policy #152), 6.0.0-rc.1 on haxelib and GitHub

After 6.0.0

• Issues labelled needs haxe (Array.iterator() is not a haxe.iterators.ArrayIterator #5, cppia: extern abstract over a native enum fails to link #8, Generated files whose names differ only in case overwrite each other #70, Provide native map iteration without copying the whole map into arrays #199, IPv6 socket host and peer queries return an IPv4 address instead of the endpoint #218, Null<UInt> compared with null is wrong on cpp #251), which wait on HaxeFoundation/haxe
• Haxe nightly (API level 500): GC stress segfaults and suite failures #164 and Avoid Dynamic boxing in typed Haxe 5 numeric array sorting #195, which wait for Haxe 5 itself
• Port the bundled mbedTLS to the 4.1 LTS before 3.6 support ends #255 (the mbedTLS 4.1 port; Lime 8.4.0 provides the 3.6 LTS, supported to March 2027), Investigate where else allocation cost and memory can come down #256, Fuzz the runtime's parsers #257, Track benchmark results in CI and report regressions #258
• Opt-in concurrent marking for the GC (6.x) #327, opt-in concurrent marking, a proposal for 6.x
• CrossByte's testASmallInt64IsAnInt64WhenWrapped expects hxcpp's old Int64 boxing and needs changing in CrossByte