Skip to content

Use canonical Kafka schema settings with Jackson 3.1.8 - #738

Merged
stevehu merged 3 commits into
masterfrom
fix/264-canonical-schema-registration
Nov 3, 2026
Merged

stevehu merged 3 commits into
masterfrom
fix/264-canonical-schema-registration

Conversation

@stevehu

@stevehu stevehu commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Generated Kafka configurations still emitted the unsupported schema.registry.auto.register.schemas key and matching external override names. Use auto.register.schemas in the producer, consumer and Streams templates, expose corresponding values entries only for the configurations each generator emits, and document the required migration. Related: networknt/light-kafka#264. light-example-4j is out of scope.

Keep Jackson pinned at 3.1.8. Adapt configuration-only YAML scalar normalization to APIs available in 3.1.8, preserving Core/legacy booleans, empty/valueless nodes, non-finite numbers, numeric parsing, quoting, anchors, encoding and stream limits. Ordinary JSON scalars bypass the event pass; unchanged text avoids an output copy.

Merge current master and preserve the removal of Jackson-specific CI. The PR uses master's plain Maven CI and makes no workflow or qualification source-pin changes relative to master.

Validation:

  • Full local mvn -o -q clean install: 86 passed, 6 skipped, zero failures/errors, without a Jackson version override.
  • Generation regression covers all eight Kafka flag combinations for both server and service generators, exact correspondence between emitted files and values entries, and actual canonical/old override injection.
  • YAML regressions cover valueless flow/explicit keys, empty leading documents, non-finite values, Unicode/encoding, anchors, quoting and numeric limits. These edge-case regressions also passed against a separate master + Jackson 3.2.4 baseline.
  • git diff --check passes. Fresh remote Maven CI is separate from the local cached-dependency evidence above.

Old external override names remain deliberately unused under the existing compatibility policy. The README and generated values warn that an old false override alone leaves the canonical true default; rename overrides before regeneration and register schemas before selecting canonical false. Light's intentional serializer legacy fallback is preserved. No deployment is included. Companion 2.x PR: #739.

@stevehu

stevehu commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Review findings

I ran GeneratorConfigYaml from this branch against master with Jackson 3.2.4 as a baseline, and generated Hybrid server output with single Kafka flags.

1. Empty-scalar rewrite corrupts valueless keys and empty documents (GeneratorConfigYaml.java:95)
The " null" rewrite splices text at the empty scalar's start mark. For valueless keys that mark is the next token, not the position right after :.

  • {a, b: 1} gives {"a null":"","b":1}; baseline {"a":null,"b":1}.
  • ? a / ? b gives {"a null? b null":""}; baseline {"a":null,"b":null}.
  • ---\n---\na: 1 gives the string "null--- a"; baseline null.

Suggested fix, verified in a scratch copy: enable YAMLReadFeature.EMPTY_STRING_AS_NULL (present in 3.1.8) on the factory builder and drop the value.isEmpty() ? " null" branch. All three cases then match the baseline, and x: and - still give null.

2. values.yml entries don't match the generated kafka ymls (values.rocker.raw:69)
HybridServerGenerator emits kafka-streams.yml when kafkaConsumer is true and never emits kafka-consumer.yml. With only kafkaConsumer=true, the server gets kafka-streams.yml reading ${kafka-streams.auto.register.schemas:true}, but values.yml only gets kafka-consumer.auto.register.schemas. Following the README and setting that to false does nothing. With only kafkaStreams=true, values.yml has a kafka-streams entry but no kafka-streams.yml exists. Please gate each values entry on the same condition as its yml in each generator.

3. Non-finite floats now throw (GeneratorConfigYaml.java:96)
.inf, -.inf, +.inf, .nan and .INF resolve to FLOAT in both resolvers, so they are never rewritten. Jackson 3.1.8 then throws StreamReadException: Malformed numeric value. The baseline returns strings such as "Infinity".

4. The old override key is silently ignored (kafkaProducerYml.rocker.raw:30, plus consumer and Streams)
An existing kafka-producer.schema.registry.auto.register.schemas: false (honored by Light's serializer fallback, per the README) is no longer referenced. The producer falls back to true and starts auto-registering with no warning. Consider keeping the old key as a fallback or failing loudly. This is a plausible risk, not something I reproduced.

5. Efficiency (GeneratorConfigYaml.java:47)
The regex fast path is gone, so every config gets a full snakeyaml event parse plus Jackson's parse. The BOOL and NULL branches also rewrite plain true/false/null to themselves. Guard with !core.equals(JSON_SCALARS.resolve(value, true)) and consider a cheap pre-check.

6. Test gaps

  • GeneratorConfigScalarCompatibilityTest has no cases for valueless keys, empty leading documents or non-finite floats.
  • KafkaSchemaRegistrationGenerationTest enables all three flags and uses a convoluted skip condition. It asserts the dead server kafka-consumer entry as correct and duplicates the final consumer render check.

7. Cleanup

  • normalizeBooleans and the class Javadoc still describe legacy booleans, but the method now handles Core null/int/float too.
  • The README's "Jackson 3.1.8 source validation" section holds merge-gating prose that goes stale after merge.

@stevehu
stevehu merged commit 6472830 into master Nov 3, 2026
2 checks passed
@stevehu
stevehu deleted the fix/264-canonical-schema-registration branch November 3, 2026 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant