[nuclearcat]

Problem

A lab can report the same job more than once. On production, baseline-arm64-sasha-lab on jetson-tx1-devkit (node 6ae228345c0836a8c2f041a8) was submitted as done/pass at 17:15 and as done/fail (kernel panic) at 17:29 on 2026-10-28. The second PUT /nodes replaced the first result. The KCIDB bridge had already sent the PASS, so the dashboard still shows PASS while Maestro shows fail. Nothing reported the mismatch; it only turned up in the dashboard validation workflow.

The cause: the state machine allows done → done, so a done node's result can change freely, and PUT /nodes/{id} replaces the root node without any transition check.

Change

• PUT /node/{id}, PATCH /node/{id} and PUT /nodes/{id} return 409 Conflict if they would change the state or result of a node that is already done. The first final result wins. For PUT /nodes, every existing node in the hierarchy is checked before anything is written.
• Still allowed:
  • resubmitting the same result (e.g. adding data or child nodes)
  • the late result for a node closed without one (incomplete or no result, e.g. by node_timeout), which the LAVA callback relies on
• Admins can override with ?force=true. For other users, force is ignored.
• Every rejected or forced change:
  • increments node_done_update_rejected_total / node_done_update_forced_total
  • publishes an event on the new node-integrity channel, with node, runtime, platform, old and attempted state/result, and user. Events are stored in eventhistory, so they can be queried with /events?channel=node-integrity. kernelci-pipeline's dashboard validation workflow posts them to Discord (ci: Report rejected changes to done nodes kernelci-pipeline#1663).
  • A failed publish never turns the 409 into a 500.

A separate commit fixes the unit-test auth mocks. They were registered on app only, while requests go to versioned_app, so every authenticated endpoint test returned 401.

Testing

Unit tests were run in the docker/api/Dockerfile image (tests requirements, kernelci-core main) against redis 6.2 and mongo 5.0:

• 8 new tests in tests/unit_tests/test_node_done_update.py pass.
• The override fix makes 7 existing tests pass.
• Otherwise the suite fails exactly as it does on main in this environment: the same 15 failures and 8 errors, all in tests this change doesn't touch.

Deployment notes

• Pipeline services that update a done node with a different result now get a 409. src/timeout.py already catches HTTPError. Each 409 is also visible on the node-integrity channel.
• With "first result wins", a later report can be the correct one, as in the example above. Each alert needs a manual check, and an admin can correct the node with force=true.