Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 92 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,15 @@ npm start
Then open the console on its configured port (3000 by default) and
launch **M3ter Manager**. The dashboard stays accessible while MQTT reconnects.

If a relay rejects, stalls, or interrupts Manager's WebSocket, the dashboard
falls back to authenticated HTTP updates, waiting five seconds between completed
refreshes. The connection indicator identifies this fallback separately from
MQTT connectivity. WebSockets retry in the background every 30 seconds; receiving
a valid stream update restores live mode and stops polling. Closing Manager or
ending its session stops these retries. Requests time out after 15 seconds so a
stalled relay cannot leave a refresh pending indefinitely. This requires no new
configuration and does not change the Console's MQTT connection or authentication.

### Architecture and API

Console is public. Only Manager's API and live stream require a Manager session.
Expand Down Expand Up @@ -181,7 +190,7 @@ event identity, handles `up`, `ack`, `txack`, `join`, `location`, `log`, and `st
and reconnects/resubscribes without creating another client when a meter changes.
Monitoring persistence and the existing collector are isolated consumers. A unique client ID is generated for each Console process.

The current decoder accepts **only FPort 2, 74 or 106 bytes**. It wraps the
The usage-report decoder accepts **FPort 2, 74 or 106 bytes**. It wraps the
existing `decodePayload` implementation: uint32 BE nonce; uint32 BE energy divided
by 1,000,000; 64-byte Ed25519 signature; uint16 BE voltage divided by 10; optional
32-byte public key. Existing console keys and subsequently learned keys support
Expand All @@ -200,17 +209,74 @@ bounded queries. Uplinks deduplicate by ChirpStack event ID, or by a stable tupl
of device/event type/source time/frame counter/payload when an ID is absent.
Identical values with distinct events are preserved. Without an event ID or
source timestamp, events are retained separately rather than risking data loss.
Source timestamps are preserved in metadata; invalid timestamps use receipt time. The Overview's
Usage-report source timestamps are preserved in metadata; invalid timestamps use receipt time. The Overview's
Last Uplink uses the persisted console reception time of a successfully decoded,
non-duplicate uplink. MQTT heartbeats, connection events, UI refreshes and decode
non-duplicate FPort 2 uplink. FPort 4, MQTT heartbeats, connection events, UI refreshes and decode
errors cannot reset it. Source timestamps still drive the existing Analytics.

MQTT payload transport is explicit: native ChirpStack `data` remains Base64;
hex producers supply `dataHex` or `data` with `dataEncoding: "hex"`. Hex must contain
complete byte pairs, without separators or a `0x` prefix. No alphabet guessing is
used. Both collector and Manager use the same byte conversion. Live Monitoring
shows the hexadecimal bytes and preserves the original field, encoding, and text
for inspection. Downlink encoding and firmware decoders are unchanged.
for inspection. The collector still receives only usage reports; monitoring never
causes an energy acknowledgment or a blockchain operation.

#### FPort 4 electrical monitoring

`src/m3ter/electrical.ts` implements the supplied Switch Cumulus version 1 layout:
exactly 97 bytes, big-endian, with signed active/reactive power, power factor, and
temperature. Other lengths/versions are retained as diagnostics without updating
meter state. Unsupported uplink ports are ignored. FPort 4 is never a downlink port.
Live Monitoring displays per-phase voltage/current/power/power factor, neutral
currents, frequency, temperature, cumulative forward/reverse counters, and reported
relay/phase/PGA status. No FPort 2 value substitutes for missing FPort 4 data.

Native ChirpStack `data` is Base64. A valid `object` with `type: "telemetry"` is
preferred; unknown/incomplete schemas fall back to raw bytes. The current normalized
object adapter accepts `version`, `flags`, `uptime`, `voltage`/`current` (`a,b,c`),
`activePower`/`reactivePower`/`powerFactor`/`forwardEnergy`/`reverseEnergy`
(`a,b,c,total`), `neutralMeasured`, `neutralCalculated`, `frequency`, and `temperature`.
All values are already scaled in the units displayed. `type: "report"` objects
can supply `nonce`, `energy` (kWh), `voltage` (V), and `signature` (hex); raw report
bytes are still required for independent signature verification. Values are never
rescaled a second time. Original codec objects, payload text, and encoding are retained
for inspection. The referenced firmware files, `telemetry.md`, and upstream
`tools/chirpstack-codec.js` were not available in this checkout during implementation;
the adapter's compatibility with that codec's actual field names still needs verification.
Raw-byte support follows the complete layout supplied in the development request.

Monitoring uses the actual ChirpStack `time`, retaining Console receipt time
separately. A missing/invalid measurement timestamp rejects that frame as a
diagnostic rather than substituting a UI refresh clock. Metadata includes device,
application, frame counter, deduplication ID, gateways, RSSI/SNR, and decoder source.
Deduplication uses the event ID, otherwise device/frame counter and observed boot
session (allowing counters to restart); a bounded persistent recent-key window
also protects against replay after history eviction. Delayed/equal-time frames
cannot replace the latest state or add energy.

Per-channel forward/reverse differences are calculated in integer counter units.
The first frame establishes a baseline. Lower uptime starts a new boot baseline
without adding reset energy or erasing observed totals. An unexplained counter
decrease retains that channel's high-water baseline until recovery or reboot,
avoiding artificial consumption. Persistent totals mean **observed increases since
monitoring began**, not complete lifetime or billing totals. Gaps may omit power
samples but do not invalidate later counter differences. Average positive total
power × elapsed time / 3,600,000 is an advisory cross-check; a discrepancy greater
than max(0.01 kWh, 50% of the estimate) is explained, never used to rewrite counters.

Freshness is separate from MQTT status and FPort 2 activity: stale after three
estimated intervals. Three consistent observed gaps establish cadence; otherwise
the last requested interval is explicitly provisional, or the 60-second firmware
default is labeled as cadence unknown. Enable/disable requests remain unverified;
subsequent frames provide evidence, and silence alone never proves disablement.

Additive `manager_electrical_readings` and `manager_electrical_state` tables keep
monitoring separate from usage accounting. Raw frames remain in `manager_events`.
Both new tables are initialized safely on an existing database; no original table
is rebuilt. `GET /api/m3ter-manager/devices/:devEui/telemetry` provides the latest
state and paginated monitoring history (50 rows, `offset` parameter), protected by
the existing Manager session and application scope.

Charts plot **raw accumulator snapshots**, not independent interval consumption.
Repeated and growing values with the same nonce stay pending. A consecutive nonce
Expand All @@ -222,18 +288,38 @@ are flagged, not credited as fresh consumption. Manager sends no automatic ackno
Firmware source is not present here: observed transitions cannot establish unobserved
energy, reset causes, downlink execution, or blockchain settlement.

The database retains **100 uplink records globally by default**, including malformed
The database retains **100 usage-report records globally by default**, including malformed
payloads. Set optional `M3TER_TELEMETRY_LIMIT` to a positive integer to change this
limit. Invalid values log a warning and safely fall back to 100. The limit applies
on startup and after every insertion, and prunes their raw event copies too. A separate small per-device state
preserves the pending nonce/high-water mark and latest reading after eviction or
restart. Existing history is replayed once before migration trims it. Non-telemetry activity, command history, device configuration and all
original Console tables are preserved. SQLite reuses freed pages; migration does
not rewrite or vacuum unrelated tables. Date ranges cover the retained window only.
FPort 4 has a separate **global cap using the same `M3TER_TELEMETRY_LIMIT`** (default
100), including monitoring decode failures and raw event copies. Thus up to 100
usage reports plus 100 monitoring records are retained by default. FPort 4 cannot
evict usage history. Monitoring baselines, latest sample, and observed totals survive
both pruning and process restarts. Command records and original Console data are untouched.

### Command safety and delivery evidence

All 13 command types and their listed parameter variations are defined centrally.
All existing command types remain, with four additional telemetry actions (three
firmware command codes) defined centrally:

| Action | FPort | Hex | Base64 |
| --- | --- | --- | --- |
| Enable telemetry | 2 | `2301` | `IwE=` |
| Disable telemetry | 2 | `2300` | `IwA=` |
| Telemetry interval: 60 seconds | 2 | `24003C` | `JAA8` |
| Telemetry interval: 300 seconds | 2 | `24012C` | `JAEs` |
| Request Telemetry Now | 2 | `25` | `JQ==` |

Intervals are 10–65535 seconds; disable requires confirmation. Firmware v1.0.4+
is required for these new operations, and older-meter support is not inferred
merely from device discovery. Existing preview, idempotency, history, and queue
controls apply to all new actions.

The backend reconstructs payloads from IDs and validated integer parameters,
enforces lengths and FPorts 2/3, and rejects raw payload/FPort overrides. Class B,
calibration, and FUOTA ports 200–202 are not exposed. Billing, restart, relay
Expand Down
9 changes: 7 additions & 2 deletions src/m3ter/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { applicationId, chirpstackRest, deviceDetails, deviceId, getDevice, list
import { commandCatalogue, generateCommand } from './commands';
import { checkActivity, inactiveMilliseconds, managerEvents, monitoring } from './monitor';
import * as store from './store';
import { electricalSnapshot, electricalHistory } from './electrical-store';
import { managerAuthRouter, requireManagerAuth, managerAuthenticated, managerSessions } from './auth';

export const managerRouter = Router();
Expand Down Expand Up @@ -42,15 +43,19 @@ managerRouter.get('/devices/:devEui', wrap(async (req, res) => {
const devEui = deviceId(req.params.devEui), application = applicationId();
const details = await deviceDetails(devEui);
store.updateDevice(application, devEui, details);
res.json({ device: details, state: store.deviceState(application, devEui), latest: store.latestReading(application, devEui) });
res.json({ device: details, state: store.deviceState(application, devEui), latest: store.latestReading(application, devEui), electrical: electricalSnapshot(application, devEui) });
}));
managerRouter.get('/devices/:devEui/history', wrap((req, res) => {
const devEui = deviceId(req.params.devEui), application = applicationId();
const to = integer(req.query.to, Date.now(), 1, 8640000000000000);
const from = integer(req.query.from, to - 86400000, 0, to);
if (to - from > 366 * 86400000) throw new ManagerError('Choose a range of at most 366 days.');
res.json({ ...store.history(application, devEui, from, to, integer(req.query.offset, 0, 0, 1_000_000), integer(req.query.limit, 50, 1, 100)),
latest: store.latestReading(application, devEui), state: store.deviceState(application, devEui), monitoring });
latest: store.latestReading(application, devEui), state: store.deviceState(application, devEui), monitoring, electrical: electricalSnapshot(application, devEui) });
}));
managerRouter.get('/devices/:devEui/telemetry', wrap((req, res) => {
const application = applicationId(), devEui = deviceId(req.params.devEui);
res.json({ ...electricalSnapshot(application,devEui), ...electricalHistory(application,devEui,integer(req.query.offset,0,0,1_000_000)) });
}));
managerRouter.get('/devices/:devEui/events', wrap((req, res) => {
res.json({ events: store.events(applicationId(), deviceId(req.params.devEui), integer(req.query.offset, 0, 0, 1_000_000)) });
Expand Down
8 changes: 7 additions & 1 deletion src/m3ter/commands.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ export type CommandDefinition = {
id: string; name: string; category: string; fPort: 2 | 3; code: number;
description: string; confirmation?: string; options?: { value: number; label: string }[];
min?: number; max?: number; defaultValue?: number;
fixedValue?: number;
};
const gain = [1, 2, 4].map(value => ({ value, label: `×${value}` }));
const billing = 'The relay will change state, the energy accumulator will reset to zero, and the nonce will increment.';
Expand All @@ -19,6 +20,10 @@ export const commandCatalogue: CommandDefinition[] = [
{ id: 'phase-mode', name: 'Phase mode', category: 'Phase configuration', fPort: 2, code: 0x22, options: [{ value: 0, label: 'Three-phase' }, { value: 1, label: 'Phase A' }, { value: 2, label: 'Phase B' }, { value: 3, label: 'Phase C' }], description: 'Select reported phases without resetting energy. Three-phase reports total energy and the SUM of phase voltages (about 690 V on a 230 V system). A queued mode is not verified configuration.' },
{ id: 'class-a', name: 'Switch to Class A', category: 'LoRaWAN class', fPort: 3, code: 0, description: 'Downlinks become available only after an uplink.', confirmation: 'Switch to Class A? Future downlinks must wait for an uplink. Class C is the normal operating mode.' },
{ id: 'class-c', name: 'Switch to Class C', category: 'LoRaWAN class', fPort: 3, code: 2, description: 'Request the normal Class C operating mode. A reachable Class C meter can receive downlinks without waiting for an uplink.' },
{ id: 'telemetry-enable', name: 'Enable telemetry', category: 'Monitoring telemetry', fPort: 2, code: 0x23, fixedValue: 1, description: 'Enable periodic FPort 4 electrical monitoring (firmware v1.0.4+). Subsequent frames provide evidence of operation; queue acceptance is not confirmation.' },
{ id: 'telemetry-disable', name: 'Disable telemetry', category: 'Monitoring telemetry', fPort: 2, code: 0x23, fixedValue: 0, description: 'Stop periodic monitoring; FPort 2 usage reports continue.', confirmation: 'Disable periodic FPort 4 telemetry? Monitoring will become stale. Silence alone cannot confirm that the setting was applied.' },
{ id: 'telemetry-interval', name: 'Set telemetry interval', category: 'Monitoring telemetry', fPort: 2, code: 0x24, min: 10, max: 65535, defaultValue: 60, description: 'Set the FPort 4 monitoring interval in seconds. Below 60 seconds uses more LoRaWAN airtime. This does not change the FPort 2 usage-report interval.' },
{ id: 'telemetry-now', name: 'Request Telemetry Now', category: 'Monitoring telemetry', fPort: 2, code: 0x25, description: 'Request one FPort 4 frame immediately, even when periodic monitoring is disabled (firmware v1.0.4+).' },
];

export function generateCommand(id: unknown, value?: unknown) {
Expand All @@ -35,7 +40,8 @@ export function generateCommand(id: unknown, value?: unknown) {
bytes.push(value >> 8, value & 255);
}
} else if (value !== undefined && value !== null) throw new Error('This command takes no parameters');
const expectedLength = command.min !== undefined ? 3 : command.options ? 2 : 1;
if (command.fixedValue !== undefined) bytes.push(command.fixedValue);
const expectedLength = command.min !== undefined ? 3 : command.options || command.fixedValue !== undefined ? 2 : 1;
if (bytes.length !== expectedLength || ![2, 3].includes(command.fPort)) throw new Error('Invalid command frame');
const payload = Buffer.from(bytes);
return { id: command.id, name: command.name, description: command.description, fPort: command.fPort,
Expand Down
23 changes: 20 additions & 3 deletions src/m3ter/decoder.ts
Original file line number Diff line number Diff line change
@@ -1,23 +1,40 @@
import { decodePayload } from '../lib/decode';
import { verifyPayloadSignature } from '../lib/utils';
import { decodeElectrical, ElectricalTelemetry } from './electrical';

export type Telemetry = {
decoder: string; nonce: number; rawEnergy: number; energy: number; rawVoltage: number; voltage: number;
signature: string; publicKey: string | null; verificationKey: string | null; verification: 'verified' | 'invalid' | 'unverified';
};
// Additional firmware ports can register their own decoder without changing raw event storage.
export const telemetryDecoders = new Map<number, (data: Buffer, key?: string) => Telemetry>([[2, (data, knownKey) => {
export const telemetryDecoders = new Map<number, (data: Buffer, key?: string) => Telemetry | ElectricalTelemetry>([[2, (data, knownKey) => {
if (![74, 106].includes(data.length)) throw new Error('FPort 2 requires exactly 74 or 106 bytes');
const decoded = decodePayload(data);
const publicKey = decoded.extensions.deviceId || null;
const key = knownKey || publicKey;
const verification = key ? (verifyPayloadSignature(data, Buffer.from(key.replace(/^0x/, ''), 'hex')) ? 'verified' : 'invalid') : 'unverified';
return { decoder: 'switch-cumulus-v1', nonce: decoded.nonce, rawEnergy: data.readUInt32BE(4), energy: decoded.energy,
rawVoltage: data.readUInt16BE(72), voltage: decoded.extensions.voltage!, signature: decoded.signature, publicKey, verificationKey: key?.replace(/^0x/, '') || null, verification };
}]]);
}], [4, decodeElectrical]]);

export function decodeTelemetry(fPort: number, data: Buffer, knownKey?: string): Telemetry {
export function decodeTelemetry(fPort: 2, data: Buffer, knownKey?: string): Telemetry;
export function decodeTelemetry(fPort: 4, data: Buffer, knownKey?: string): ElectricalTelemetry;
export function decodeTelemetry(fPort: number, data: Buffer, knownKey?: string): Telemetry | ElectricalTelemetry;
export function decodeTelemetry(fPort: number, data: Buffer, knownKey?: string): Telemetry | ElectricalTelemetry {
const decode = telemetryDecoders.get(fPort);
if (!decode) throw new Error(`No telemetry decoder for FPort ${fPort}`);
return decode(data, knownKey);
}

/** A codec's report object may supply scaled fields; raw bytes still authenticate
* nonce/energy. Never accept an upstream claim that a signature was verified. */
export function reportObject(value: any, data: Buffer, knownKey?: string): Telemetry | null {
if (!value || value.type !== 'report' || ![74,106].includes(data.length)) return null;
if (value.nonce !== data.readUInt32BE(0) || value.energy !== data.readUInt32BE(4) / 1e6 || value.voltage !== data.readUInt16BE(72) / 10 ||
typeof value.signature !== 'string' || value.signature.toLowerCase().replace(/^0x/, '') !== data.subarray(8,72).toString('hex')) return null;
const publicKey = data.length === 106 ? data.subarray(74).toString('hex') : null;
const key = knownKey || publicKey;
return { decoder: 'switch-cumulus-v1', nonce: value.nonce, energy: value.energy, rawEnergy: data.readUInt32BE(4), voltage: value.voltage,
rawVoltage: data.readUInt16BE(72), signature: data.subarray(8,72).toString('hex'), publicKey, verificationKey: key?.replace(/^0x/,'') || null,
verification: key ? (verifyPayloadSignature(data,Buffer.from(key.replace(/^0x/,''),'hex')) ? 'verified' : 'invalid') : 'unverified' };
}
Loading
Loading