Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,11 @@ PORT=3000
# Listen on LAN interfaces as well as localhost.
CONSOLE_HOST=0.0.0.0

# Only M3TER Manager requires login. Console itself is public.
# Only M3ter Manager requires login. Console itself is public.
M3TER_MANAGER_USERNAME=admin
M3TER_MANAGER_PASSWORD=
# Optional global telemetry history limit; invalid values fall back to 100.
# M3TER_TELEMETRY_LIMIT=100

# Module Configuration
BACKEND_MODULES="core/arweave,core/is_on,core/prover,core/streamr"
Expand All @@ -17,11 +19,15 @@ APPLICATION_ID=...
CHIRPSTACK_HOST=localhost
# Optional web UI origin; defaults to http://CHIRPSTACK_HOST:8080.
# Browsers access it through the console's /chirpstack/ proxy.
# CHEFSTACK_URL=http://localhost:8080
# CHIRPSTACK_WEB_URL remains supported; CHEFSTACK_URL takes precedence.
# CHIRPSTACK_WEB_URL=http://localhost:8080
# M3TER Manager uses the existing ChirpStack v4 REST proxy and the same API_TOKEN.
# M3ter Manager uses the existing ChirpStack v4 REST proxy and the same API_TOKEN.
# This is the REST proxy's port, NOT the ChirpStack web UI port.
# CHIRPSTACK_REST_URL=http://localhost:8090
# MQTT reuses CHIRPSTACK_HOST on port 1883, just like the collector.
# Both consumers share this optional broker URL (mqtt/mqtts/ws/wss).
# When absent, preserve CHIRPSTACK_HOST on port 1883, default localhost.
# MQTT_URL=mqtt://localhost:1883
# M3TER_INACTIVE_SECONDS=300

# Contract & Network Configuration
Expand Down
65 changes: 53 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,9 +76,9 @@ npm install
npm run dev
```

## M3TER Manager
## M3ter Manager

**M3TER Manager** is a built-in desktop application for Switch Cumulus telemetry
**M3ter Manager** is a built-in desktop application for Switch Cumulus telemetry
and technician downlinks. Launch it from the desktop, dock, or application
launcher. Its Overview, Live monitoring, Analytics, Commands, Queue, and Meter
information sections share the selected device, desktop theme, and responsive
Expand Down Expand Up @@ -114,8 +114,10 @@ is needed. The API token needs device/application/profile read access and device
queue enqueue/read/flush access for the configured application. MQTT requires
permission to subscribe to that application's events. Configure ChirpStack's
MQTT integration to publish JSON. Manager reuses the Console's original
`CHIRPSTACK_HOST` connection on port 1883. No new MQTT settings or separate client
are required. Opening or closing Manager does not own or stop this connection.
`CHIRPSTACK_HOST` connection on port 1883 (localhost by default). Optional
`MQTT_URL=mqtt://broker.example:1883` overrides the shared broker address for both
consumers; `mqtts://`, `ws://`, and `wss://` are also supported. URL credentials,
when required by the broker, stay on the backend. No separate client is created. Opening or closing Manager does not own or stop this connection.
Both consumers receive messages independently; monitoring errors cannot disable
the collector. The old monitor-only toggle has been removed: collection remains enabled.

Expand All @@ -125,7 +127,7 @@ npm start
```

Then open the console on its configured port (3000 by default) and
launch **M3TER Manager**. The dashboard stays accessible while MQTT reconnects.
launch **M3ter Manager**. The dashboard stays accessible while MQTT reconnects.

### Architecture and API

Expand Down Expand Up @@ -198,7 +200,17 @@ bounded queries. Uplinks deduplicate by ChirpStack event ID, or by a stable tupl
of device/event type/source time/frame counter/payload when an ID is absent.
Identical values with distinct events are preserved. Without an event ID or
source timestamp, events are retained separately rather than risking data loss.
Source timestamps are preserved in metadata; invalid timestamps use receipt time.
Source timestamps are preserved in metadata; invalid timestamps use receipt time. The Overview's
Last Uplink uses the persisted console reception time of a successfully decoded,
non-duplicate uplink. MQTT heartbeats, connection events, UI refreshes and decode
errors cannot reset it. Source timestamps still drive the existing Analytics.

MQTT payload transport is explicit: native ChirpStack `data` remains Base64;
hex producers supply `dataHex` or `data` with `dataEncoding: "hex"`. Hex must contain
complete byte pairs, without separators or a `0x` prefix. No alphabet guessing is
used. Both collector and Manager use the same byte conversion. Live Monitoring
shows the hexadecimal bytes and preserves the original field, encoding, and text
for inspection. Downlink encoding and firmware decoders are unchanged.

Charts plot **raw accumulator snapshots**, not independent interval consumption.
Repeated and growing values with the same nonce stay pending. A consecutive nonce
Expand All @@ -210,8 +222,10 @@ are flagged, not credited as fresh consumption. Manager sends no automatic ackno
Firmware source is not present here: observed transitions cannot establish unobserved
energy, reset causes, downlink execution, or blockchain settlement.

The database retains at most **100 uplink records globally**, including malformed
payloads, and prunes their raw event copies too. A separate small per-device state
The database retains **100 uplink records globally by default**, including malformed
payloads. Set optional `M3TER_TELEMETRY_LIMIT` to a positive integer to change this
limit. Invalid values log a warning and safely fall back to 100. The limit applies
on startup and after every insertion, and prunes their raw event copies too. A separate small per-device state
preserves the pending nonce/high-water mark and latest reading after eviction or
restart. Existing history is replayed once before migration trims it. Non-telemetry activity, command history, device configuration and all
original Console tables are preserved. SQLite reuses freed pages; migration does
Expand Down Expand Up @@ -273,7 +287,10 @@ Open **ChirpStack** from the desktop, dock, or application launcher. Its web
interface opens inside a console window; the browser uses `/chirpstack/` on the
console's own port. The console server connects to
`http://CHIRPSTACK_HOST:8080` (localhost by default). Set
`CHIRPSTACK_WEB_URL=http://your-chirpstack-host:8080` to override the web endpoint,
`CHEFSTACK_URL=http://your-chirpstack-host:8080` to override the web endpoint.
The existing `CHIRPSTACK_WEB_URL` remains supported, with `CHEFSTACK_URL` taking
precedence. Use an HTTP(S) origin without a path or URL credentials. This changes
the web integration only; keep the REST proxy and gRPC configuration as appropriate,
or use an HTTPS origin. Only an origin is supported, without credentials or a
subpath. In Docker this address must be reachable from the console container.

Expand Down Expand Up @@ -835,10 +852,10 @@ appearance controls; the containing window follows the application setting.
To update the existing Docker deployment from GitHub Actions images:

```sh
./install.sh
./install.sh --project-name your-existing-project
./auto-compose.sh
./auto-compose.sh --project-name your-existing-project
# To select a fork/branch image published by the existing workflow:
./install.sh --image ghcr.io/fluxion9/console:feat-revamp
./auto-compose.sh --image ghcr.io/fluxion9/console:feat-revamp
```

Docker and its Compose plugin must already be installed. The script identifies
Expand All @@ -853,3 +870,27 @@ volumes, database files or `.env` are removed. Other services are never brought
`compose.local.yaml` remains available for deliberate developer builds; the update
script does not use it. Readiness does not certify broker or blockchain availability.
Use the original project name and data directory to preserve the same volumes.

## Targeted desktop refinements

The old **M3ters** meter-list application has been removed from the desktop,
along with its exclusive `/delete-meter` UI endpoint. The collector's database,
MQTT service, and meter records are preserved. **M3ter Manager** remains available.

The Browser starts with a Google search interface. Queries use
`https://www.google.com/search?q=` with URL encoding and open in a new tab because
Google blocks embedded search pages. HTTP(S) addresses can still open inside the
Browser; its external link supports sites that block embedding.

`compose.yaml` passes the existing `.env` into the single Console container,
which hosts both consumers. The update script leaves that file and persistent
volumes intact, including `MQTT_URL`, `CHEFSTACK_URL`, `M3TER_TELEMETRY_LIMIT`, and
`M3TER_MANAGER_PASSWORD`. No deployment-script or workflow changes are required.

Application feedback is scoped to its owning window. Extension actions preserve
the originating window even if another app is focused while a request completes.
Use `showNotification(message, type, appElementOrId)` for extension feedback, and
pass `null` for a desktop-wide notice. `AppFeedback.showDialog(dialog, appElement)`
provides an app-contained dialog; `AppFeedback.confirm(appElement, title, message)`
returns a promise for confirmations. These dialogs allow switching to other apps
and inherit the owning application's light/dark mode.
2 changes: 1 addition & 1 deletion install.sh → auto-compose.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ while [[ $# -gt 0 ]]; do
if [[ "$1" == --project-name ]]; then project="$2"; else image="$2"; fi
shift 2 ;;
--help|-h)
echo 'Usage: ./install.sh [--project-name NAME] [--image ghcr.io/OWNER/console:TAG]'
echo 'Usage: ./auto-compose.sh [--project-name NAME] [--image ghcr.io/OWNER/console:TAG]'
echo 'Pulls the published image for this device, updates only Console, and preserves .env and all volumes.'
exit 0 ;;
*) echo "Unknown option: $1" >&2; exit 2 ;;
Expand Down
2 changes: 1 addition & 1 deletion compose.local.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Used by ./install.sh to deploy the current checkout with production data volumes.
# Optional local source build override; auto-compose.sh deploys published images using compose.yaml.
services:
console:
image: m3tering-console:local
Expand Down
20 changes: 20 additions & 0 deletions src/lib/uplink-payload.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
/** Transport decoding only. Firmware decoders consume bytes, never encoded text.
* ChirpStack v4's unmarked `data` is Base64. Hex producers must use `dataHex`
* or explicitly mark `dataEncoding: 'hex'`; alphabet guessing is ambiguous.
*/
export function uplinkPayload(event: { data?: unknown; dataHex?: unknown; dataEncoding?: unknown }) {
const explicitHex = event.dataHex !== undefined;
const field = explicitHex ? 'dataHex' : 'data';
const raw = explicitHex ? event.dataHex : event.data;
const encoding = explicitHex ? 'hex' : event.dataEncoding ?? 'base64';
if (typeof raw !== 'string' || !raw || raw.length > 4096) throw new Error('Missing or oversized uplink payload');
if (encoding !== 'hex' && encoding !== 'base64') throw new Error('Unsupported payload encoding; use hex or base64');
if (encoding === 'hex') {
if (!/^(?:[a-fA-F0-9]{2})+$/.test(raw)) throw new Error('Invalid hex payload: expected complete hexadecimal byte pairs');
} else if (!/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/.test(raw)) {
throw new Error('Invalid Base64 payload; hexadecimal data must use dataHex or dataEncoding: hex');
}
const bytes = Buffer.from(raw, encoding);
if (encoding === 'base64' && bytes.toString('base64') !== raw) throw new Error('Invalid Base64 payload');
return { bytes, raw, field, encoding, hex: bytes.toString('hex').toUpperCase() };
}
2 changes: 1 addition & 1 deletion src/m3ter/auth.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ export function managerAuthenticated(req: IncomingMessage): boolean {
}
export function requireManagerAuth(req: Request, res: Response, next: NextFunction): void {
res.setHeader('Cache-Control', 'no-store');
if (!managerAuthenticated(req)) { res.status(401).json({ error: 'Sign in to M3TER Manager to continue.' }); return; }
if (!managerAuthenticated(req)) { res.status(401).json({ error: 'Sign in to M3ter Manager to continue.' }); return; }
next();
}
function setCookie(req: Request, res: Response, value: string, maxAge: number) {
Expand Down
10 changes: 10 additions & 0 deletions src/m3ter/config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
import { createLogger } from '../utils/logger';

const logger = createLogger('[manager]');
export function telemetryLimit(): number {
const value = process.env.M3TER_TELEMETRY_LIMIT;
if (value === undefined) return 100;
if (/^[1-9]\d*$/.test(value) && Number.isSafeInteger(Number(value))) return Number(value);
logger.warn('Invalid M3TER_TELEMETRY_LIMIT; using the safe default of 100 records. Set a positive integer.');
return 100;
}
16 changes: 12 additions & 4 deletions src/m3ter/monitor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { decodeTelemetry } from './decoder';
import * as store from './store';
import { applicationId, deviceId } from './chirpstack';
import { getMeterByDevEui } from '../store/sqlite';
import { uplinkPayload } from '../lib/uplink-payload';

export const managerEvents = new EventEmitter();
managerEvents.setMaxListeners(128);
Expand Down Expand Up @@ -40,23 +41,30 @@ export function ingestEvent(topic: string, blob: Buffer): boolean {
const receivedAt = Date.now(), parsedTime = Date.parse(event.time);
const timestamp = Number.isFinite(parsedTime) && parsedTime > 0 && parsedTime < receivedAt + 300_000 ? parsedTime : receivedAt;
const kind = parts[3].toLowerCase();
const identity = event.deduplicationId || (event.time ? [event.time, event.fCnt, event.devAddr, event.data, event.queueItemId, event.acknowledged, event.downlinkId] : randomUUID());
const fields = [event.time, event.fCnt, event.devAddr, event.data, event.queueItemId, event.acknowledged, event.downlinkId];
if (event.dataHex !== undefined || event.dataEncoding !== undefined) fields.push(event.dataHex, event.dataEncoding);
const identity = event.deduplicationId || (event.time ? fields : randomUUID());
const eventKey = createHash('sha256').update(JSON.stringify([devEui, kind, identity])).digest('hex');
const rx = Array.isArray(event.rxInfo) ? event.rxInfo[0] : null;
const metadata: Record<string, unknown> = { sourceTime: typeof event.time === 'string' ? event.time.slice(0,100) : null, timestampSource: timestamp === parsedTime ? 'ChirpStack' : 'console receipt', name: String(event.deviceInfo.deviceName || devEui).slice(0, 200),
applicationName: String(event.deviceInfo.applicationName || '').slice(0, 200), deviceProfileName: String(event.deviceInfo.deviceProfileName || '').slice(0, 200),
fCnt: Number.isInteger(event.fCnt) ? event.fCnt : null, rssi: typeof rx?.rssi === 'number' ? rx.rssi : null,
snr: typeof rx?.snr === 'number' ? rx.snr : null, gatewayId: typeof rx?.gatewayId === 'string' ? rx.gatewayId.slice(0, 32) : null };
const rawPayload = typeof event.data === 'string' ? event.data.slice(0, 4096) : null;
const suppliedPayload = event.dataHex !== undefined ? event.dataHex : event.data;
const rawPayload = typeof suppliedPayload === 'string' ? suppliedPayload.slice(0, 4096) : null;
metadata.payloadField = event.dataHex !== undefined ? 'dataHex' : 'data';
metadata.payloadEncoding = event.dataHex !== undefined ? 'hex' : event.dataEncoding ?? 'base64';
if (typeof event.data === 'string' && event.dataHex !== undefined) metadata.upstreamData = event.data.slice(0, 4096);
const record: store.ManagerEvent = { application, devEui, kind, source: 'ChirpStack', timestamp, receivedAt, eventKey, rawPayload,
fPort: Number.isInteger(event.fPort) ? event.fPort : null, metadata, message: `ChirpStack ${kind} event` };
let inserted: boolean;
if (kind === 'up') {
let telemetry;
try {
if (!rawPayload || !/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/.test(rawPayload) || rawPayload !== event.data) throw new Error('Invalid Base64 payload');
const payload = uplinkPayload(event);
metadata.payloadHex = payload.hex;
const knownKey = store.deviceState(application, devEui)?.publicKey || getMeterByDevEui(devEui)?.publicKey;
telemetry = decodeTelemetry(event.fPort, Buffer.from(rawPayload, 'base64'), knownKey);
telemetry = decodeTelemetry(event.fPort, payload.bytes, knownKey);
record.message = telemetry.verification === 'invalid' ? 'Telemetry decoded; signature INVALID. Voltage is outside the signed fields.' : 'Uplink decoded';
} catch (error) { record.kind = 'decode-error'; record.message = (error as Error).message; }
inserted = store.recordUplink(record, telemetry, inactiveMilliseconds());
Expand Down
Loading
Loading