Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ STREAMR_CRONSCHEDULE="0 * * * *" # Every hour
# CENTRAL_DECISION_TIMEOUT_MS=8000
# CENTRAL_RECOVERY_HEALTHY_MS=15000
# AGENT_QUEUE_LIMIT=10000
# Gateway billing pruning (UTC): confirmed readings are removed only after central accepts them.
# GATEWAY_PRUNE_CRONSCHEDULE="*/5 * * * *"
# Never enable this outside isolated development fixtures.
# CENTRAL_ALLOW_INSECURE_DEVELOPMENT=false
# Only explicit reverse-proxy peer IPs; forwarded headers are otherwise ignored.
Expand Down
28 changes: 28 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -682,6 +682,34 @@ integration tests are complete.

## Central Console and gateway agent (opt-in test branch)

Managed gateways also transfer stored billing history to central, including
readings collected before enrollment. Every five seconds, while the central
connection is healthy, the agent can transfer one batch of up to 100 readings
from one meter, oldest nonces first. Central verifies signatures and nonce
continuity and merges missing rows without overwriting conflicting transactions.
Accepted batches are acknowledged durably on the gateway. Lost responses are
retried idempotently; historical transfers do not send device replies or create
new agent operation records. A rejected batch stays on the gateway for correction
and retry. Central's existing chain pruning and publishing policies still apply
to imported history; this is not a permanent archive of confirmed readings.

The gateway Console runs a pruning cron job every five minutes (UTC). Set
`GATEWAY_PRUNE_CRONSCHEDULE="0 * * * *"` for hourly pruning, for example. For each
meter, it reads the confirmed nonce from the chain and removes transaction rows
with `nonce <= confirmedNonce` only when the exact record was accepted by the
configured central/enrollment. Unconfirmed readings and readings awaiting central
acceptance remain. Chain failures or failed transfers preserve data for a later
run. A separate checkpoint retains at most one last confirmed reading per meter
so a device retry can still be acknowledged after its transaction row is pruned.
The checkpoint is not part of the pending `transactions` table.

Update central and both gateway services (`console` and `agent`) to enable this
policy. The agent negotiates history support with central; an older central keeps
the existing live billing connection and no new history acknowledgments are
created. Existing databases gain the receipt/checkpoint tables automatically.
This policy does not prune the agent's duplicate-detection operation journal or
change `AGENT_QUEUE_LIMIT`.

The default `standalone` role preserves the local collector and LAN desktop.
`central` adds authenticated multi-gateway access; `gateway` keeps a local fallback
Console while an outbound agent owns MQTT. The same default multi-architecture
Expand Down
Binary file modified docs/M3tering-Console-Manual.pdf
Binary file not shown.
12 changes: 12 additions & 0 deletions docs/deployment-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,18 @@ While central is unreachable, a managed gateway sends the readings it decides it
its own `.env` sets `PREFERRED_PROVER_NODE`. Central's scheduled runs resend them later anyway. A gateway
never publishes to Streamr or Arweave; central does that.

To send a batch now instead of waiting for the schedule, run this in the Console folder on central or
a standalone Console:

```sh
./prover-batch.sh # each meter's lowest 100 unconfirmed nonces
./prover-batch.sh --limit 250 # each meter's lowest 250
./prover-batch.sh --all # every unconfirmed reading, like the scheduled job
```

It prints, per meter, which nonces it sends, then the prover's answer. The lowest nonces go first because
they are the ones the prover needs next.

Logs: `Sending N transactions to prover at …`, then `Prover response (text): …`.

## The Streamr app
Expand Down
36 changes: 36 additions & 0 deletions prover-batch.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/usr/bin/env bash
# Send readings not yet confirmed on-chain to the prover now. For each meter: its lowest 100 nonces.
# Run in the Console folder on central or a standalone Console.
# ./prover-batch.sh lowest 100 unconfirmed nonces per meter
# ./prover-batch.sh --limit 250 lowest 250 per meter
# ./prover-batch.sh --all every unconfirmed reading (what the scheduled job sends)
set -euo pipefail
limit=100
while [[ $# -gt 0 ]]; do
case "$1" in
--limit) [[ $# -ge 2 && "$2" =~ ^[1-9][0-9]*$ ]] || { echo 'Use --limit followed by a positive whole number.' >&2; exit 2; }; limit=$2; shift 2 ;;
--all) limit=0; shift ;;
-h|--help) sed -n '2,6p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "Unknown option: $1 (use --limit N or --all)" >&2; exit 2 ;;
esac
done
cd -- "$(dirname -- "${BASH_SOURCE[0]}")"
[[ -f .env ]] || { echo 'No .env here: run this in the Console folder of a deployed Console.' >&2; exit 2; }
env_value() {
sed -n "s/^[[:space:]]*$1[[:space:]]*=[[:space:]]*//p" .env | tail -1 \
| sed -e 's/[[:space:]]*$//' -e 's/^"\(.*\)"$/\1/' -e "s/^'\(.*\)'\$/\1/"
}
# Use the role's compose file: the one ./auto-compose.sh recorded, or the one CONSOLE_ROLE implies.
compose_file=$(env_value COMPOSE_FILE)
if [[ -z "$compose_file" ]]; then
case "$(env_value CONSOLE_ROLE)" in
central) compose_file=compose.central.yaml ;;
gateway) compose_file=compose.gateway.yaml ;;
*) compose_file=compose.yaml ;;
esac
fi
project=$(env_value COMPOSE_PROJECT_NAME)
args=(docker compose -f "$compose_file")
[[ -z "$project" ]] || args+=(--project-name "$project")
[[ -n "$("${args[@]}" ps -q console 2>/dev/null)" ]] || { echo "The Console isn't running here ($compose_file). Start it with ./auto-compose.sh." >&2; exit 1; }
exec "${args[@]}" exec -T -e PROVER_BATCH_LIMIT="$limit" console node - < scripts/prover-batch.cjs
29 changes: 29 additions & 0 deletions scripts/prover-batch.cjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
// Sends readings not yet confirmed on-chain to the prover, now. For each meter it sends the lowest
// PROVER_BATCH_LIMIT nonces (default 100; 0 = all), the ones the prover needs next.
// Run through ./prover-batch.sh, inside the Console container.
const role = process.env.CONSOLE_ROLE || 'standalone';
const url = process.env.PREFERRED_PROVER_NODE;
if (role === 'gateway') { console.log('This is a managed gateway: central sends batches to the prover. Run this on central.'); process.exit(2); }
if (!url) { console.log('PREFERRED_PROVER_NODE is not set in .env, so there is no prover to send to.'); process.exit(2); }
require('/opt/app/dist/store/sqlite').default();
if (role === 'central') require('/opt/app/dist/central/distribution').configurePublications('.data/central.db');
const Prover = require('/opt/app/dist/lib/core/prover').default;
const { getSyncedPendingTransactions } = require('/opt/app/dist/lib/sync');
(async () => {
const pending = await getSyncedPendingTransactions(); // checks each meter on-chain and prunes confirmed readings first
console.log(pending.length + ' reading(s) not yet confirmed on-chain.');
if (!pending.length) return;
const limit = Number(process.env.PROVER_BATCH_LIMIT ?? 100);
if (!Number.isInteger(limit) || limit < 0) { console.log('PROVER_BATCH_LIMIT must be a whole number (0 = all).'); process.exit(2); }
const byMeter = new Map();
for (const record of pending) byMeter.set(record.identifier, [...(byMeter.get(record.identifier) || []), record]);
const batch = [];
for (const [meter, records] of [...byMeter].sort((a, b) => a[0] - b[0])) {
const lowest = records.sort((a, b) => a.nonce - b.nonce).slice(0, limit || records.length);
console.log('meter ' + meter + ': sending nonces ' + lowest[0].nonce + '-' + lowest[lowest.length - 1].nonce +
' (' + lowest.length + ' of ' + records.length + ' unconfirmed)');
batch.push(...lowest);
}
const response = await new Prover().sendPendingTransactionsToProver(url, batch);
console.log(response ? 'Prover answered: ' + await response.text() : 'Nothing was sent.');
})().then(() => process.exit(0), e => { console.error('Batch failed: ' + e.message); process.exit(1); });
22 changes: 20 additions & 2 deletions src/agent/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import { LocalDecider } from '../central/local-channel';
import { SocketDecider, parseOperation } from '../central/rpc';
import { parseMessage, MAX_MESSAGE_BYTES, Health, UplinkEvent } from '../central/protocol';
import { Replication } from '../central/replication';
import { syncGatewayHistory } from '../central/history';
import { mqttConnection } from '../lib/mqtt-config';
import { applicationId } from '../lib/chirpstack-config';
import { managedQueue } from '../services/grpc';
Expand Down Expand Up @@ -55,6 +56,7 @@ export async function runAgent():Promise<()=>Promise<void>> {
const local=new LocalDecider(gatewayFallbackPort(),readOwnerSecret(config.fallbackSecretPath),config.decisionTimeoutMs);
let socket:WebSocket|undefined,stopped=false,reconnect:ReturnType<typeof setTimeout>|undefined;
let connectedAt=0,attempt=0,mqttState:Health['mqtt']='connecting',lastError:Health['lastError']='none';
let historyAvailable=false;
const central=new SocketDecider(()=>socket,config.decisionTimeoutMs);
const replication=new Replication(journal,central,local);
const healthy=()=>socket?.readyState===WebSocket.OPEN&&Date.now()-connectedAt>=config.recoveryHealthyMs;
Expand All @@ -78,9 +80,11 @@ export async function runAgent():Promise<()=>Promise<void>> {
let failing:string|null=null;
const connectCentral=()=>{
if(stopped)return;
historyAvailable=false;
const ws=socket=new WebSocket(new URL('/agent',config.centralUrl),{headers:{Authorization:`Bearer ${currentCredential}`},maxPayload:MAX_MESSAGE_BYTES,
perMessageDeflate:false,handshakeTimeout:8000,followRedirects:false});
const inputLimit=new RateLimit(512,1000,1);
ws.on('upgrade',response=>{historyAvailable=response.headers['x-console-history']==='1';});
const tunnels=new TunnelMux(ws,request=>gatewayTarget(request));
ws.on('open',()=>{connectedAt=Date.now();attempt=0;lastError='none';
logger.info(failing?`Connected to central again (${config.centralUrl}).`:`Connected to central (${config.centralUrl}).`);failing=null;
Expand Down Expand Up @@ -158,16 +162,30 @@ export async function runAgent():Promise<()=>Promise<void>> {
}
};
let draining:Promise<void>|undefined;
let historySync:Promise<void>|undefined;
let historyFailed=false;
const timer=setInterval(()=>{
sendHealth();if(draining)return;
sendHealth();
// A bounded history batch runs independently of live decisions/replay. Slow
// chain validation on central must not hold up the gateway's fallback loop.
if(healthy()&&historyAvailable&&!historySync){
historySync=syncGatewayHistory(gateway,local,central).then(count=>{
if(count)logger.info(`Central accepted ${count} historical readings.`);
historyFailed=false;
}).catch(()=>{
if(!historyFailed)logger.warn('Historical synchronization deferred; readings stay on this gateway until central accepts them.');
historyFailed=true;
}).finally(()=>{historySync=undefined;});
}
if(draining)return;
draining=(async()=>{
try{const state=await updater.status();updatesEnabled=state.enabled;
if(socket?.readyState===WebSocket.OPEN)for(const job of state.jobs){if(updateStatuses.get(job.id)!==job.status){socket.send(JSON.stringify({v:1,type:'update-progress',id:job.id,status:job.status}));updateStatuses.set(job.id,job.status);}}
}catch{updatesEnabled=false;}
try{await replication.flush('local');}catch{}try{await replication.flush('central');}catch{}await coordinator.drain();})()
.catch(()=>{lastError='storage';}).finally(()=>{draining=undefined;});
},5000);timer.unref();connectCentral();
const stop=async()=>{stopped=true;stopResolverWatch();clearTimeout(reconnect);clearInterval(timer);central.disconnect();socket?.terminate();await mqtt.endAsync(false);await coordinator.stop();await draining;journal.close();};
const stop=async()=>{stopped=true;stopResolverWatch();clearTimeout(reconnect);clearInterval(timer);central.disconnect();socket?.terminate();await mqtt.endAsync(false);await coordinator.stop();await draining;await historySync;journal.close();};
const stopResolverWatch=watchResolver(()=>{
logger.error('This container has no DNS server (Docker copied the host\'s DNS settings before its network was up). Restarting so Docker provides fresh ones.');
void stop().finally(()=>process.exit(1));
Expand Down
4 changes: 2 additions & 2 deletions src/central/billing.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ export function createBillingProcessor(filename:string):DurableProcessor {
const logger=createLogger('[central]');
/** Accept another decider's billing state (a gateway's fallback decisions) after
* checking it against the meter's signatures, its token and the chain. */
export async function importBillingSnapshot(snapshot:BillingSnapshot):Promise<void> {
export async function importBillingSnapshot(snapshot:BillingSnapshot, distribute=true):Promise<void> {
const meter=snapshot.meter;if(!meter)return;
const known=getMeterByPublicKey(meter.publicKey);
const token=known?.tokenId ?? await meterIdFromPublicKeyContractCall(meter.publicKey);
Expand All @@ -48,6 +48,6 @@ export async function importBillingSnapshot(snapshot:BillingSnapshot):Promise<vo
mergeBillingSnapshot(snapshot);
// Only central publishes what another decider handed over. (Older gateways may list readings they
// already sent to the prover; that is ignored: the prover is resent until the chain confirms.)
if(process.env.CONSOLE_ROLE!=='central')return;
if(process.env.CONSOLE_ROLE!=='central'||!distribute)return;
void distributeManaged(snapshot.transactions).catch(()=>{});
}
30 changes: 30 additions & 0 deletions src/central/gateway-pruning.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
import cron from 'node-cron';
import { getAllMeterRecords, pruneGatewayHistory } from '../store/sqlite';
import { meterLatestNonceContractCall } from '../utils/contracts';
import { createLogger } from '../utils/logger';

const logger = createLogger('[gateway-pruning]');

export async function pruneConfirmedGatewayHistory(target: string): Promise<number> {
let removed = 0;
for (const meter of getAllMeterRecords()) {
try {
const confirmed = await meterLatestNonceContractCall(meter.tokenId);
removed += pruneGatewayHistory(target, meter.tokenId, confirmed);
} catch {
// A failed chain read is never evidence that a local record may be deleted.
logger.warn(`Kept readings for meter ${meter.tokenId}: confirmation or pruning failed.`);
}
}
if (removed) logger.info(`Removed ${removed} confirmed readings already accepted by central.`);
return removed;
}

export function startGatewayPruning(target: () => string) {
const schedule = process.env.GATEWAY_PRUNE_CRONSCHEDULE || '*/5 * * * *';
if (!cron.validate(schedule)) throw new Error('GATEWAY_PRUNE_CRONSCHEDULE must be a valid cron expression.');
return cron.schedule(schedule, async () => {
try { await pruneConfirmedGatewayHistory(target()); }
catch { logger.warn('Pruning deferred until gateway enrollment is ready.'); }
}, { name: 'gateway-prune-confirmed-history', noOverlap: true, timezone: 'UTC' });
}
13 changes: 13 additions & 0 deletions src/central/history.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import type { Decider } from './processor';

/** Separate from live operation replay: historical rows must never enqueue a
* device reply or consume an agent operation/idempotency slot. A lost response
* leaves the batch unacknowledged locally, so the same bytes are retried. */
export async function syncGatewayHistory(gateway: string, local: Decider, central: Decider): Promise<number> {
if (!local.history || !central.history) throw new Error('History synchronization is unavailable.');
const batch = await local.history(gateway, 'history-read');
if (!batch) return 0;
await central.history(gateway, 'history-import', batch);
await local.history(gateway, 'history-ack', batch);
return batch.transactions.length;
}
7 changes: 6 additions & 1 deletion src/central/hub.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@ export class GatewayHub {
};
mount(app:Express,server:http.Server):void {
const wss=new WebSocket.Server({noServer:true,maxPayload:MAX_MESSAGE_BYTES,perMessageDeflate:false});
// Capability negotiation keeps upgraded gateways from sending new RPCs to
// older central servers, which would close the live billing connection.
wss.on('headers',headers=>headers.push('X-Console-History: 1'));
registerSocket(server,'/agent',wss,req=>{
if((!this.development&&!this.auth.secure(req))||!this.upgradeLimit.allow(req.socket.remoteAddress||'unknown'))return false;
const token=req.headers.authorization?.match(/^Bearer ([A-Za-z0-9_-]{43})$/)?.[1];
Expand All @@ -55,7 +58,9 @@ export class GatewayHub {
let message;try{message=parseOperation(data.toString());if(message.type!=='operation')throw new Error();}catch{socket.close(1008,'Invalid operation');return;}
inFlight++;
void serveOperation(this.processor(),gateway,message).then(reply=>{// Learn which gateway serves a meter, also from reports it decided while central was away.
if(!reply.error&&(message.action==='offer'||message.action==='reconcile')&&message.event)this.identities.observeDevice(message.event.deviceInfo.devEui,gateway);if(socket.readyState===WebSocket.OPEN)socket.send(JSON.stringify(reply));}).catch(()=>socket.close(1011,'Operation storage unavailable')).finally(()=>inFlight--);
if(!reply.error&&(message.action==='offer'||message.action==='reconcile')&&message.event)this.identities.observeDevice(message.event.deviceInfo.devEui,gateway);
if(!reply.error&&message.action==='history-import'&&message.history?.meter?.devEui)this.identities.observeDevice(message.history.meter.devEui,gateway);
if(socket.readyState===WebSocket.OPEN)socket.send(JSON.stringify(reply));}).catch(()=>socket.close(1011,'Operation storage unavailable')).finally(()=>inFlight--);
return;
}
try {
Expand Down
Loading