Reduce duplicate R2 cache refresh writes

docs/live-visitors.md

@@ -7,14 +7,46 @@ function. Admin queries refresh every three seconds while the page is visible.
 The dashboard opens with a company grid. Individual sessions are hidden until
 a company is expanded, and each session has its own expandable tabs and activity.
 Unmatched traffic stays in one collapsed row at the bottom. Columns support
-sorting, and search matches companies, industries, libraries, and page paths.
+multi-column sorting with Shift-click, column visibility, row selection,
+pagination, compact density, and CSV export. Search matches companies, industries,
+libraries, and page paths. Library, industry, size, and funnel-stage filters can
+be combined with minimum website views and recent actions. Values within a
+filter match with OR, while different filters match with AND.
 
 The overview counts identified companies once per domain. Library breakdowns
 count a company once per library, so one company can appear in several bars.
-Industry breakdowns exclude unmatched traffic. The company trend records real
-polling snapshots in the open dashboard tab, up to five minutes, and resets on
-reload. Grid action totals count named interactions in the last five minutes,
-excluding page views.
+Industry and size breakdowns exclude unmatched traffic. Clicking a breakdown
+applies its filter. Size buckets use the lower bound of Scarf's employee range.
+The status widget and sortable Status column classify a company as Active when
+at least one online tab is visible, Background when all online tabs are hidden,
+or Left when no tabs remain online. Each company occupies one status, even when
+it has both visible and hidden tabs. Active describes visibility, not confirmed
+attention or interaction. Status filters combine with the other company filters.
+Grid action totals count named interactions in the last five minutes, excluding
+page views. Pause freezes polling, including expanded sessions and history.
+
+## Scarf company history
+
+The server uses the existing `ANALYTICS_SCARF_API_TOKEN` bearer token for Scarf's
+`/v3/insights/TanStack/aggregations/export` endpoint. Reporting is separate from
+the lookup key and requires Scarf reporting access. Exports cover the selected
+7, 30, 90, or 365 UTC calendar days, including today. Growth compares website
+pixel views with the preceding equal-length window, so today's totals are
+incomplete. These are window totals, not lifetime totals.
+
+Website views include only the site's registered Scarf pixels. Site events
+come from the TanStack Site Telemetry package, including browser interactions
+and server requests. Events are not added to views. Sources and endpoints are
+Scarf's distinct network counts, not people. Company reports join live company
+domains exactly, without guessing parent-company relationships. Missing rows or
+failed reports show N/A rather than zero. No company data is automatically
+unlocked and no Scarf credits are spent on enrichment.
+
+Successful aggregate exports are cached at the edge for 15 minutes and partial
+or unavailable responses for one minute. Live presence still refreshes every
+three seconds. Sorting and filtering do not request new exports. Only admins
+can request reporting data, and source identifiers and credentials are never
+returned to the browser.
 
 ## Enable the experiment
 
@@ -46,9 +78,11 @@ contact, not proof that a person is looking at the page. `pageshow` handles
 restoration from the browser's back/forward cache. Sequence numbers prevent a
 late heartbeat or departure from overwriting newer tab state.
 
-Each tab retains up to 40 named actions. The admin includes departures from the
-last five minutes when requested and returns at most 500 recently seen tabs,
-with a visible truncation notice. Sessions disappear from reads at 24 hours and
+Each tab retains up to 40 named actions. The company overview aggregates all
+eligible tabs without a 500-tab cap. Individual company sessions are fetched
+only when expanded, with at most 500 eligible recently seen tabs per company
+and a visible truncation notice. The admin includes departures from the last
+five minutes when requested. Sessions disappear from reads at 24 hours and
 are deleted, with their tabs, on the next minute cleanup.
 
 ## Company lookup
@@ -82,17 +116,35 @@ DNT and GPC prevent collection in both the browser and server. Browser collectio
 is disabled in development, staging, and outside `tanstack.com`.
 
 `isLivePage` allows public library content and selected public site routes.
+Showcase submission, editing, and individual detail pages are excluded because
+detail pages can include owner-only drafts. Existing records with excluded paths
+are filtered out of admin responses and expire through normal retention.
 Account, admin, auth, chat, private builder projects, shop, and internal endpoints
 are excluded. Unknown paths are excluded until added to the public allowlist.
 Queries, fragments, clipboard text, form values, event properties, account IDs,
 emails, and user agents are not stored. Only explicitly named actions from the
 existing analytics hooks are collected; there is no arbitrary click-text capture.
 
-Visitor IPs exist only during the server-side lookup and are not written to the
-presence tables or admin responses. Lookup tracing and Sentry breadcrumbs are
+Visitor IPs are used transiently for the server-side lookup and Cloudflare edge
+rate-limit counters, and are not written to the application database or admin
+responses. Lookup tracing and Sentry breadcrumbs are
 suppressed to keep IP query parameters out of telemetry. The lookup key stays
 server-side. There is no joining sessions by IP, fingerprint, or login.
 
+## Intake limits
+
+Cloudflare edge bindings limit intake to 600 requests per IP per minute and new
+company lookups to 30 per IP per minute, per Cloudflare location. These are
+approximate abuse limits, not a global billing quota. Intake returns 429 before
+reading a body or writing presence when limited, and 503 if the limiter is
+unavailable. A lookup limit leaves presence intact with a separate rate-limited
+status. Corporate networks share an IP, so these limits can affect shared
+networks during large bursts. Raw IPs are not added to the database-backed MCP
+rate-limit table. Presence cleanup runs independently of workflow processing, so
+a cleanup failure does not prevent scheduled workflow work.
+
+[Cloudflare rate-limit bindings](https://developers.cloudflare.com/workers/runtime-apis/bindings/rate-limit/)
+
 ## Validation
 
 Focused tests cover privacy signals, intake validation, public-route boundaries,

docs/visitor-chat.md

@@ -0,0 +1,112 @@
+# Visitor chat
+
+Visitor chat is enabled in production with `VISITOR_CHAT_ENABLED=true`.
+Set it to `false` to stop chat requests and invitations.
+
+Admins open a company, choose a browser session, and send the first message.
+That message opens a small visitor chat using the sender's display name and
+profile image. Existing conversations open their transcript directly. Every
+admin with access to the live dashboard shares the same inbox and can reply in
+the same thread. Senders are assigned from the authenticated server session,
+never from browser-provided names or account IDs. No access is granted to new
+users by this feature.
+
+## Loading and delivery
+
+The existing 20-second presence request carries the invitation signal. Its
+response stays empty until a conversation has a committed first message.
+Only then does the browser import the visitor chat module and stylesheet.
+Chat-specific utilities are excluded from the initial site stylesheet and built
+into deferred stylesheets for the visitor window and admin inbox. The shared admin
+inbox is also a separate lazy module, opened by Chats or a session's chat action.
+The bridge adds no always-open chat connection or additional idle polling loop.
+A non-secret browser hint restores an existing chat on reload, including when
+presence tracking is off or the visitor is on a private route. The API still
+requires the private signed cookie, and the hint never grants access to messages.
+The root page contains only shared feature flags, so public HTML caching cannot
+cause chat resources to load for another visitor. This does not collect private
+page paths. Visitors with a page loaded
+before this release must reload before chat becomes
+available, and the admin action explains that state.
+
+Invitations can take up to 20 seconds to appear. Once loaded, visitor and admin
+messages poll every two seconds. The browser acknowledges the messages it
+actually received, and only marks them seen while the chat is expanded in a
+visible tab. These are browser signals, not proof of attention. The admin's
+Chats badge and inbox show conversations whose latest message needs a reply.
+Chat updates continue independently of pausing the live-company grid.
+
+Conversation row locks order simultaneous sends. Client message IDs make retries
+idempotent. History loads in pages of 100, and older pages remain available.
+Messages are plain text. Enter sends and Shift+Enter inserts a line break.
+There are no attachments, email delivery, push notifications, or typing indicators
+in this version. Leaving the site does not turn a message into an email.
+
+## Identity and retention
+
+A random private proof accompanies the existing anonymous presence session. The
+server stores its hash. The proof is never returned in admin responses. It binds
+the initial invitation to the intended browser, rather than treating a visible
+session ID as permission to read messages.
+
+Once invited, the visitor receives a signed, HttpOnly, Secure, SameSite cookie.
+It has a separate signing purpose from account cookies and lasts one year.
+Reloads, tabs, and later presence sessions can resume the same conversation.
+Clearing cookies, changing browsers, expiration, or signing-key rotation can
+remove the visitor's connection to retained history. No IP, fingerprint, or
+logged-in visitor account is used to recover it. Conversations target individual
+browser sessions, not everyone at the matched company.
+
+Chat history remains until an admin manually deletes it. Closing a conversation
+keeps its history and permits reopening. Stop chat hides the visitor interface,
+mutes the conversation, and sets a browser opt-out cookie. Admins cannot reopen
+a visitor-muted conversation. Minimizing the window persists across reloads and
+does not mark new messages seen. The latest unread messages are counted in its
+collapsed header.
+
+The chat tables do not depend on the 24-hour lifetime of presence records.
+Sender names and avatar snapshots stay with their messages if an admin account
+changes or is deleted. The company snapshot describes the network at initiation,
+not a verified employer, and is shown only to admins. Manual deletion cascades
+to all messages and clears presence references.
+
+## Boundaries
+
+All admin reads and writes check the current admin capability. Visitor APIs use
+the signed chat cookie, not a supplied chat ID, and reject cross-site writes.
+The new Cloudflare `VISITOR_CHAT_RATE_LIMITER` binding bounds requests per IP
+without storing IPs in the application database. Missing or unavailable limits
+fail closed. Message size and request body size are bounded.
+
+Chat regions are excluded from clipboard analytics, including selections that
+span a chat region. Message bodies, cookies, visitor details, and database
+parameters are removed from Sentry events and transactions for these endpoints.
+Driver write errors are replaced with a message that contains no SQL parameters.
+The chat interface does not send message content to Scarf or GA.
+
+## Release
+
+1. Apply migration `0043_visitor_chat` through the normal migration flow.
+2. Deploy the site Worker with the `VISITOR_CHAT_RATE_LIMITER` binding. The existing
+   `SESSION_SECRET` signs the separately scoped visitor credential.
+3. Verify invitation loading, two different admins in one thread, reload recovery,
+   manual closure/deletion, visitor opt-out, and cold-page chunk loading.
+4. Enable `VISITOR_CHAT_ENABLED=true` when ready to invite production visitors.
+
+Local integration tests require a disposable, migrated `tanchat_test` PostgreSQL
+database, and refuse a production database URL. The preview used synthetic users
+and conversations and did not contact real site visitors.
+
+## Test your own chat
+
+Click **Test chat** in the live dashboard. It opens the public site in a new tab
+with a test message from your admin profile, and opens the matching admin tab
+labeled **Test chat · You**. Reply in either tab to test both sides. The visitor
+tab uses the same window, history, reload recovery, and opt-out as a real chat.
+Allow pop-ups for tanstack.com if the browser blocks the new tab.
+
+This action requires admin access. It creates a separate conversation labeled
+Test chat, without waiting for Scarf or adding a synthetic presence session.
+The signed visitor cookie connects only your own browser to that conversation.
+Each test creates a new conversation and makes it this browser's visitor chat.
+Delete the test from the admin tab when finished. Other visitors are unaffected.

drizzle/migrations/0043_visitor_chat.sql

@@ -0,0 +1,35 @@
+CREATE TABLE "visitor_chat_messages" (
+	"id" uuid PRIMARY KEY NOT NULL,
+	"chat_id" uuid NOT NULL,
+	"sequence" integer NOT NULL,
+	"sender_id" uuid,
+	"sender_role" text NOT NULL,
+	"sender_name" text NOT NULL,
+	"sender_image" text,
+	"body" text NOT NULL,
+	"created_at" timestamp with time zone DEFAULT now() NOT NULL
+);
+--> statement-breakpoint
+CREATE TABLE "visitor_chats" (
+	"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
+	"origin_session_id" uuid NOT NULL,
+	"visitor_proof_hash" text NOT NULL,
+	"company" jsonb,
+	"next_sequence" integer DEFAULT 0 NOT NULL,
+	"delivered_through" integer DEFAULT 0 NOT NULL,
+	"read_through" integer DEFAULT 0 NOT NULL,
+	"last_visitor_seen_at" timestamp with time zone,
+	"muted_at" timestamp with time zone,
+	"closed_at" timestamp with time zone,
+	"created_at" timestamp with time zone DEFAULT now() NOT NULL,
+	"updated_at" timestamp with time zone DEFAULT now() NOT NULL,
+	CONSTRAINT "visitor_chats_origin_session_id_unique" UNIQUE("origin_session_id")
+);
+--> statement-breakpoint
+ALTER TABLE "live_sessions" ADD COLUMN "chat_proof_hash" text;--> statement-breakpoint
+ALTER TABLE "live_sessions" ADD COLUMN "visitor_chat_id" uuid;--> statement-breakpoint
+ALTER TABLE "visitor_chat_messages" ADD CONSTRAINT "visitor_chat_messages_chat_id_visitor_chats_id_fk" FOREIGN KEY ("chat_id") REFERENCES "public"."visitor_chats"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
+ALTER TABLE "visitor_chat_messages" ADD CONSTRAINT "visitor_chat_messages_sender_id_users_id_fk" FOREIGN KEY ("sender_id") REFERENCES "public"."users"("id") ON DELETE set null ON UPDATE no action;--> statement-breakpoint
+CREATE UNIQUE INDEX "visitor_chat_messages_sequence_idx" ON "visitor_chat_messages" USING btree ("chat_id","sequence");--> statement-breakpoint
+CREATE INDEX "visitor_chats_updated_at_idx" ON "visitor_chats" USING btree ("updated_at");--> statement-breakpoint
+ALTER TABLE "live_sessions" ADD CONSTRAINT "live_sessions_visitor_chat_id_visitor_chats_id_fk" FOREIGN KEY ("visitor_chat_id") REFERENCES "public"."visitor_chats"("id") ON DELETE set null ON UPDATE no action;