Skip to content

Scale imports and evaluations ,presign uploads(csv), FlexPrice batching - #143

Open
MSami625 wants to merge 86 commits into
mainfrom
scaling-platform
Open

MSami625 wants to merge 86 commits into
mainfrom
scaling-platform

Conversation

@MSami625

Copy link
Copy Markdown
Collaborator

What Changed?

Imports scale (Linear imports epic) on branch scaling-platform: backend and frontend changes so large CSV/platform imports, diarisation, and evaluations (100k+ rows) stay usable without timing out or melting Postgres/Redis/FlexPrice.

  • API structure: Split monolithic call_imports / call_import_evaluations routes into modules; bulk cancel/retry/delete with server-side row filters (not huge UUID lists from the browser).
  • Uploads: Presigned CSV staging + manual audio presign (parallel PUTs); legacy in-API upload paths removed or deprecated.
  • Platform imports: Hydrate/discovery pipeline fixes (_hand_off), eval allowed when rows are terminal even if discovery is still listing; stale slice reclaim + finalize when hydrate is idle; debounced rollup / batch status.
  • Workers & concurrency: Fair import/platform/eval dispatch, shared import retry backoff for telephony/eval slots; Celery revoke fix for thread pool aborts; eager worker import guard.
  • Sharding: Scatter/gather for filtered rows, eval list pages, aggregates; keyset-sampled completed rows for TLDR / clusters / user insights.
  • Charts / aggregates: Migration 109 cached metric aggregates; on-demand refresh on aggregate GET (no every-25-rows worker rebuild); terminal-only cache rebuild on worker.
  • Search: Migration 111 pg_trgm + row_text_search (escaped ILIKE) on conversation id + transcripts across eval rows and import filters.
  • Billing: FlexPrice Redis buffer (ack after successful ingest, requeue on failure); batched recording minutes (~50 events per large eval); buffered eval pass deltas.
  • UX polling: Active import/eval polling (importPolling, bulk-op progress, any_evaluation_poll_active on import detail/list).
  • Data: recording_duration_seconds on rows (migrations 108/110).

(This branch may include other work outside imports; the above is the imports-scale scope for this PR.)

Why?

Imports were hitting limits at tens/hundreds of thousands of rows: full-table scans for search and charts, discovery/import stuck in processing, browser-driven bulk ops, upload timeouts, FlexPrice event storms, and insights/clusters loading every row pair. This PR keeps behavior backward-compatible while moving work to presign + worker paging + cache/sampling + indexes + safer billing.

How to Test?

  1. Migrations: Apply 108–111 on catalog + shard DBs; restart API, import-control, imports, eval, usage workers.
  2. CSV: Presign stage → map → diarise/eval on a medium batch; confirm no CORS/401 on PUT.
  3. Platform: ElevenLabs (or similar) import; rows finish hydrate; eval can run before last discovery page; import leaves processing when rows idle.
  4. Scale UX: Large eval — bulk abort/retry with filter; list/search q; charts tab while running (cache stale → refresh on open).
  5. FlexPrice (if enabled): large eval completes; recording minutes not ~1 event/row; buffer survives failed ingest then retries.
  6. Automated:
    .venv/bin/pytest tests/test_workers/test_call_import_worker_modules.py tests/test_services/test_platform_call_imports.py tests/test_services/test_billing/test_flexprice_buffer.py tests/test_services/test_recording_minutes_batch_size.py tests/test_services/test_metric_aggregate_cache.py tests/test_api/test_call_import_evaluation_rows_search.py tests/test_workers/test_evaluate_call_import_row.py -k rollup_parent -q

Release Label

  • major
  • minor — backward-compatible imports-scale features and performance
  • fix
  • No label

Checklist

  • I have read the CONTRIBUTING.md guide.
  • My code follows the project's style guidelines.
  • I have added tests that prove my fix is effective or my feature works.
  • I have updated documentation where needed. (S3 CORS example script exists; ops note: migrations 108–111 + worker restarts.)

TEJASNARAYANS and others added 30 commits October 22, 2026 16:45
… export functionality with workspace_id filtering
… config coercion and improving option label normalization
…ation run summary to include failed trial errors
… and update related processing logic to handle audio duration in various tasks
… and evaluations, enhancing API request handling
… call imports, enhancing audio file handling capabilities
Comment thread app/services/call_imports/bulk_ops.py Resolved
MSami625 and others added 2 commits November 9, 2026 13:16
…usters_wave.py and adjust bulk operations for improved handling of cancellation scenarios
Comment thread app/services/harbor/failure_analysis.py Resolved
@TEJASNARAYANS TEJASNARAYANS added the minor Minor version update label Nov 11, 2026
Point main's two new LLM-service redaction tests at
resolve_integration_for_call, which this branch renamed from
resolve_integration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread app/services/call_imports/bulk_ops.py Resolved
TEJASNARAYANS and others added 2 commits November 11, 2026 15:42
Keep the 8 remote commits and the 3 local commits.
Comment thread app/services/usage/read_cache.py Resolved
Comment thread frontend/src/pages/usage/Usage.tsx Resolved
Comment thread frontend/src/pages/usage/Usage.tsx Resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

minor Minor version update

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants