Skip to content

Cache choice weight tables, and stop wasting SQL on discarded exact-distinct queries - #174

Open
yhong123 wants to merge 3 commits into
mainfrom
yhong123/147-optimise_choice_proposers
Open

yhong123 wants to merge 3 commits into
mainfrom
yhong123/147-optimise_choice_proposers

Conversation

@yhong123

@yhong123 yhong123 commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Closes #147.
Works towards #160.

weighted_choice/zipf_choice_direct were rebuilding their weight tables on every single generated row (#147). Also, there is a separate, pre-existing bottleneck in ChoiceProposerFactory.get_proposers — the SQL step that builds proposers in the first place — dominates total time for high-cardinality columns and was masking the improvement. This PR bundles all three fixes together.

Changes

1. weighted_choice/zipf_choice_direct per-row caching

datafaker/providers.py

  • weighted_choice(a) previously rebuilt its (values, counts) lists from scratch on every call, and random.choices(..., weights=...) redid its own O(n) cumulative-weight setup every time too, even though a is the same list object across an entire generate_data(count) loop.
  • Added an id(a)-keyed cache (DistributionProvider._weighted_choice_cache) so the derived (values, cumulative_counts) are only rebuilt when a is actually a new list, and switched random.choices to cum_weights= (precomputed) instead of weights= (recomputed every call).
  • Same idea for zipf_choice_direct: zipf_cumulative_weights(size) is functools.cached by size.

Measured (generate_data(4000), concept_id column, isolated from any SQL changes):

proposer before after speedup
zipf_choice [sampled] 0.0483s 0.0043s 11x
weighted_choice [sampled] 0.0983s 0.0054s 18x
choice [sampled] 0.0008s 0.0018s unaffected (never the bottleneck)

2. Stop sorting stmt_count results that get thrown away

datafaker/proposers/choice.py

  • ChoiceProposerFactory.get_proposers's exact stmt_count query had ORDER BY count DESC, sorting the entire table in the database — even though, for any column over MAXIMUM_CHOICES (500) distinct values (the common case for a real column), the result is discarded unseen once the row-count cap check fails.
  • Removed the SQL-side ORDER BY; when the cap check does pass (≤500 rows), the small result is sorted in Python instead, preserving ZipfChoiceProposer's most-frequent-first ordering requirement exactly.

Measured (get_proposers, concept_id, a near-unique PK column): ~1.0s → 0.397s.

3. Skip the exact query entirely for high-cardinality columns

datafaker/proposers/choice.py

  • For an unindexed, very-high-cardinality column (e.g. concept_name), stmt_count's full-table GROUP BY was taking 12.6s, confirmed via EXPLAIN (ANALYZE, BUFFERS). Its result is discarded unseen anyway, same as above.
  • get_proposers now runs the cheap, bounded SAMPLE_COUNT-row sample query first. If that sample comes back ≥90% distinct (EXACT_SKIP_SAMPLE_DISTINCT_RATIO), the exact query is skipped outright, both unaffordable and pointless in that case.
  • The 90% threshold is deliberately conservative, so this only fires when cardinality is genuinely far over the cap (MAXIMUM_CHOICES).

Measured (get_proposers):

column before (Fix 1 only) after (+ skip heuristic) notes
concept_name (high cardinality, unindexed) 12.654s 1.320s ~9x speedup
concept_id (near-unique PK) 0.397s 0.455s no real regression — its exact query was already cheap via PK-index early exit, so skipping it saves little; difference is DB timing noise

Testing

  • New regression tests in tests/test_proposer.py:
    • ChoiceProposerFactoryOrderingTests — values still come back most-frequent-first after removing the SQL-side sort (verified to fail without the fix).
    • ChoiceProposerFactoryExactSkipTests — the exact query is skipped for a high-cardinality column and still runs for a low-cardinality one (captured via SQLAlchemy's before_cursor_execute; verified to fail without the fix).
  • Full test suite passes.
  • All numbers above were measured against a live OMOP-mimic Postgres database, not just synthetic test fixtures.

@yhong123 yhong123 self-assigned this Nov 13, 2026
@yhong123 yhong123 changed the title Cache choice weight tables, and stop wasting SQL Cache choice weight tables, and stop wasting SQL on discarded exact-distinct queries Nov 16, 2026

@nikk-nikaznan nikk-nikaznan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking good, thanks Yean Hoon. I have some small comment below.

Comment thread datafaker/proposers/choice.py Outdated
Comment on lines +388 to +389
results = connection.execute(stmt_count)
if results is not None and results.rowcount <= MAXIMUM_CHOICES:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this work with duckDB as well?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like .rowcount is not generally useful with Select queries: https://docs.sqlalchemy.org/en/20/core/connections.html#sqlalchemy.engine.CursorResult.rowcount

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know why we were getting away with it before.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you both! I have applied the suggestions.

Comment thread datafaker/proposers/choice.py Resolved
@yhong123
yhong123 force-pushed the yhong123/147-optimise_choice_proposers branch from db20e2f to b75fdbf Compare November 19, 2026 10:55

@tim-band tim-band left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great!

Comment thread datafaker/providers.py
vs.append(vc.get("value", None))
c = random.choices(vs, weights=counts)[0]
return c
cached = self._weighted_choice_cache.get(id(a))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An alternative approach would be to make a method that figures out the cumulative counts then decorate it with @lru_cache. This would behave a bit differently but would look more readable. Would it be better?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the good question @tim-band
I actually benchmarked this to check. weighted_choice’s elements are dicts, so they’re unhashable and lru_cache can’t take a directly. The cached method would need a hashable key built from a first (e.g. a tuple of counts), and building that key means walking all of a on every call, which is the exact O(n) extraction loop the id(a) cache is designed to skip on a hit. So the lru_cache version ends up paying that walk every single call, cache hit or not.

I ran benchmark test against the mimic concept table (generate_data(4000), which calls weighted_choice ~4000 times with the same list object:

proposer id(a) cache (current) lru_cache version
weighted_choice [sampled] (~500-row list) 0.0055s 0.0989s (~18x slower)

So in practice the lru_cache approach is slower than the current one for the case this fix targets, it reintroduces the per-call O(n) cost that #147 was written to remove. I’d like to keep the id(a)-based cache as is.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dist_gen.weighted_choice/zipf_choice rebuild their weight tables on every single call

3 participants