Repository navigation
Conversation
nikk-nikaznan
left a comment
There was a problem hiding this comment.
This is looking good, thanks Yean Hoon. I have some small comment below.
| results = connection.execute(stmt_count) | ||
| if results is not None and results.rowcount <= MAXIMUM_CHOICES: |
There was a problem hiding this comment.
Will this work with duckDB as well?
There was a problem hiding this comment.
Looks like .rowcount is not generally useful with Select queries: https://docs.sqlalchemy.org/en/20/core/connections.html#sqlalchemy.engine.CursorResult.rowcount
There was a problem hiding this comment.
I don't know why we were getting away with it before.
There was a problem hiding this comment.
Thank you both! I have applied the suggestions.
…ggestion Co-authored-by: nikk-nikaznan <48319650+nikk-nikaznan@users.noreply.github.com>
db20e2f to
b75fdbf
Compare
| vs.append(vc.get("value", None)) | ||
| c = random.choices(vs, weights=counts)[0] | ||
| return c | ||
| cached = self._weighted_choice_cache.get(id(a)) |
There was a problem hiding this comment.
An alternative approach would be to make a method that figures out the cumulative counts then decorate it with @lru_cache. This would behave a bit differently but would look more readable. Would it be better?
There was a problem hiding this comment.
Thanks for the good question @tim-band
I actually benchmarked this to check. weighted_choice’s elements are dicts, so they’re unhashable and lru_cache can’t take a directly. The cached method would need a hashable key built from a first (e.g. a tuple of counts), and building that key means walking all of a on every call, which is the exact O(n) extraction loop the id(a) cache is designed to skip on a hit. So the lru_cache version ends up paying that walk every single call, cache hit or not.
I ran benchmark test against the mimic concept table (generate_data(4000), which calls weighted_choice ~4000 times with the same list object:
| proposer | id(a) cache (current) | lru_cache version |
|---|---|---|
weighted_choice [sampled] (~500-row list) |
0.0055s | 0.0989s (~18x slower) |
So in practice the lru_cache approach is slower than the current one for the case this fix targets, it reintroduces the per-call O(n) cost that #147 was written to remove. I’d like to keep the id(a)-based cache as is.
Summary
Closes #147.
Works towards #160.
weighted_choice/zipf_choice_directwere rebuilding their weight tables on every single generated row (#147). Also, there is a separate, pre-existing bottleneck inChoiceProposerFactory.get_proposers— the SQL step that builds proposers in the first place — dominates total time for high-cardinality columns and was masking the improvement. This PR bundles all three fixes together.Changes
1.
weighted_choice/zipf_choice_directper-row cachingdatafaker/providers.pyweighted_choice(a)previously rebuilt its(values, counts)lists from scratch on every call, andrandom.choices(..., weights=...)redid its own O(n) cumulative-weight setup every time too, even thoughais the same list object across an entiregenerate_data(count)loop.id(a)-keyed cache (DistributionProvider._weighted_choice_cache) so the derived(values, cumulative_counts)are only rebuilt whenais actually a new list, and switchedrandom.choicestocum_weights=(precomputed) instead ofweights=(recomputed every call).zipf_choice_direct:zipf_cumulative_weights(size)isfunctools.cached bysize.Measured (
generate_data(4000),concept_idcolumn, isolated from any SQL changes):zipf_choice [sampled]weighted_choice [sampled]choice [sampled]2. Stop sorting
stmt_countresults that get thrown awaydatafaker/proposers/choice.pyChoiceProposerFactory.get_proposers's exactstmt_countquery hadORDER BY count DESC, sorting the entire table in the database — even though, for any column overMAXIMUM_CHOICES(500) distinct values (the common case for a real column), the result is discarded unseen once the row-count cap check fails.ORDER BY; when the cap check does pass (≤500 rows), the small result is sorted in Python instead, preservingZipfChoiceProposer's most-frequent-first ordering requirement exactly.Measured (
get_proposers,concept_id, a near-unique PK column): ~1.0s → 0.397s.3. Skip the exact query entirely for high-cardinality columns
datafaker/proposers/choice.pyconcept_name),stmt_count's full-tableGROUP BYwas taking 12.6s, confirmed viaEXPLAIN (ANALYZE, BUFFERS). Its result is discarded unseen anyway, same as above.get_proposersnow runs the cheap, boundedSAMPLE_COUNT-row sample query first. If that sample comes back ≥90% distinct (EXACT_SKIP_SAMPLE_DISTINCT_RATIO), the exact query is skipped outright, both unaffordable and pointless in that case.MAXIMUM_CHOICES).Measured (
get_proposers):concept_name(high cardinality, unindexed)concept_id(near-unique PK)Testing
tests/test_proposer.py:ChoiceProposerFactoryOrderingTests— values still come back most-frequent-first after removing the SQL-side sort (verified to fail without the fix).ChoiceProposerFactoryExactSkipTests— the exact query is skipped for a high-cardinality column and still runs for a low-cardinality one (captured via SQLAlchemy'sbefore_cursor_execute; verified to fail without the fix).