All pull requests

Pull requests

• fix: [branch-1.1] keep the values of sliced batches when coalescing a broadcast (#6900)

  bug
  #7328·laxmanmamididd opened Oct 29, 2026·

  ·

• build: bump pyo3 to 0.29 and add a cargo audit config

  buildenhancementrun-pyarrow-udf-tests
  #7327·andygrove opened Oct 29, 2026Member·

  ·

• fix: evaluate the right operand of AND, OR and IN only for rows the left side has not decided

  area:expressionsbugrun-spark-4.1-tests
  #7326·andygrove opened Oct 29, 2026Member·

  ·
  1

• feat: preserve columnar batches across observations

  enhancement
  #7325·pingzh opened Oct 29, 2026Contributor·

  ·
  1

  9

• feat: run filter lambdas natively with DataFusion's array_filter

  area:expressionsenhancementrun-all-spark-profilesrun-benchmark-checkrun-spark-4.1-tests
  #7322·andygrove opened Oct 29, 2026Member·

  ·
  1

  9

• fix: cap the rows per batch so a large string scalar fits 32-bit offsets

  area:scanbug
  #7316·zhangfengcdt opened Oct 29, 2026Member·

  ·
  1

  11

• fix: [branch-1.2] copy rows retained by identity broadcasts (#7260)

  area:ffibug
  #7314·andygrove opened Oct 29, 2026Member·

  ·

• ci: run the pull request tier for fork pull requests on the fork

  buildenhancement
  #7311·andygrove opened Oct 28, 2026Member·

  ·
  1

  31

• perf: reduce per-batch broadcast serializer allocations

  enhancementperformancerun-spark-4.1-tests
  #7310·sunchao opened Oct 28, 2026Member·Queued
  1

• fix: give zstd the real destination capacity when compressing cached batches

  bug
  #7307·andygrove opened Oct 28, 2026Member·

  ·
  1

  13

• deps: bump arrow-java to 19.0.0

  dependencies
  #7306·andygrove opened Oct 28, 2026Member·

  ·

• feat: trim the native heap when an executor's last native plan finishes

  enhancement
  #7304·comphead opened Oct 28, 2026Contributor·

  ·
  1

• fix: apply the session's Comet configs in map stages of jobs outside a SQL execution

  area:shufflebackport-1.2bugrun-spark-4.1-tests
  #7302·comphead opened Oct 28, 2026Contributor·Queued
  1

  8

• feat: add comet-scheduler and comet-executor for Comet Native Executors

  area:ffiarea:scanenhancement
  #7299·andygrove opened Oct 28, 2026Member·

  ·
  15

• feat: prototype Comet Native Executors built on DataFusion Ballista

  area:scanenhancement
  #7289·andygrove opened Oct 28, 2026Member·

  ·

• fix: preserve make_date evaluation across lazy operators

  area:expressionsbug
  #7283·pingzh opened Oct 28, 2026Contributor·Queued
  30

• Iceberg bin pack compaction test

  area:Iceberg
  #7282·comphead opened Oct 28, 2026Contributor·

  ·

• fix: pass the session batch size to the native FilterExec

  bug
  #7280·parthchandra opened Oct 28, 2026Contributor·

  ·
  1

• perf: coalesce native shuffle batches before broadcast serialization

  area:shuffleenhancementperformancerun-benchmark-checkrun-spark-4.1-tests
  #7258·sunchao opened Oct 27, 2026Member·

  ·
  1

  58

• fix: declare the input order of a native window over a sorted cache

  bugrun-all-spark-profilesrun-spark-4.1-tests
  #7256·comphead opened Oct 27, 2026Contributor·

  ·
  1

  11

• fix: use historic zone offset for extreme-year string-to-timestamp casts

  area:expressionsbug
  #7255·ump45nose opened Oct 27, 2026Contributor·

  ·
  1

  15

• feat: run user-defined generators inside the native generate operator

  area:expressionsenhancementrun-spark-4.1-tests
  #7253·andygrove opened Oct 26, 2026Member·

  ·
  1

  12

• feat: run inline, inline_outer, and stack generators natively

  enhancementrun-spark-4.1-tests
  #7251·andygrove opened Oct 26, 2026Member·

  ·
  1

  14

• docs: describe the long-lived branch for upgrading DataFusion

  documentation
  #7250·andygrove opened Oct 26, 2026Member·

  ·
  4

• deps: upgrade to DataFusion 56.0.0

  area:Deltaarea:expressionsarea:ffiarea:Icebergarea:joinsarea:scanarea:shuffledependenciesrun-all-spark-profilesrun-iceberg-tests
  #7249·andygrove opened Oct 26, 2026Member·

  ·