Why is my TPC-DS SQL query running slower with Auron compared to native Spark?

[Dormant7]

For example, when running TPC-DS (1TB) query 6, it took 500 seconds with Spark 3.4.1 + Auron, but only 60 seconds with native Spark 3.4.1.Many other SQL queries show the same behavior.

Spark on k8s , this is my base config:
spark.memory.offHeap.enabled false
spark.executor.memoryOverhead 4096
spark.executor.memory 4g
spark.executor.cores 4
spark.memory.fraction 0.6

What could be the cause? 3q

[weiqingy]

Thanks for reporting this. A couple of things might help narrow it down.

The config you shared looks close to the README quick-start, which is meant as a minimal starting point rather than a benchmarking profile. It might be worth comparing it against the setup behind the published 1TB numbers (https://auron.apache.org/documents/benchmarks.html) — especially spark.executor.memoryOverhead, spark.executor.cores, and spark.auron.memoryFraction, since native execution in Auron draws its memory from the overhead region rather than Spark's off-heap pool (offHeap.enabled=false is expected). Do the two configs differ much?

A few questions that would help:

1. Does the slowdown show up across most queries, or mainly a few like q6?
2. In EXPLAIN FORMATTED (or the Spark UI SQL tab), does the Auron plan mostly use Native*/Auron* operators, or are there stretches of regular Spark operators mixed in?
3. In the Spark UI for q6, how do the spill and shuffle read/write numbers compare between the Auron and native-Spark runs?

That should give us a good sense of whether it's a memory/spill effect or something in how the plan is being executed.