Native Execution Engine Landscape
Different Distributed Query Execution Architectures. Same Velox Library. Different Results.
Why Architecture Matters
ClipperDB/Spark
Fully Native Spark Worker Architecture
Fully native Spark query plan execution
Shared asynchronous data cache
Streaming exchange
Low query variance
Efficient query fault tolerance
Prestissimo
Fully Native Presto Worker Architecture
Fully native Presto query plan execution
Shared asynchronous data cache
Streaming exchange
Low query variance
Gluten/Lightning
JVM-Based Velox Integrations
JVM executors in workers JNI with Velox
Separate JVM, Arrow and Velox memory domains
Duplicated Velox caches in each executor
Exchange materialization overhead
Clipper DB and Prestissimo
Same Economics: 2x+ Speedup on Half the Infrastructure
140-
120-
Stock EMR Spark
Stock PrestoDB
EMR ClipperDB
EMR Prestissimo
120
100-
GeomeanElapsed Time (s)
80-
60-
40-
20-
0-
57
66
26
15
15
10
35
13
10
7
6
Stock
Cold Cache
Warm Cache
Two slave cluster
Stock
Cold Cache
Warm Cache
Four slave cluster
Lower is Better
Environment: AWS EMR 75, PrestoDB 0.287, TPC-DS Subset with Same Query Plans, 2 and 4 worker r6dni.4xlarge, at 3TB scale in S3
Clipper DB vs Gluten/Lightning
2x Cold Cache 3x Warm Cache
ClipperDB speeds up jobs to 6x
7-
6-
5.65x
5-
4-
3-
Geomean Speedup Factor (x)
1.94x
2.00x
2-
1-
0-
Lightning
Cold
1.00x
Lightning
Warm
ClipperDB
Cold
ClipperDB
Warm
Higher is better
Environment: GCP Dataproc 3.4, TPC-DS Full Sweep, 8 Worker C4N-highmem-16, at 3TB scale in GCS
ClipperDB vs Gluten/Lighting
TPC-DS Power Benchmark
ClipperDB speed up: 3.4x vs Gluten/Lightning, 4.4x vs Stock Dataproc
20-
18-
16-
16.01
14-
12-
12.28
10-
Elapsed Time (s)
4-
0-
3.65
2-
6-
8-
Stock Dataproc
Lightning
ClipperDB
Lower is better
Environment: GCP Dataproc 3.4, TPC-DS Full Sweep sequentially executed, 8 Worker C4N-highmem-16, at 3TB scale in GCS