In the past 2 to 3 years, Presto has set the bar for fast analytical processing in modern cloud data lake architectures. Qubole has offered a Presto service as part of its open data lake platform since 2014. We have tried to make sure we are offering one of the best performances and also easy-to-use Presto-based services.
In this blog post, we compare the Presto service on Qubole against one of the popular competitors’ Presto offerings (referred to as “ABC”) in AWS. In order to do an apples-to-apples comparison, we run TPC-DS (Scale 1000) benchmark queries on identical cluster configurations in terms of number and type of worker nodes, and use identical Presto configurations for both offerings.
We conducted two types of benchmarking runs –
- Run without any additional data preparation or Hive metastore statistics to represent the experience that most users commonly have with the service.
- Run with Hive metastore statistics that are configured to update regularly in the background to represent how experts and practitioners leverage the Presto Cost-Based Optimizer (CBO) to realize significant performance gains. Metastore statistics such as table size, number of rows, and column cardinality help Presto’s CBO generate efficient query plans that reduce data scanned, and lower peak memory requirements resulting in higher overall performance.
In summary, Presto on Qubole was 2.6x faster than ABC Presto in terms of overall Geomean of the 100 TPC-DS queries for the no-stats run. More importantly, 94% of queries were faster on Presto on Qubole with 41% of the queries being more than 3x faster and another 23% of the queries being 2x-3x faster.
Benchmarking Configuration:
Our benchmarking runs consisted of TPC-DS (Scale 1000) queries on unpartitioned ORC data stored on AWS S3. We used on-demand nodes on AWS for the Presto cluster with the following configuration:
Master node: r5.4xlarge – 16 cores, 128GiB memory
Worker nodes: r5.4xlarge – 16 cores, 128GiB memory
We used the latest Presto versions supported on both platforms – Presto 317 on Qubole, and Presto 0.227 on ABC. AWS RDS was shared for both Presto on Qubole and ABC so as to have a common metastore, database, and statistics.
Following Presto configurations were enabled on both platforms:
- query.max-memory-per-node=78GB
When Hive metastore statistics were available, the following Presto configurations were enabled on both platforms:
- optimizer.join-reordering-strategy=AUTOMATIC
- join-distribution-type=AUTOMATIC
- join-max-broadcast-table-size=1GB
In addition, dynamic filtering (not available on ABC) was enabled on Presto on Qubole for both sets of runs. For the no-stats run, we enabled size-based statistics on Presto on Qubole. Size-based statistics is a new feature that is unique to the Qubole platform. This involves estimating table statistics on the basis of the table’s size on the storage layer. It is useful in cases where metastore statistics for tables are not available. If Presto on Qubole is unable to find table statistics in the metastore, it can fetch the size of the table on the storage layer and estimate the size and the number of rows in the table. This estimate can be used by the CBO to select the optimal join distribution type and reorder joins.
Benchmarking Results:
We now present the results of our benchmarks. We first present the overall geomean of the 100 TPC-DS queries to provide a score for the overall run, and then we provide a breakdown of how individual queries fared.
No metastore statistics (no-stats) run:
For this run, we enabled size-based stats on Presto on Qubole. We compare size-based stats configuration in Presto on Qubole against the no-stats run in ABC Presto.
Geomean for Presto on Qubole with size-based statistics was 2.56 times faster than ABC Presto when run without any metastore statistics.
- 94% of the queries were faster on Presto on Qubole
- 41% of the queries were more than 3X faster on Presto on Qubole
- 23% of the queries were between 2X and 3X faster on Presto on Qubole
The graphs below show individual queries that were more than 3x, 2x-3x, and 1.5x-2x faster on Presto on Qubole. In these graphs, the blue bars represent the Query timings for Presto on Qubole and the red bars represent ABC Presto.
Queries that were 3x and faster (41% of them):
Queries that were 2x-3x faster (23% of them):
Queries that were 1.5x-2x faster:
With Hive metastore statistics run:
For this run, hive metastore statistics were collected, and AWS RDS was shared between Presto on Qubole and ABC so as to have a common metastore, database, and set of statistics.
Geomean for Presto on Qubole was 1.56 times faster than ABC Presto when run with metastore statistics.
- 84% of the queries were faster on Presto on Qubole
- 10% of the queries were more than 3X faster on Presto on Qubole
- 8% of the queries were between 2X and 3X faster on Presto on Qubole
- 44% of the queries were at least 1.5X faster on Presto on Qubole
Graphs below show individual queries that were more than 3x, 2x-3x, and 1.5x-2x faster on Presto on Qubole. In these graphs, the blue bars represent the Query timings for Presto on Qubole and the red bars represent ABC Presto.
Queries that were 3x and faster (10% of them):
Queries that were 2x-3x faster (8% of them):
Queries that were 1.5x-2x faster (25% of them):
Conclusion:
Our benchmarking results show that Presto on Qubole was 2.6x faster than ABC Presto in terms of overall Geomean of the 100 TPC-DS queries for the no-stats run. More importantly, 94% of queries were faster on Presto on Qubole with 41% of the queries being more than 3x faster and another 23% of the queries being 2x-3x faster. Even when Hive metastore statistics are available, Presto on Qubole was 1.6x faster than ABC Presto in terms of overall Geomean of the 100 TPC-DS queries. In this run, overall, almost 84% of the queries were faster on Presto on Qubole while 44% of the queries were at least 1.5x or faster on Presto on Qubole.
A large part of these performance gains can be attributed to Dynamic Filtering and Size-based statistics – features that are not available on ABC. Furthermore, the Presto service on Qubole offers this high performance at a lower cost than the competition by leveraging differentiated features around the use of low-cost compute with higher reliability and workload-aware autoscaling.
